> > [email protected]:
>
> > Probably the _ideal_ way of handling the DB change would be to integrate
> > in the source code for CDB. <http://cr.yp.to/cdb.html> The source code
> > is 51K long, uncompressed, so it would not be too terribly troublesome
> > to throw it into a subdirectory.
>
> DCB is basically read-only. it is handy if you have a database of constant
> items, like IP numbers (the tcpserver(1)s "-x" option uses CDB directly) or
> domain names (see djbdns), but in case of change, the entire CDB has to be
> rebuilt. what made you think it was suitable for this task?
>
> you were proposing the DB for faster access and update, so CDB is out of
> the question, because you'd loose due to the complete rebuilding for the
> change of only one item.
I did _not_ propose CDB for "faster update;" only for "faster access."
The thing about CDB is that it is _really, really fast_ at compiling
input data into a database file.
I'm _fine_ with .idata generally being a text file, that needs to be
"compiled" into a database, in much the same way ifile.c is expected to
be compiled into ifile.o when it changes.
Here's a perfectly good ".idata compiler.":
#!/usr/bin/perl
use CDB_File;
my $idata = new CDB_File ("/home/cbbrowne/.idata.cdb", "cdb.tmp");
open(IDATA, "</home/cbbrowne/.idata");
$folders = <IDATA>;
@fold = split(/ /, $folders);
my $fnum = 0;
foreach $folder (@fold) {
$fnum++;
$idata->insert("$fnum", "$folder");
}
$idata->insert('Number of Folders', $fnum);
$words = <IDATA>;
$messages = <IDATA>;
$idata->insert('Word Counts', $words); # Arguably should get split(/ /)
$idata->insert('Message Counts', $messages); # Ditto...
while ($wordline = <IDATA>) {
$wordline =~ /^(.+) (\d+) (.+)$/;
$idata->insert($1, "$2 $3");
}
$idata->finish;
It "compiles" a plain old standard .idata file 8MB in size into a
very-slightly-larger .cdb file in about five seconds.
I see no fundamental problem with needing to "recompile" it when there's
a change.
>> I'd think the sensible approach would be to add the option "--cdb-file"
>> whereby it would read .idata as a CDB file.
> ... thus restricting ifile to non-djb installations. think of debian:
> they can't use any djb stuff, because it isn't open-source. also,
> about every modern computer-installation (except mine :) has some
> database already installed, say MySQL or Postgres. these are
> different enough to make building and maintenance a nightmare.
No, there is no necessary restriction.
Copyright:
You may distribute unmodified copies of the cdb package. The C
source files are in the public domain; you may use them as you
wish. (D. J. Bernstein)
If you look, you'll see that "freecdb" is included in Debian, as are
python-cdb, libcdb-file-perl, libvpopmail-freecdb, dbskkd-cdb, and
skkdic-cdb.
Looking at /usr/share/doc/freecdb/README, I find:
What is freecdb?
---------------
It's a totally-public domain fork of cdb. cdb was originally created by
Dan Bernstein. See the original web page at
ftp://koobera.math.uic.edu/www/cdb.html for more on cdb.
What I would suggest be done is that the source code that is "in the
public domain; you may use them as you wish" as collected in freecdb be
outright _included_ as a subdirectory of ifile. There's no need to
worry about where the dependencies are if ifile outright _contains_ the
sources.
chvatal:/var/cache/apt/archives/freecdb-0.62# wc *.{c,h,pod} Makefile
README
16 33 208 cdb_hash.c
97 285 1842 cdb_seek.c
12 34 201 cdb_unpack.c
61 155 1160 cdbdump.c
36 71 572 cdbget.c
128 387 3001 cdbmake.c
117 331 2340 cdbmake_add.c
10 24 154 cdbmake_hash.c
11 32 186 cdbmake_pack.c
103 371 2529 cdbstats.c
85 211 1647 cdbtest.c
12 19 164 freecdb.h
35 82 705 freecdbmake.h
21 42 320 cdb_hash.pod
35 124 800 cdb_seek.pod
17 46 373 cdbdump.pod
28 93 624 cdbget.pod
58 205 1469 cdbmake.pod
44 117 826 cdbstats.pod
55 178 1164 cdbtest.pod
70 174 1674 Makefile
60 266 1808 README
1111 3280 23767 total
23K of code.
A perfectly reasonable quantity of code _in the public domain_ that
could trivially be added to the code base. The fact that it's a small
bit of SOURCE that could be added is what makes it a reasonable option
in comparison with the horrors of searching for "which DBM is installed
on this system, what API do we use, and how do we link to it?"
> > I don't personally care if there's a space after the "-- "; let me make
> > the observation that in my Spam folders, there are 2855 messages at the
> > moment. 73 of those contain a 'legitimate' .signature. 35 contain a
> > "--" (with-no-space) not-legitimate .signature.
> >
> > Either way, about 2% of the spam contains .signatures, which seems
> > suggestive that noticing signatures is likely to be handy, right?
>
> these are good points, because they are based on concrete data. it might
> be worth to gather some more statistics like that.
>
> > I think we can get excellent milage out of some very _simple_
> > heuristics. I don't see there being much value in trying to extract
> > meaningful data out of uuencoded or base64 encoded data. I'd be
> > _perfectly happy_ with the following options:
> >
> > if ((current_token[0] == 'M') && (strlen(current_token) == 61)) {
> > current_token = "lookslikeuuencoding";
> > }
> > if (strlen(current_token) == 76) {
> > current_token = "maybebase64";
> > }
> > if (strlen(current_token) > 35) {
> > current_token = "BigLongString";
> > }
>
> i don't understand these. what is a "token" here, and what property
> relates it's length to an encoding?
OK, how about I make it more nearly literal. Add to some function like
ifile_lexer_simple_postprocess_word(), the code:
int wordlen = strlen(buf);
if ((buf[0] == 'M') && (wordlen == 61)) {
buf = "lookslikeuuencoding";
}
if (wordlen == 76) {
buf = "maybebase64";
}
if (wordlen > 35) {
buf = "BigLongString";
}
Whether it's called "buf" or "current_token" or
"this_is_the_word_we_are_processing_right_now" isn't too terribly
important.
The point is that in looking at some UUencoded files, I see that lines
start with "M" and that the lines are 61 characters long. Similarly,
base64 material consists of strings 76 bytes long.
We may not be able to distinguish what's inside them, but if we simply
indicate what they best match against, that's some information, isn't
it? Being able to tell apart uuencoding from (probable) BASE64 from
other "big strings" is of _some_ value. The point is for there to be
lines in .idata looking like:
lookslikeuuencoding 475 23:1791 24:2477 27:82
maybebase64 470 7:2384 8:921 9:751
BigLongString 74 1:7 2:35 7:17 9:47 23:57 24:171 27:17
That's likely more useful than either TOTALLY throwing those stats away,
or folding them into a single line like:
bigstring 74 1:7 2:35 7:2401 8:921 9:798 23:1962 24:2648 27:99
It's _certainly_ a lot more useful than reporting the 5000 individual
strings that likely never recur.
>> If we worked _really_ a lot harder at it, we might run messages
>> through metamail, find the list of files generated, and if they are
>> JPEGs, run them through /usr/bin/file, and maybe get something like
>> the following:
>
> well, at the moment my entire anti-spam technology (and qmails
> queuing) delays incoming emails upto two minutes. with analysis like
> the proposed, this will rise dramatically, with not much benefit. the
> overwhealming majority of the pictures i get are spam-related.
> pictures in emails are a clear spam-sign for me, so i toss them some
> place i rarely visit.
>
> going for MIME-analysis is beyond ifiles purpose, i think, aside from
> looking into HTML (if you don't reject it anyway).
>
> > baseball.jpg: JPEG image data, JFIF standard 1.01, aspect
> > ratio, "ppm created by umax", 1 x 1
>
> you could even throw steganographic-detection at it ...
I'm not seriously proposing analyzing MIME material.
The JPEG example is there to indicate just how _little_ added meaningful
information would be added by doing rather a lot of fiddly work.
Replacing "a bunch of fiddly work" with three simple if statements that
essentially look at the length of the "token" certainly fits with the
notion of it being a useful heuristic.
--
(concatenate 'string "chris" "@cbbrowne.com")
http://cbbrowne.com/info/rdbms.html
DOS: n., A small annoying boot virus that causes random spontaneous
system crashes, usually just before saving a massive project. Easily
cured by Unix. See also MS-DOS, IBM-DOS, DR-DOS.
-- from David Vicker's .plan
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.