Re: Fwd: Re: ifile vs. Graham modified naive bayes?

> [email protected]:

> Probably the _ideal_ way of handling the DB change would be to integrate
> in the source code for CDB.  <http://cr.yp.to/cdb.html>  The source code
> is 51K long, uncompressed, so it would not be too terribly troublesome
> to throw it into a subdirectory.

DCB is basically read-only.  it is handy if you have a database of constant
items, like IP numbers (the tcpserver(1)s "-x" option uses CDB directly) or
domain names (see djbdns), but in case of change, the entire CDB has to be
rebuilt.  what made you think it was suitable for this task?

you were proposing the DB for faster access and update, so CDB is out of
the question, because you'd loose due to the complete rebuilding for the
change of only one item.

> I'd think the sensible approach would be to add the option "--cdb-file"
> whereby it would read .idata as a CDB file.

... thus restricting ifile to non-djb installations.  think of debian: 
they can't use any djb stuff, because it isn't open-source.  also, about
every modern computer-installation (except mine :) has some database
already installed, say MySQL or Postgres.  these are different enough to
make building and maintenance a nightmare.

> I don't personally care if there's a space after the "-- "; let me make
> the observation that in my Spam folders, there are 2855 messages at the
> moment.  73 of those contain a 'legitimate' .signature.  35 contain a
> "--" (with-no-space) not-legitimate .signature.  
> 
> Either way, about 2% of the spam contains .signatures, which seems
> suggestive that noticing signatures is likely to be handy, right?

these are good points, because they are based on concrete data.  it might
be worth to gather some more statistics like that.

> I think we can get excellent milage out of some very _simple_
> heuristics.  I don't see there being much value in trying to extract
> meaningful data out of uuencoded or base64 encoded data.  I'd be
> _perfectly happy_ with the following options:
> 
> if ((current_token[0] == 'M') && (strlen(current_token) == 61)) {
>    current_token = "lookslikeuuencoding";
> }
> if (strlen(current_token) == 76) {
>   current_token = "maybebase64";
> }
> if (strlen(current_token) > 35) {
>   current_token = "BigLongString";
> }

i don't understand these.  what is a "token" here, and what property
relates it's length to an encoding?

> If we worked _really_ a lot harder at it, we might run messages  through
> metamail, find the list of files generated, and if they are JPEGs, run
> them through /usr/bin/file, and maybe get something like the following:

well, at the moment my entire anti-spam technology (and qmails queuing)
delays incoming emails upto two minutes.  with analysis like the proposed,
this will rise dramatically, with not much benefit.  the overwhealming
majority of the pictures i get are spam-related.  pictures in emails are a
clear spam-sign for me, so i toss them some place i rarely visit.

going for MIME-analysis is beyond ifiles purpose, i think, aside from
looking into HTML (if you don't reject it anyway).

> baseball.jpg:               JPEG image data, JFIF standard 1.01, aspect
> ratio, "ppm created by umax", 1 x 1

you could even throw steganographic-detection at it ...

clemens