Re: Fwd: Re: ifile vs. Graham modified naive bayes?

> > [email protected]:
> >    What would be nice would be to stuff the stats into a DBM file on a
> >    word by word basis.
> 
> is it worth the complications in building?  

There lies the problem.  I have looked at the code, and found it
troublesome to hook anything of the sort in.

> now ifile does a prertty good job and runs on about every platform.
> if we wanted speed, we might use mmap(2).  it is feasable with current
> database sizes and might perform better than a DB without beeing as
> complicated.  difficult to draw the line, though.  more features could
> lead to a situation where a DB might help much more.  the pure
> text-format of the current database made it easier in case someone
> comes up with tools to improve the classifier "afterwards", i mean
> some sort of an optimizer.  does that make sense?

Probably the _ideal_ way of handling the DB change would be to integrate
in the source code for CDB.  <http://cr.yp.to/cdb.html>  The source code
is 51K long, uncompressed, so it would not be too terribly troublesome
to throw it into a subdirectory.

I'd think the sensible approach would be to add the option "--cdb-file"
whereby it would read .idata as a CDB file.

It's dead easy to write a Perl script that reads an .idata file and
pushes it into a CDB file.

I'd be perfectly happy to see a "translation to CDB" be something that
was not done completely automagically.

>> It is always possible with email to detect where the headers are, and
>> is usually possible to detect a .signature.
>
> i agree re. headers and MIME, but everything else mutates ifile into
> something bigger.  something bigger than ifile should be well designed.  i
> find particular appealing the small size of it, the robustness and the
> simplicity.  everybody can understand it, even the way it (tries to) adapt
> to stuff appearing in headers versus possibly the same looking stuff in the
> body.

>> As a pointed example, it has the result that some messages with
>> .signatures containing the word "sexy" would not lead to any
>> confusion of this with messages containing the word "sexy" in the
>> body.  They will be regarded as totally distinct.  Headers will
>> correlate with headers; body with body; .sig with .sig.  That's not
>> turning it into being _totally_ "contexted," but it adds a _little_
>> more context.

> i have to admit that i'm against .sig distinguishing, because there
> are so many people out there who either don't know how sigs are
> formatted or whose mailers have a propriatary idea about it.  you know
> how we mean :)

<sarcasm>
_Competently formatted_ .sigs are likely a significant indicator, all by
themselves, of the message not being spam, right?
</sarcasm>

I don't personally care if there's a space after the "-- "; let me make
the observation that in my Spam folders, there are 2855 messages at the
moment.  73 of those contain a 'legitimate' .signature.  35 contain a
"--" (with-no-space) not-legitimate .signature.  

Either way, about 2% of the spam contains .signatures, which seems
suggestive that noticing signatures is likely to be handy, right?

I don't think there's any need for spectacularly sophisticated
algorithms here.  

I think we can get excellent milage out of some very _simple_
heuristics.  I don't see there being much value in trying to extract
meaningful data out of uuencoded or base64 encoded data.  I'd be
_perfectly happy_ with the following options:

if ((current_token[0] == 'M') && (strlen(current_token) == 61)) {
   current_token = "lookslikeuuencoding";
}
if (strlen(current_token) == 76) {
  current_token = "maybebase64";
}
if (strlen(current_token) > 35) {
  current_token = "BigLongString";
}

I'll bet that this combination of three tests would be perfectly
excellent at giving us what we want, namely the idea that a particular
message contains a whole bunch of chunks of likely-to-be-encoded data.

If we worked _really_ a lot harder at it, we might run messages  through
metamail, find the list of files generated, and if they are JPEGs, run
them through /usr/bin/file, and maybe get something like the following:

baseball.jpg:               JPEG image data, JFIF standard 1.01, aspect
ratio, "ppm created by umax", 1 x 1
biggun.jpg:                 JPEG image data, JFIF standard 1.01, aspect
ratio, "ppm created by umax", 1 x 1
corba_book.jpg:             JPEG image data, JFIF standard 1.01, aspect
ratio, 1 x 1
dcp00027.jpg:               JPEG image data, EXIF standard 0.77, 42 x 0
dinky_car.jpg:              JPEG image data, JFIF standard 1.01, aspect
ratio, "CREATOR: XV Version 3.10a  Rev:", 1 x 1
ifile.jpg:                  JPEG image data, JFIF standard 1.01, aspect
ratio, 1 x 1
lpi-but-w96.jpg:            JPEG image data, JFIF standard 1.01,
resolution (DPI), 72 x 72
mauve.jpg:                  JPEG image data, JFIF standard 1.01, aspect
ratio, 1 x 1
multics.jpg:                JPEG image data, JFIF standard 1.01,
resolution (DPI), 72 x 72
php-small-trans-light.jpg:  JPEG image data, JFIF standard 1.01, aspect
ratio, "CREATOR: XV Version 3.10a  Rev:", 1 x 1
sapgui.jpg:                 JPEG image data, JFIF standard 1.01, aspect
ratio, "CREATOR: XV Version 3.10a  Rev:", 1 x 1
shooting.jpg:               JPEG image data, JFIF standard 1.01, aspect
ratio, 1 x 1

Would that be better than a bunch of counts of "lookslikeuuencoding"?
Maybe.  Can we be sure?  It would be a LOT more work trying to analyze
what's in the binaries, and I think we'd get nicely statistically
significant results (informally, if not formally) from something as
simple as the nine lines of code up above.

I'm certainly with you on trying to "keep things simple."  I don't think
I'd go as far as saying "at all costs."  I think efficiency is currently
suffering, and that the added complication of drawing in 51K of CDB code
is probably warranted for the performance improvents that may result.
--
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://cbbrowne.com/info/finances.html
cc hello.c, in Canada, results in:
  eh.oot