Re: Fwd: Re: ifile vs. Graham modified naive bayes?

> Some of you might find this discussion interesting.  Robert later e-mailed
> me to say that Paul's system and ifile give similar results on a larger
> corpus.

There are two changes I could see that would likely be helpful for the "next 
level;"

-> On the one hand, performance on a large corpus gets pretty painful.  All the
   word stats get read through, whether they get used or not.

   What would be nice would be to stuff the stats into a DBM file on a
   word by word basis.

   Keeping it compact, it could be done thus:

   $DBM{"spam"} = "25 1:17 2:23 3:72 4:5 17:21";
   $DBM{"sex"} = "27 1:2 2:291 17:4";
   $DBM{"sexy"} = "9 2:9 17:1";

   That would cut the number of file accesses to be one per word, and
   would significantly speed things up.

-> It would also be nice to do more sophisticated "symbol tagging."

   It is always possible with email to detect where the headers are, and
   is usually possible to detect a .signature.

   I'd think it helpful to be able to tell that certain words are found
   in those two areas of the message.

   So, in this message, you might find word stats for signature elements
   sig::criticize, sig::servis, sig::mile, and sig::shoes.  (And several
   other words :-).)

   That very nicely distinguishes a .signature that makes some smart
   remark about "gratuitously sexy user interfaces" from the use of the
   word "sexy" in spam.  (Graham observed that the word "sexy" was
   pretty highly correlated with spam; in my stats, it's a _much_
   stronger indicator than the word "sex.")

   Similarly, by doing "header stuff," we might find such elements as
   subject:ifile, subject:graham, subject:modified, subject:naive,
   subject: bayes, and in-reply-to:message, in-reply-to:from,
   in-reply-to:jason, and in-reply-to:rennie.

What this effectively does is to make the "Naive Bayesian Filtering"
very slightly less "Naive."  

As a pointed example, it has the result that some messages with
.signatures containing the word "sexy" would not lead to any confusion
of this with messages containing the word "sexy" in the body.  They will
be regarded as totally distinct.  Headers will correlate with headers;
body with body; .sig with .sig.  That's not turning it into being
_totally_ "contexted," but it adds a _little_ more context.
--
(reverse (concatenate 'string "ac.notelrac.teneerf@" "454aa"))
http://cbbrowne.com/info/languages.html
Never criticize anybody until  you have walked  a mile in their shoes,
because by that time you will be a mile away and have their shoes.
-- email sig, Brian Servis