Fwd: Re: ifile vs. Graham modified naive bayes?

Some of you might find this discussion interesting.  Robert later e-mailed
me to say that Paul's system and ifile give similar results on a larger
corpus.

Jason

------- Forwarded Message

To: Robert Feldt <[email protected]>
Cc: [email protected]
Subject: Re: ifile vs. Graham modified naive bayes? 
From: Jason Rennie <[email protected]>
Date: Thu, 22 Aug 2002 14:42:40 -0400

[email protected] said:
> I just tried it out and compared to a simple implementation of Paul
> Grahams spam filtering technique. It seems Grahams method get somewhat
> better results on my (very limited data). Is this something you have
> seen or should I expect things to get better as I collect more spam (I
> only have about 70 spam messages for training so far, while the normal
> messages count in the thousands)? 

From reading Paul's description, I think Paul's filter and ifile have
identical classification engines.

I'd be happy to answer questions about what exactly ifile does if
you're interested in learning the differences between the two systems.
It may very well be that Paul's system works better.  If so, it should
be easy to modify ifile to perform more like Paul's system.

[email protected] said:
> I think a potential reason that grahams technique fares somewhat
> better on my data is that he is not as picky as to what is a word
> (email adresses, html markup etc is all part of the stats) while ifile
> seem to focus on "ordinary" words. 

Quite possibly.  You can change the way that ifile chooses words.  The 
'-w' option only breaks words at spaces (e-mail addresses, html markup is 
kept intact).  If you use the ifilter.mh and irefile.mh scripts, you'll 
need to edit them and add '-w' to the string of $ifile_args (change it 
from "-h" to "-h -w").

Jason D. M. Rennie
MIT AI Lab
[email protected]
(617) 253-5339
http://www.ai.mit.edu/~jrennie/

P.S. Mind if I post this to the ifile discussion list?


------- End of Forwarded Message