Re: important words

k b wrote:
> How did you decide on the number 27?
> I think our old friend Paul Graham used 15,
> and Pantel/Lin used every token (which is just wrong, IMHO).

Trial and error with my own email.  I ran a ton of tests looking for the 
optimal set of values (for my email).  I provided command line options 
to allow people to try their own values in case mine didn't work well 
for them.

27 with 2 repeats is pretty close to Paul's 15.  You could wind up with 
as few as 14 or as many as 27 unique terms in the array.  I haven't 
figured out what the average number of terms in the array works out to 
be in practice.

In my own tests adding all terms was a losing proposition with 
SpamProbe.  Some filters get away with it though.  Probably because they 
use a more mathematically sound algorithm.  Paul's is very much an 
engineer's formula, which is why I like it an mathematicians hate it. 
:-)  It works EXTREMELY well but isn't mathematically pure.

All the best,
++Brian


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar