Re: important words

> It's fairly involved but here are the basics:
> 
>   1. parse email to produce terms (words, phrases, url parts, etc)
>   2. score each term (ignore brand new terms)
>   3. break terms into groups based on their dfm (dfm is abs(score - 0.5))
>   4. within each group sort based on good/spam (ham first), frequency
>      in message (terms seen more often first), and score
>   5. take terms from the sorted list and add them to the top terms
>      array until it's full
> 
> The top terms array using default settings has 27 slots.  Any given term
> can be added to the array multiple times if it appears in the message
> more than once.  By default a term can only be added to the top terms
> array twice (so if it appears in the email 100 times it still only gets
> two slots).


Thank you for the information.

How did you decide on the number 27?
I think our old friend Paul Graham used 15,
and Pantel/Lin used every token (which is just wrong, IMHO).

k b


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar