Re: important words

k b wrote:
> Could you elaborate on how SpamProbe chooses the important words?
> I've noticed the number of important words varies, how is this decided?
> 
> ( could it be they must be above/below some probability threshold, and
> the number of important words must be in some range? )
> 
> I see the -r option changes the number of times a token can be
> 'important', what is the default?

It's fairly involved but here are the basics:

  1. parse email to produce terms (words, phrases, url parts, etc)
  2. score each term (ignore brand new terms)
  3. break terms into groups based on their dfm (dfm is abs(score - 0.5))
  4. within each group sort based on good/spam (ham first), frequency
     in message (terms seen more often first), and score
  5. take terms from the sorted list and add them to the top terms
     array until it's full

The top terms array using default settings has 27 slots.  Any given term 
can be added to the array multiple times if it appears in the message 
more than once.  By default a term can only be added to the top terms 
array twice (so if it appears in the email 100 times it still only gets 
two slots).

Step 4 is pretty involved.  You can see the details in the source code 
if you're interested.  I use qsort() to do the sorting with a function 
that computes a sort value and resolves ties.

The general goal is to favor hammy terms over spammy ones to avoid false 
positives and to give terms which appear frequently in the message more 
weight than ones that appear only once.

All the best,
++Brian


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar