Re: Query/suggestion

John Chandler wrote:
> So I am a happy user, 

Great!


> but am always wondering about results that other users get, 
> and about improved algorithms for spam detection.
 >
 > Query:
 > Are other users still getting 98% or better efficiency from spamprobe?

I don't count since I'm highly biased (being the author) and spamprobe 
is tuned using my own email corpus.  Having said that (just for fun) in 
February 2005 I received 16,876 spams (all but 884 of them with scores 
over 0.995) that SP correctly classified as spam.  I also received 17 
emails that SP incorrectly classified as ham.  Nine of those had scores 
over 0.5 (I use the default 0.6 as my cutoff).  That's nearly 99.9% for 
my own email in February.




> Now the first line,
> 
>     0.9999967       4       0      93  aamnjgithiifeua8brqtaxihqdykeg
> 
> seems to me to be highly relevant, 
> whereas later lines such as
> 
>     0.0000032       1      47       0  he might
> 
> would seem to be much less relevant.

Remember that for most people (certainly for me) the database will 
contain many more spams than hams so good counts can be lower but 
produce a more relevent score.

You can see that effect in your example.  The spammy term 
"U_aamnjgithiifeua8brqtaxihqdykeg" with zero good count and 49 spam 
count was less significant.  (You can determine which terms are more or 
less significant to SP by taking the distance from 0.5.

"he might" ==> 0.4999968

"U_aamnjgithiifeua8brqtaxihqdykeg" ==> 0.4999938


SP's term sorting algorithm is somewhat involved.  SP breaks the terms 
into groups based on their difference from 0.5 (i.e. two terms scores 
that are different but very close together will be in the same group) 
and tries to give hammy terms and terms with higher in message counts 
higher priority.



> Ignoring of common words has long been done 
> in information indexing and retrieval systems.
> It would seem to me that counting occurrences of such common words as
> "he", "it", "his", "your", "what", "she", "to", etc.
> may be counterproductive, 
> and even "might", "says", "mind", "follow", etc.
> may be of questionable utility.
> Not considering these words would remove 
> almost all of the "good" words from the list above.
> Of course other "good" words would then appear.

SP and other Grahamian filters effectively do this already.  SP only 
uses the most significant terms when it computes its scores.  That means 
that common words, which tend to have neutral scores (around 0.5) are 
generally ignored.

In my own database here are the scores for a few of the common words you 
mentioned:

0.3665643    4165    9639  0x000003a5  he
0.4167525    2422    6921  0x000003a5  his
0.5000000   29321  116320  0x000003a5  your
0.2077130   12568   13177  0x000003a5  what
0.5822307     966    5384  0x000003a5  she


> Would it be reasonable to add a feature to spamprobe
> (or other Bayesian spam detectors) to allow the user
> to specify a file of common words 
> that are not to be counted by the detector,
> either singly or in pairs, 
> or possibly specify them as being counted in pairs but not singly,
> or some such feature?

I don't think this would be a desirable feature for several reasons:

1. From the examples above you can see that many of the words that 
people assume will be neutral aren't.  For example "what" in my own 
database is fairly hammy and "she" is fairly spammy.  In messages with 
few terms these might make their way into the top terms list and allow 
SP to make a correct determination that would be impossible if they were 
ignored.

2. Stop terms will be highly language dependent and, as illustrated 
above, highly user dependent.  Why make users manage such a thing when 
the computer can do it for them?

3. If the stop word list is large then it will have to be stored 
somewhere and in a manner that allows it to be loaded/used efficiently. 
  I'd probably wind up putting it into the database anyway since ISAM is 
a lot more efficient than loading a text file on startup.  If it's in 
the database then any disk space savings would be trivial.  That being 
the case why not let SP build and adjust the list for me automatically?

4. You can already override SP's judgement on a term yourself using the 
"edit-term" command to set counts on a term manually.  Just use some 
arbitrary (and large) count to force SP in either direction.  For 
example if you want to make "foobar"  always look hammy you could do 
something like this:

    spamprobe edit-term foobar 100000000 0

or you can force a term to be neutral using a large number for both counts:

    spamprobe edit-term foobar 1000000 1000000



> Or the detector itself could count all words 
> and not use those with a relative frequency above some threshold
> in computing the spam score.

You can do this with SP if you like.  Just run it with a really large 
value to the -w option.

   spamprobe -w1000 -r1 score

That would use the top 1000 terms from each email and would prevent any 
single word from being counted more than once in the array.  I don't 
recommend this!  In my experiments adding in too many terms tends to 
degrade accuracy by allowing a bunch of neutral terms to overwhelm the 
most significant ones (exactly what spammers try to hard to accomplish).


> No doubt this has occurred to authors of spam detectors...

Can't speak for others but I certainly have given it some thought. :-)

Thanks for the idea though!  I'm always interested in suggestions for 
improving SP.

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click