Re: Re: Error (signal 11) during auto-train

David Rosen wrote:
> Regarding the code that repeatedly scores and re-classifies a message up 
> to 25 times, I imagine it helps in the short term, but I'm curious to 
> know how it affects accuracy in the long-term.  I have a few thoughts on 
> this:

Just to explain to anyone reading who doesn't know what we're 
discussing, that loop is the "stubborn classification" where spamprobe 
classifies messages multiple times if necessary to make them score 
properly.  That allows SP to learn much faster when a new type of email 
is seen or when a type of email that used to be considered ham is now to 
be considered spam (a mailing list gone bad for example).

The limit of 25 times is a failsafe to avoid any potential for infinite 
looping.  Very few emails hit that limit.  You can see how many times 
the loop ran for a message by dumping your database and looking at the 
MD5_ terms.  A big majority will have counts of 1.  Very few will have 
counts over 5 or so.


> 1. If the message doesn't score correctly after adding its term counts 
> 2-3 times, it probably doesn't help to have its term counts in the 
> database more than once, if at all.  Consider a spam that contains a 
> single image plus a few paragraphs of unrelated text pulled from an 
> article about programming.  Considering that a lot of my emails relate 
> to programming, many of the terms from the text will have high good 
> counts, therefore this could not possibly score as spam using my 
> database.  I would think that it is not beneficial to have these terms 
> counted as spam in the database 25 times.

Sorry but my experiments don't agree with your intuition.  You can't 
make a blanket statement that such a message will never be recognizable 
as spam.  Generally the spammer has to guess the "right" programming 
terms to get through.  He might send dozens of these and only one makes 
it through.  Once SP trains on that message the likely hood of those 
same terms allowing the mail to pass is much lower.

I also disagree that having those randomly chosen (by the spammer) terms 
added to the database multiple times as spammy is bad.  The counts in 
the database are dynamic and adapt to the flow of email.  If spamers use 
a term then it's not pure anymore.  If they stop using it then the term 
will float back towards purity again.  That's the nature of statistical 
filtering.  Applying heuristic rules or intuition is much less accurate 
than letting the algorithm decide based on what it sees.  There are 
already biases towards innocence in the algorithm.

I agree that stubborn classification isn't perfect.  There will be some 
mail that hits the 25 limit (I have a few in my database) but even if 
the first such message can't be made to score properly in 25 tries then 
the next similar message will be that much easier to score.


> 2. The purge/cleanup commands become biased toward removing terms from 
> messages that were more obviously spam.  Terms made up of random 
> characters can end up in the database up to 25 times and would not be 
> purged as soon as they should, using a typical cleanup schedule.

Actually that's not the case.  Cleanup operates on term counts, not 
message counts.  If a message was classified by the loop 10 times then 
its terms have counts of 10 (or 20, 30, etc) so they are actually less 
likely to be cleaned up than if the message was classified once.


>  3. Does this code get called when using the auto-train/auto-learn 
> commands?  

All classification is stubborn so yes, the loop applies to auto-train.


 > Probably the optimal way to do this while building a database
> would be to classify all the messages in both corpuses first, then go 
> back and score and re-classify each message as necessary. 

That's called TUNE by some other filters (Train Until No Errors).  It's 
a great ivory tower way to go but rescanning a complete corpus multiple 
times is time consuming.  SP's stubborn classification is, IMHO, more 
practical since it can be done very fast (with caching no database 
changes are written to disk until loop is finished) and achieves a 
significant improvement in accuracy.

Nothing is stopping a person from using multiple auto-train passes 
themselves.  I just didn't want to build that in as default behavior 
because it would take significantly longer to train and would not yield 
a commensurate improvement in accuracy.


 > I imagine the
 > random alternating spam/good of the auto-train/auto-learn commands
 > probably has a similar result.

Not really.  The random element is just an effort to keep the training 
of spam/ham in proportion to their representation in the corpus.  That's 
why auto-train is so much better than separate train-good and train-spam 
runs.  Emails are classified in proportion to the number of each type 
just as they will be in a real email flow.

These are good thoughts and it's great to have such discussions on the 
list.  Sorry I disagreed so much...

All the best,
++Brian


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642