Re: train-good and train-spam; cron error

Daniel Higgins wrote:
> email deliveries still go through, but they don't get the spamprobe
> header (that's how i know something is wrong usually, there's a lot of
> spam in my box :)

I've observed in my own mail that train-spam on really huge spamprobe 
folders can keep the lock for a long time and cause procmail to back up 
and eventually time out.  I don't know if that's the issue here but 
something to consider.

I'm thinking of changing SP to use read locks while scoring and then 
switch to a write lock only while writing out modifications to term 
counts.  The upside is far better concurrency.  In particular train-spam 
could run with no write lock at all if all of the mail in the file it's 
processing is already properly classified (as it usually is in the 
periodic scans of the spamprobe folder).

The down side would greater memory demands (I'd have to keep both the 
old and new counts in memory during processing so I can calculate by how 
much to change the current values in the database) and more database I/O 
(I'd have to reread each term's counts just before writing so I can 
compute the correct new counts to write to disk).  Both are necessary 
because if I have a read lock while scoring then the term counts can 
change before I get a chance to update the database.  (For example if 
another process is scanning another mail file at the same time)

So the trade off question is what is more important?  RAM?  Or concurrency?

Opinions?

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/