Re: train-good and train-spam; cron error

Depends how much RAM ... I wouldnt mind using up an extra meg for better concurrency,
but high-volume systems may differ ... (maybe yet another flag ?)

As for more IO ... couldnt you use an atomic increment instead of read/add/write ?

as in something like
   
      UPDATE table SET count = count + N 

Or does PBL not support this ?  Hmm it probably doesnt :(
But then you'd only have to double-read on the rows which have to get updated.



> Daniel Higgins wrote:
> > email deliveries still go through, but they don't get the spamprobe
> > header (that's how i know something is wrong usually, there's a lot of
> > spam in my box :)
> 
> I've observed in my own mail that train-spam on really huge spamprobe 
> folders can keep the lock for a long time and cause procmail to back up 
> and eventually time out.  I don't know if that's the issue here but 
> something to consider.
> 
> I'm thinking of changing SP to use read locks while scoring and then 
> switch to a write lock only while writing out modifications to term 
> counts.  The upside is far better concurrency.  In particular train-spam 
> could run with no write lock at all if all of the mail in the file it's 
> processing is already properly classified (as it usually is in the 
> periodic scans of the spamprobe folder).
> 
> The down side would greater memory demands (I'd have to keep both the 
> old and new counts in memory during processing so I can calculate by how 
> much to change the current values in the database) and more database I/O 
> (I'd have to reread each term's counts just before writing so I can 
> compute the correct new counts to write to disk).  Both are necessary 
> because if I have a read lock while scoring then the term counts can 
> change before I get a chance to update the database.  (For example if 
> another process is scanning another mail file at the same time)
> 
> So the trade off question is what is more important?  RAM?  Or concurrency?
> 
> Opinions?
> 
> All the best,
> ++Brian
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/