Re: train-good and train-spam; cron error

i think the current system is fine. i was just pointing out some
shortcomings with BDB one has to be aware of. 

using PBL doesn't seem to cause any such problem, so i don't think it's
worth wasting the time to fix it in BDB. i'm in the process of moving
everyone to PBL, it just takes some time due to corrupted DBs that
prevent me from automating it

i can live with a single address lacking concurency, that's what a mail
queue is for (that, and the fact i keep telling everyone that there's no
guarantee of a speedy delivery for email). i just found it odd that it
was jamming the other accounts :)


btw, i stopped using the hashed database a while ago, i thought it had
been dropped in recent versions but i saw a post that made reference to
it recently. if you want to drop it that's fine by me :) though you
might want to check if anyone else is using it 

and hopefully the mailman will stop bugging you now :)

On Sat, 2004-12-11 at 14:44, Brian Burton wrote:
> Daniel Higgins wrote:
> > email deliveries still go through, but they don't get the spamprobe
> > header (that's how i know something is wrong usually, there's a lot of
> > spam in my box :)
> 
> I've observed in my own mail that train-spam on really huge spamprobe 
> folders can keep the lock for a long time and cause procmail to back up 
> and eventually time out.  I don't know if that's the issue here but 
> something to consider.
> 
> I'm thinking of changing SP to use read locks while scoring and then 
> switch to a write lock only while writing out modifications to term 
> counts.  The upside is far better concurrency.  In particular train-spam 
> could run with no write lock at all if all of the mail in the file it's 
> processing is already properly classified (as it usually is in the 
> periodic scans of the spamprobe folder).
> 
> The down side would greater memory demands (I'd have to keep both the 
> old and new counts in memory during processing so I can calculate by how 
> much to change the current values in the database) and more database I/O 
> (I'd have to reread each term's counts just before writing so I can 
> compute the correct new counts to write to disk).  Both are necessary 
> because if I have a read lock while scoring then the term counts can 
> change before I get a chance to update the database.  (For example if 
> another process is scanning another mail file at the same time)
> 
> So the trade off question is what is more important?  RAM?  Or concurrency?
> 
> Opinions?
> 
> All the best,
> ++Brian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/