Re: consistant failures on a new kind of spam

> regular cron cleanup). Instead, you would only be removing a 'useless'
> term if forced to by collision. I can't see how there would be a big
> performance hit, since you will quickly encounter unused terms that need
> cleaning. If you don't encounter any such (worst case) and have to
> iterate a lot, then the hash db is obviously too small to accommodate
> all the useful tokens encountered.

Oh, this lead to another thought -

Perhaps implementing the above (along with output on a verbose run) would 
give the user feedback on whether their database is large enough! Spamprobe 
could say how much iteration it had to do when adding new terms - just 1 if 
there's a free slot (perfect), or the whole db if it can't find an unused 
term.

So you could kill two birds with one stone. It would answer the questions:
"How large should my database be?" and 
"How often should I clean, and what parameters?"

If you have a tiny hash db that is filling with useful terms and nothing 
that can be removed-on-collision, then you need a larger db. The user could 
then increase their db size. A decent sized db results in a small number of 
iterations on any database addition, both maximizing performance and 
maximizing useful tokens for scoring.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV