Re: consistant failures on a new kind of spam

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Jem wrote:
>> regular cron cleanup). Instead, you would only be removing a 'useless'
>> term if forced to by collision. I can't see how there would be a big
>> performance hit, since you will quickly encounter unused terms that need
>> cleaning. If you don't encounter any such (worst case) and have to
>> iterate a lot, then the hash db is obviously too small to accommodate
>> all the useful tokens encountered.

Yeah that's a very cool idea.  I'll have to think about the implications 
a bit since I would not want to make any remaining terms unreachable. 
What I mean about that is that the cleanup command actually builds a 
whole new file and inserts the terms into it to ensure the terms are 
well distributed (for example if a term you are removing had caused 
another term to use a later slot due to a collision the cleanup will 
allow that other term to occupy the slot it really wants for better 
performance).  I don't think that would really be a problem though since 
  the slots on the way would remain filled, allowing the collision 
resolution to skip them just like they would have skipped the term 
you're overwriting.

I suppose it could use the standard cleanup parameters (min term count, 
max time since last use, etc) to decide when to overwrite a term during 
a collision.  Those could be pulled from the config file to keep the 
command line reasonable.

Yeah, I like this idea a lot at the moment.


[snip]
> If you have a tiny hash db that is filling with useful terms and nothing 
> that can be removed-on-collision, then you need a larger db. The user could 
> then increase their db size. A decent sized db results in a small number of 
> iterations on any database addition, both maximizing performance and 
> maximizing useful tokens for scoring.

All of these are good thoughts.

All the best,
++Brian

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.1 (Build 1557)
Charset: ISO-8859-1

wsBVAwUBRVtl4jxRyEoJfXIFAQjMVQf9HmlRUViSehBiGfJ8VXUfSBlozQBLTpso
baouIw0EOqkxMlSsa5xwF9al0sc0c+/thxZ3Vcniq9GX+kFlp1AcN04GX+gEjYdw
Pi4k9mVe7mcJ/4Skeu71yKuHYw1LWa1ESKIKGPkpYg03bEMkRvaPOyfW+KUz4kXc
BAFoATTMfAMRge1n75Bcb6GB6aK5VVcgic2fr0W0ZPcgjRmM328dmELkl96oboNY
pHcYs4XTcW/QD+ZYWmms1hzpwFCuAd9uK5fQjfylFPi6/7S88ii6J/r8RlozFtMl
TB3BPWDqm9xuoOBdSNJ4HBNqGIC6DsNPc4jfWqcWGxjWDLIC0mFapQ==
=6npu
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV