Re: spamprobe-1.1x7 released

David A. Lee wrote:
> Any ideas how well crash and recovery are working with the Hash database ?
> Considering all of my DB's eventually grow to > 40MB .. I might as well use the hash DB and then ignore them.
> But I am concerned about stability issues ... such as when the system crashes in the middle of a spamprobe ,
> or when spamprobe itself gets stuck (This happens to me about once a month, I get one spamprobe totally stuck,
> which then locks up procmail which locks up sendmail and I end up with zillions of stuck processes ...)
> If you kill spamprobe at unoportune times how badly does the Hash DB get hit ? Better or worse then PBL ?

My guess is that it'd handle a bad crash better than PBL and much better 
than Berkeley DB.  I have a lot of theories as to why.

The hash format uses mmap() so there is no user level buffering going 
on.  The kernel manages the dirty pages and writes them out when it sees 
fit just like it would for the swap partition.

The hash format itself couldn't be any more trivial.  It's just an array 
of 12 byte records with no fancy data structures.  Worst case I could 
think of is if the system crashed in such a way that only part of a 
record was written to disk.  At worst that would make one term invalid 
so I don't think it'd be a disaster.

The cleanup and purge commands actually build a new hash file (necessary 
to rehash the file when records are deleted) and then swap it into place 
to replace the old one (using link/unlink).  That means that crashing in 
the middle of a purge or cleanup won't leave the original file in an 
indeterminate state.  The worst case would be if the kernel crashed 
after the old file was renamed but before the new one is renamed.  That 
would be easy to recover from since the original file would still be in 
the directory (named sp_words.hash.bak).

Finally all scans of the file (i.e. by dump, export, purge, etc) are 
just linear scans through the array, not traversals of a tree, so they 
can't really get stuck in a loop or jump off the end of the file if the 
file is corrupted.


 > Is locking still done the same way ?

Locking is just the same as for all of the other formats.  I grab a 
simple shared or exclusive fctnl() lock on the file named lock in the 
.spamprobe directory while SP is running.  Couldn't be any easier.



One problem that you could run into is that if the hash file is way too 
small it can become full.  If that happens you'll get "caught 
runtime_error" messages and SP will exit with a non-zero exit code.  If 
that happens you can export/import into a larger hash file and 
everything should be fine.


WRT file size I was able to go from a 128 MB PBL database to a 70 MB 
hash database for my own email.  70MB was bigger than the necessary but 
I wanted to leave some more growing room.  It's best to keep hash tables 
about 1/2 empty to avoid excessive collisions.

On systems with lots of users I think the hash format will be a big win. 
  If your users don't currently have the option to run the dump command 
and see the terms in their database then they wouldn't even notice the 
difference.

The hash database is pretty solid but I'm not promising that it's 
flawless yet.  Best to test it for a while before putting it into 
production wholesale.

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click