Spamprobe 1.1x6 released

Just in time for the holiday I've released a new "unstable" version of 
SP.  This version replaces the old hash file database implementation 
with a brand new and greatly improved one.  The old version had several 
problems.  The worst of which was that it would drop terms fairly 
frequently (around 3% of the time) which is why I never advocated its 
production use.

http://prdownloads.sourceforge.net/spamprobe/spamprobe-1.1x6.tar.gz?download

This new hash implementation solves the collision problem by storing the 
32 hash code in the data file and using it to resolve collisions.  There 
is still a small possibility of confusing terms if they happen to have 
the same hash code (maybe once in a few million terms) but in my tests 
with a 16,000 message corpus I saw exactly the same accuracy using the 
hash format as using the PBL format.

There are two big advantages to the hash format.  The first is raw, 
blazing speed.  In my experiments the hash file format is around 2x the 
speed of PBL (ranged from 1.8x to 3.5x). The second advantage is that 
the hash data file size is fixed.  You choose a size when you create the 
file and it never changes.  File size can be anywhere from 1-100 MB. 
You need to choose a size large enough to hold your terms with room to 
spare.  More on that later.

To create a hash data file you add a prefix to the directory name in the 
-d command line option.  You can specify just the directory like this:

   spamprobe -d hash:$HOME/.spamprobe

or you can add a size in megabytes for the file like this:

   spamprobe -d hash:42:$HOME/.spamprobe

The size is only used when a file is first created.  SP auto detects the 
size of an existing hash file.  You need to allow enough space for twice 
as many terms as you are likely to have in your file.  In my database I 
have 2.2 million terms.  That required a database of are 53 MB.  SP uses 
12 bytes per term in the hash file so you can estimate the file size 
you'll need by multiplying the number of terms by 24.

The hash format does not store the original terms.  Instead it stores 
the 32 bit hash code for each term.  You can do just about anything with 
a hash file that you could with a PBL file including import/export, 
edit-term, cleanup, purge, etc.  You can use export your PBL database 
and import it to build a hash file (note that you cannot go the other 
direction) and you can export one hash file and import into a new one to 
enlarge your file.

This release also documents the auto-train feature.

If anyone could try out the hash file format and report back any 
problems or issues I'd be very grateful.  I'm using it for my own email 
now but still consider it to be beta code.

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click