Re: improving spamprobe a la popfile

Sami Dalouche wrote:
> Being able to automatically sort ALL my e-mail (not just good/junk) is
> something I'm really looking for. 
> It looks like popfile (http://popfile.sf.net) can do it, but the way it
> works is not what I'm looking for (I don't want the mail filter to act
> as a proxy..).

My take on this has always been that filing ham in the proper mailbox is 
a trivial problem that has already been solved by just about every mail 
client.  Detecting spam is hard and needs a sledge hammer (Graham's 
method) but detecting mail from this mailing list is trivial (header, 
subject, etc are consistent).

Multi-box filing looks nice on the old feature list but does it really 
buy you anything?  Also there is a price to be paid:

  * database size grows since you have to store separate counts for each 
<term,mailbox> combination.

  * processing time increases since you have to score the message for 
each mailbox and decide which one wins

  * reliability is questionable since a message that scores 90% for one 
mailbox and 88% for another will probably be misclassified


> So, I'm either considering to write my own Bayesian mail filter, or to
> improve spamprobe to suit my needs.

Writing a mail filter is a lot of fun.  You're cetainly free to base 
your code on SP as long as you abide by the QPL and give me credit in 
your distro and source code.


> First of all, do you think enhancing spamprobe to support N folders
> instead of just 2 (good/junk) is easily doable ? 

Define easy. :-)

It wouldn't be that big a stretch.  You'd need a new set of command line 
args (i.e. train-box boxname filename... or something).  You'd need to 
enhance the database to efficiently handle multiple mailboxes (e.g. 
don't store the box name in each term record so you now need multiple 
database files). You'd need to score each message for each mailbox and 
somehow pick a winner with some way of resolving ties and near ties. 
etc etc.


> Secondly, are you interested by the modifications, or do you consider
> spamprobe as a spam-detector only piece of software ? 
> I think it'd be sad to fork() and  have 2 different codebases, but if
> spamprobe's goal is not to sort emails,...  that would mean I would have
> to maintain my own branch, and in this case, I certainly wouldn't use C
> ++, but something like C# to ease future maintenance.

<gratuitous slap at C++ ignored>

The changes would be extensive and would not be backward compatible. 
They'd probably introduce a cost for people who don't want multi-mailbox 
classification.  For those reasons and others I doubt I'd want to roll 
them into SP.


> Finally, can anyone give me a few hints, pointers to doc, or anything
> that can help me understand spamprobe's architecture please ? 

Why not layer SP and ifile?  Let SP separate out the spam and then let 
ifile process the ham to put it into a mailbox for you?  No coding to do 
and you get want you want for free.

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/