Re: improving spamprobe a la popfile

Le dimanche 28 novembre 2004 à 11:04 -0500, Brian Burton a écrit :
> Sami Dalouche wrote:
> > Being able to automatically sort ALL my e-mail (not just good/junk) is
> > something I'm really looking for. 
> > It looks like popfile (http://popfile.sf.net) can do it, but the way it
> > works is not what I'm looking for (I don't want the mail filter to act
> > as a proxy..).
> 
> My take on this has always been that filing ham in the proper mailbox is 
> a trivial problem that has already been solved by just about every mail 
> client.  Detecting spam is hard and needs a sledge hammer (Graham's 
> method) but detecting mail from this mailing list is trivial (header, 
> subject, etc are consistent).
> 

Well, classical mail filters are definitely not the right solution for
everything. Of course, sorting mailing-lists is pretty easy with
procmail, since you can write generic filters that fetch the name of the
mailing-list automatically to sort into several folders. 
But, what about mails that are not regular ? Like.. I like to sort
emails from friends in a specific folder. If I write procmail rules, it
means I have to constantly tweak my rc file to match the newly-changed
email of a specific contact, or to add the work email of a friend. This
leads to messy rc file that grows over time, and needs maintenance.
Some bayesian filter used intelligently (mailing-lists would not pass
through the filter, since they're sorted easily, for example) would help
me doing my task, even if it fails sometimes, it's not a BIG issue.. 

> Multi-box filing looks nice on the old feature list but does it really 
> buy you anything?  Also there is a price to be paid:
> 
>   * database size grows since you have to store separate counts for each 
> <term,mailbox> combination.
> 
>   * processing time increases since you have to score the message for 
> each mailbox and decide which one wins
> 
>   * reliability is questionable since a message that scores 90% for one 
> mailbox and 88% for another will probably be misclassified
> 
> 
Yes, right of course, it has a price. But if spamprobe is more generic,
it doesn't mean everyone will use of price of multi-box filing. the
price would be the same as the current one for the person who only uses
2 folders, right ?

> > So, I'm either considering to write my own Bayesian mail filter, or to
> > improve spamprobe to suit my needs.
> 
> Writing a mail filter is a lot of fun.  You're cetainly free to base 
> your code on SP as long as you abide by the QPL and give me credit in 
> your distro and source code.
> 
Yes, of course I would have to accept the licensing.. ;) BTW, what are
the differences (roughly) between the QPL and the GPL ? 
But yes, in any case I would give credit, that's a must :)

> 
> > First of all, do you think enhancing spamprobe to support N folders
> > instead of just 2 (good/junk) is easily doable ? 
> 
> Define easy. :-)
> 
good :)

> It wouldn't be that big a stretch.  You'd need a new set of command line 
> args (i.e. train-box boxname filename... or something).  You'd need to 
> enhance the database to efficiently handle multiple mailboxes (e.g. 
> don't store the box name in each term record so you now need multiple 
> database files).

Do you really think that using multiple databases would be more
efficient than using only one ? Using one table (boxname, term, count)
with a primary key (boxname, term) would allow indexing on
(boxname,term), and wouldn't necessary be a bod choice, would it ?

>  You'd need to score each message for each mailbox and 
> somehow pick a winner with some way of resolving ties and near ties. 
> etc etc.
> 

ok, thanks for the tips !

> 
> > Secondly, are you interested by the modifications, or do you consider
> > spamprobe as a spam-detector only piece of software ? 
> > I think it'd be sad to fork() and  have 2 different codebases, but if
> > spamprobe's goal is not to sort emails,...  that would mean I would have
> > to maintain my own branch, and in this case, I certainly wouldn't use C
> > ++, but something like C# to ease future maintenance.
> 
> <gratuitous slap at C++ ignored>
> 

Hmm, sorry for bashing C++. But the problem here is more MY ability to
maintain C++ code compared to some other higher-level language. I
haven't touched any C# code yet, but it definitively looks easier to
maintain for a non-expert ;-)

> The changes would be extensive and would not be backward compatible. 
> They'd probably introduce a cost for people who don't want multi-mailbox 
> classification.  For those reasons and others I doubt I'd want to roll 
> them into SP.
> 
Hmm, ok, so that means I'd have to work on a separate work base.. No
problem!, But I just wanted to be sure, before forking and then
realizing the orginial owner would have been interested. I'm pretty much
against the usual way of forking, or duplicating work, so whenever
possible, collaboration is a good thing :-)


> 
> > Finally, can anyone give me a few hints, pointers to doc, or anything
> > that can help me understand spamprobe's architecture please ? 
> 
> Why not layer SP and ifile?  Let SP separate out the spam and then let 
> ifile process the ham to put it into a mailbox for you?  No coding to do 
> and you get want you want for free.

The problem with this  approach is that I still have to use popfile's
proxy way of sorting mail. And I hate it. I don't want to screw up w/
fake redirections between procmail and popfile to have part of my mail
sorted w/ procmail, and the other part w/ popfile... Oh, and I also get
some emails directly on my box, and I'm not sure how popfile would deal
with it ? I'd have to fetch mail from my local pop3 folder ?? mmh..
nice ... 

> 
> All the best,
> ++Brian

In any case, thanks a lot for your help ! Since you don't look
interested by the modifications, I guess I'll start some new project by
copying your ideas  (or start with your software if I feel motivated
enough to do it in C++). 

Regards,
Sam



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/