Re: Database and diminishing returns

Jem wrote:
> Now that I've been using spamprobe since 2003 I'm wondering if there has 
> been a wrong impression about the size of a database you need to have 
> effective filtering. For instance, sure we know there are diminishing 
> returns in effectiveness as your database grows. But perhaps a small 
> database really can be much more capable than we originally thought?
> 
> Because I was concerned about database growth in the old days (DB etc) I 
> kept using score only, and then trained on corrections. The approx 6 months 
> of recent mail I've used to train now gives me about 99% accuracy, but what 
> I discovered today is I really have very few terms in my database:

Have you been running cleanup really aggressively too?  It's certainly 
possible to get great accuracy for a specific corpus with a small 
database.  For example in my "quick test" corpus (around 1500 emails) SP 
gets over 99% accuracy.  However if I use that database with no further 
training on a much larger corpus the accuracy is significantly less.

I think the problem is that spam (and ham to a lesser degree) changes 
over time.  The database that works great today won't be so good next 
month.  You've been handling that by using a "Train on Error" (TOE) 
method to allow your database to react to mistakes only rather than 
using SP's "Train on Unsure" (TOU).

I think that TOE is probably best for building an optimal size database. 
   The problem is that it's likely to make more mistakes over time than 
TOU and be more frustrating for average users.  I really need to make an 
automated test to compare the two under controlled conditions.


> berkes@mail:~$ spamprobe counts
> GOOD 1903 SPAM 3438
> berkes@mail:~$ spamprobe export | wc -l
>   139142
> 
> With the latest hash database, this 1.6 MB database is enough to give 99% 
> accuracy. That seems pretty amazing! I have two other distinct accounts, 
> and the database size and effectiveness statistics are comparable.

What you say gave me a neat idea.  Wouldn't it be cool to have a tool 
that could take an existing database and a set of emails and then build 
a new database that contains only the "top terms" used in scoring all of 
those emails.  Theoretically you'd wind up with the optimal size 
database for that set of email.  SP uses up to 27 terms when scoring so 
for 1,000 emails you could theoretically wind up with a database of at 
most 27,000 terms that could score those emails perfectly.

A person could run that process periodically to reset their database to 
a minimal size.  Kind of an "intelligent" cleanup.

The question is how would that minimal database do with incoming emails? 
   I suspect it might be disappointing.  The problem would be that 
common terms that were neutral before would suddenly become "unknown" 
and would be given the default score in incoming emails.  That might 
skew the scoring and allow some spams to start slipping through that 
wouldn't have with the more complete database.


Personally I was stunned to see that my database had grown to 2.24 
million terms!  Wow!  It inspired me to make a better cleanup routine 
for SP.  I'll be posting it in the next few days.  I added the ability 
to specify multiple cleanup params on a single command line so that you 
can do something like this:

   spamprobe cleanup 2 14 20 30 1000 180

So you'd remove terms that didn't become significant after two weeks 
plus somewhat significant terms that haven't been seen in a month and 
just about everything that hasn't been seen in 6 months.  Previously you 
would have had to run SP three times to do the same cleanup.

All the best,
++Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click