Re: Problems with Bayesian filtering

On Feb 26 2004, John Graham-Cumming wrote:

> And that final sentence is a mistake.  Looking at the meta data helps 
> greatly in correctly filtering messages.  Very often the meta data is 
> one of the best indicators of spamminess.  I believe it was Paul Graham 
> who pointed out long ago that the color #FF0000 was a great spam 
> indicator for him.

Surely you see different types of meta data as differently valuable?
There's an endless supply of meta data out there, are you going to
go looking for the valuable ones one by one forever? How do you
know that the latest indicator you find isn't an artifact of your dataset
only?

A few years ago, these same questions would have been asked about
simple pattern matching filtering rules. Progress was achieved by
going beyond the special cases (ie cases such as "does the From
address contain xxx", "does the body contain yyy") and treating the 
patterns statistically. 

I'm not denying that there is valuable information in the meta data,
just that handling it one trick at a time is untenable in the long
run. And you are dealing with this one trick at a time, since each new
trick needs its own small modification to the parser/tokenizer. 

I'm of course also glossing over what precisely constitutes meta
data. In some cases, it's easy to recognize (e.g. font and colour
attributes). As a rough idea, data which can change without affecting
the content for the reader/user is meta data. 

So to take up your example, #FF0000 is easily changeable to #FF0001
and constitutes clear meta data. Does your parser go out of its way
looking for those tags? Does it also look for dynamically created
pieces of HTML which produce the same effect? There has to be a better
way in the long run.

-- 
Laird Breyer.