Re: Problems with Bayesian filtering

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


John Graham-Cumming writes:
>Laird Breyer wrote:
>> On Feb 25 2004, Justin Mason wrote:
>> 
>> 
>>>>IMHO, picking up explicitly on individual spammer tricks is at best a stop
>>>>gap until a unified way of dealing with those is found.
>>>
>>>Could you expand on what you mean here?
>> 
>> 
>> Fair request ;-)
>> 
>> There's a finite amount of information in a message. Part of that
>> information conveys content, part conveys meta data such as display
>> and formatting suggestions or other things. 
>> 
>> Bayesian filters generally try to pin down the content by measuring
>> it against content that's been seen and classified before.
>> They try to skip the meta data, because the latter doesn't convey 
>> useful information.
>
>And that final sentence is a mistake.  Looking at the meta data helps 
>greatly in correctly filtering messages.  Very often the meta data is 
>one of the best indicators of spamminess.  I believe it was Paul Graham 
>who pointed out long ago that the color #FF0000 was a great spam 
>indicator for him.

FWIW, in our testing, we found the *exact* opposite.  Matching on HTML
colours is very error-prone.   I suspect Paul's high hit rate was a result
of him not receiving very little mail in HTML that was not spam.

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFAPjEyQTcbUG5Y7woRAtLWAJ9YD+oOacT4+IVpGr+yt1hhVYtFzwCfUQyt
p6Rffw55aFLu5MUol1jQvNc=
=djvn
-----END PGP SIGNATURE-----