Re: Problems with Bayesian filtering (Example #2 from AOL)

[Justin Mason]
> Yep, that meshes with what SpamAssassin and spambayes both found as
> well: the key to a good implementation of a Bayes-like classifier is
> the tokenizer.  Several %age points of accuracy are made and lost
> there.

I'll just note that the biggest single win the spambayes project ever got
was via special tokenizing of embedded URLs.  That happened early, and cut
the false negative rate in half all by itself.  It's also one of the only
*pure* wins we ever saw (meaning that it didn't lead to an increase in false
positives on any cross-validation test run on anybody's corpus).

If the probability engine is the brain, the tokenizer forms the eyes, and
the brain can only judge what it sees (well, except for our experimental
highly intuitive classifier <wink>).