Re: Problems with Bayesian filtering (Example #2 from AOL)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


"Jesse Dougherty" writes:
>Any split-on-whitespace Naïve Bayes implementation will have a harder
>time on smaller messages (simply because there are fewer occurances of
>tokens).  JGC's (and a lot of other bayes implementations) have added
>other, non-word tokens which are more "deterministic".
>
>In our research, we've found that the smartest algorithms in the world
>suck with bad tokens and really strong tokens work very well even with
>simple algorithms.

Yep, that meshes with what SpamAssassin and spambayes both found as
well: the key to a good implementation of a Bayes-like classifier is the
tokenizer.  Several %age points of accuracy are made and lost there.

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFAPSgmQTcbUG5Y7woRAg2OAJ95TTgqMOOoMLqjA2fgb5FgYwjuhwCg3IZt
Jda8nIpw4E+y2lkWomBIdvg=
=fCuq
-----END PGP SIGNATURE-----