Re: Problems with Bayesian filtering (Example #2 from AOL)

Any split-on-whitespace Naïve Bayes implementation will have a harder
time on smaller messages (simply because there are fewer occurances of
tokens).  JGC's (and a lot of other bayes implementations) have added
other, non-word tokens which are more "deterministic".

In our research, we've found that the smartest algorithms in the world
suck with bad tokens and really strong tokens work very well even with
simple algorithms.

Ex: The obfuscation of buy ( "B U Y" ) is a stronger sign than "B" & "U"
& "Y".


> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Carl Hutzler
> Sent: Wednesday, February 25, 2004 12:51 PM
> Cc: [email protected]
> Subject: Re: Problems with Bayesian filtering (Example #2 from AOL)
>
>
> Return-Path: <[email protected]>
> Received: from  rly-yf04.mx.aol.com (rly-yf04.mail.aol.com
> [172.18.205.68]) by air-yf01.mail.aol.com (v98.6) with ESMTP id
> MAILINYF13-8664037e6bc1ce; Sat, 21 Feb 2004 18:16:43 -0500
> Received: from  rwcrmhc11.comcast.net (rwcrmhc11.comcast.net
> [204.127.198.35]) by rly-yf04.mx.aol.com (v98.5) with ESMTP id
> MAILRELAYINYF48-8664037e6bc1ce; Sat, 21 Feb 2004 18:16:16 -0500
> Received: from c-24-7-108-51.client.comcast.net ([24.7.108.51])
>           by comcast.net (rwcrmhc11) with SMTP
>           id <2004022123153401300ddtl9e>; Sat, 21 Feb 2004
> 23:15:35 +0000
> Message-ID:
> <[email protected]>
> From: "Stephen Henderson" <[email protected]>
> Subject: hi, buy meridia    zscedxdckxikpdfotkcjcyutjqlre
> Date: Sat, 21 Feb 2004 23:15:37 +0000
> MIME-Version: 1.0
> Content-Type: text/plain
> Content-Transfer-Encoding: 8bit
> X-AOL-IP: 204.127.198.35
> To: undisclosed-recipients:;
> X-Mailer: Unknown (No Version)
>
> B U Y   Y O U R   M E D S   H E R E:
> http://www.url98u.com/
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Face it: crave  will prefers sweaty bears,  roundabouts  of first
> hatreds.Hrm. Let's assume that hairdryer licks  of scrotum  which
> prefers  other Koopa's edges.
> After all, scrape catnip megalithic  and Bud_Lite consumes filtered
> scissors.
>