Re: Spam article in SciAm

> David A. Lee wrote:
> > It discusses Bayesean filters
>
> Do they say anything about accuracy relative to heuristic filters?
>
>
>  > as well as a modification I've seen discussed here which creates terms
>  > not by spaces, but by overlapping characters, which aparently works
>  > well with asian text as well as html encoded hacks.
>
> This would be pretty trivial to implement in SP.  If I get a chance 
> maybe I'll run some experiments with it.  Especially if I implement a 
> config file something like this might be a nice user selectable option.

Brian, if you do feel like experimenting with other approaches to
tokenization, can I twist your arm to try the trie-based approach
I suggested last year (which is effectively a superset of that suggestion
above):

  http://www.gtoal.com/archives/2004/02/the_best_of_bot.html
  http://www.gtoal.com/spam/devel-temp/tokra3.c.html

Unfortunately I've not had enough free time to take this
beyond proof of concept[*] :-/  I'ld really love to see it tried
out in a real filter that has all the ancilliary bits that
are needed to make it usable.

Graham
*: not laziness, just too many projects with higher priorities!


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click