Re: (no subject)

Carl,

>From: Carl Hutzler <[email protected]>
>
>Excellent idea. The key is in the similarity detection engine. If
...........
>http://www.pornosite.com/
>ht<htmlcomment>tp://w<comment>ww.po<comment>rnosi<anotherone>te.com
>http://rd.yahoo.com/?http://w<comment>ww.po<comment>rnosi<anotherone>te.com
>...and consider using plain text URL with html tables, etc.
>
>This is the difficult area. Whether it is phone numbers, emails, URLs, or 
>the attributes of a bayesian filter, the comparison algorithms and the 
>parser that preprocesses the mail is the gold mine.

I remove all web tags, correct char codings and break the http:-tags at / 
and ? etc., so if the spammer have a redirection, I get two address lines. I 
also shorten these to least common denominator and remove some common 
domains as AOL(;-)/Yahoo etc. so the only address I see and process would be 
"pornosite.com" (I work with a copy of the mail, so I can maul it as much as 
I want). And this gives a very high hit rate, matching ca 19000 domains I 
collected from ca 150000 spams.

I have looked at it for a year now and it works, though since I only have 
time to work with it as a shell script PoC, it is now to slow for else than 
a local mailbox. Phone-nos is not as easy, but I have it working resonably, 
but it is fewer of those in my test mailbox. With a faster filter and a 
cache, it should be more than feasable catching undiscovered domains/numbers 
and (since in this context quite usable) the smtp routing, checking in 
parallel.

Regards Kurt

_________________________________________________________________
The new MSN 8: smart spam protection and 2 months FREE*  
http://join.msn.com/?page=features/junkmail