Re: Some last info regarding analysis

Jose Marcio,

>We use URL blocking here, but it seems, for our domain, that it is somewhat 
>limited.
>
>We begun adding some URLs a day, and the number of URLs grew till little 
>more than a thousand. After that, I begun tweaking the URLs to lower the 
>number. I also begun to remove URLs that weren't seen for more than four 
>months. Now, I have only 130 URLs.

Seems that you found the same as I did, i.e. if you removes the prefix
domains and works with the least common denominator, you get far fewer
domains. Then the filter only need to grep them properly, because there
is only that many ways to disguise them.

My problem is that I use the Solaris grep and something is wrong in that
one, making the match not only from the found URL's against the data
file, but also for the domains in the data file against other words in the
mails, a completely wrong behaviour.

I have some 19000 domains in my data file, but most of those I sieved
out from ca 150000 spams I retreived from Spamarchive, so I block
url's I never seen.

>OBS - the time spent by the filter looking for regular expressions grows 
>fast with the number of URLs (faster than linear).

True, true, that is the fault of my own filter, it works OK for a single
mbox as in my shellscript PoC, but you need to have some hashed dbs
to speed it up. My problem is that redoing it, is out of scope for me.

Terry have looked at alternate methods, but it seems that it have to be
exact matches to be really efficient.

>So, I think there shall be some number of tweaked URLs that can be used to 
>get some effectiveness with little maintainance - adding/removing few URLs 
>a week.
>
>After that threshold, the work necessary to increase effectiveness grows 
>vary fast, and you need to add/remove too many expressions a week.

I simply has a honeytrap, to which I send over the spams. It extracts
and build a new data file. I then have a cron job updating the data
file based on the reworked file from the honeypot. I.e. the manual
part is the remailing. I don't do statisticals on the existing URL's, so
I can't remove unused domains.

>So, IMO, it's better to use URLs to get some effectiveness and complete it 
>with a different approach.

I have today a second step, where my ISP's SpamAssasin filter labels
spams in subject. Spam Earnest did not caught, I then take and
send automaticlly to the honeypot.

This is the drawback of all content blacklists, new content can't be
caught automaticlly, which was Kee Hinckleys main critisism when I
presented Earnest last year. I still is to get to the finish line on that,
but while my ISPs SA presently letting through 5 of 35 spams a day,
I am now back to 1-2 a day.

Regards Kurt

_________________________________________________________________
MSN 8 with e-mail virus protection service: 2 months FREE* 
http://join.msn.com/?page=features/virus