Re: Technical spam books, my quest for knowledge.

naughty jinx wrote:
> I read the book cover to cover and not only was happy with it, but even 
> learned a fair bit :)
> Anyone else suggest any good spam books?

In general, no.

I have been greatly disappointed in most books and articles about spam 
or computer security.  In most cases they totally ignore critical 
information that the reader needs to know.

In the case of computer security, they usually fail to take into account 
the low cost social engineering techniques that will usually defeat the 
most sophisticated technological approach that they are usually advocating.

In the case of spam filtering, most of what I have read in commercial 
publications is aimed at selling commercial spam filters, or more 
precisely, convincing the commercial spam filter vendors to advertise in 
that publication.

There are also a lot of publications about people describing their 
fantastic new content filtering algorithms, and these totally ignore 
obvious weaknesses in those algorithms, and even after these weaknesses 
have been taken advantage of by spammers, there are still proponents of 
them that totally ignore that.

Most of the commercial spam content filters that I have seen so far are 
simply ineffective placebo filters that are usually not effective unless 
used with other techniques that are so effective that the content filter 
is only looking at a small percentage of the incoming e-mail.

> I was after a book with detailed technical information used by
> spammers, not trival stuff like open proxy servers and formmail,
> because we all know how those work. I wanted to know more about the
> filter evasion side of things, how messages are constructed and what
> work goes into it.

Almost all content filtering methods are trivial for all but the dumbest 
spammers to avoid.

The main thing useful about studying them is if you are a legitimate 
newsletter sender or commercial mail sender so that you can know to 
avoid somethings that are sure to help trip an aggressive content filter.

For example, one of the key things that will help get a real e-mail to 
be misclassified by a content filter is for it to be sent in HTML to an 
e-mail address that did not expressly request HTML mailings to be sent.
The presence of HTML in an e-mail to some addresses indicates between a 
50% to 90% chance of it being spam.

And typically when a corporate mail server uses a content filter, the 
messages are silently deleted.  If outgoing mail causes whitelisting, a 
company can usually get away with running a very aggressive spam filter, 
with almost none of their users catching on.  After all SMTP e-mail has 
no guarantee of delivery or non delivery information, and if a message 
just disappears, most people can not determine at what point it was 
dropped or why.

Just about every content analysis method has proved that it will give 
you a choice of either leaking a significant portion of spam, or 
catching more than a noticeable amount of real e-mail by mistake if used 
on a mail server.  And sometimes you do not have a choice and it does both.

The larger the population that shares a content filter's algorithms, the 
less accurate the content filter will be.  Consider if you share a mail 
server with a legitimate mortgage broker and some medical doctors.
Right away you have eliminated most of the keywords used in spam, 
including their mis-spelled variants.

Besides the fact that content filtering is inherently inaccurate, what 
the articles covering it fail to point out is that it is expensive to 
implement.

The usual implementation of a content filter as a primary spam defense 
is by someone that wants be able to claim that the mail server has state 
of the art spam defenses, and hope that most of the users do not know 
better.  This is in a hope that the users will think they are doing all 
they can do and not complain.

For content filtering to be applied, the entire spam must be inside the 
mail server.

Now you may be on a fixed cost ISP link, or not be aware of your 
company's network costs, but be aware that once your network load gets 
large enough, you pay a rate based on how much data you pass on a 
network.  And with some plans, if you go over your bandwidth quota, the 
rates increase significantly.  If you have a T1 quota on a line that can 
handle a higher rate, and you go over the T1 quota, even by a small 
amount you can get charged for a second entire T1 quota for that month.

And the bandwidth bill for a large mail server can be significant.

Now the way to control that costs are to use DNS based blocking lists to 
prevent the spam from entering the mail server.

And the conservative DNSbls have shown that they are far more accurate 
at separating spam from the real e-mail than content filters.  Reports 
are that between 80% to 95% of spam blockage with zero false positives 
can be achieved with just the conservative DNSbls.

Now it is unlikely that you can come up with a spam filtering solution 
that is perfect.  You have to decide how close that you can come.

Now to handle errors in false positives, the spam filtering must be done 
before the SMTP transaction is over, as that is the only way to 
non-abusively notify a real sender that their message was not delivered, 
because it is their ISP's mail server that notifies them.

After the SMTP transaction is over, which is the time that most content 
filtering is done, there is no way to non-abusively notify a mail sender 
that the mail was not accepted.  The sending information is too easily 
forged, and sending non-delivery messages to suspected spam or viruses 
is having that mail server take part in a DDOS attack on some other 
victim of the spammer.

So a false positive from a content filter is far worse than a false 
positive from an DNSbl, as usually with a content filter, neither the 
sender or the receiver knows that the e-mail was not delivered.

It has also been shown if you monitor any of the help forums for DNS 
based blocking list that a large number of mail server operators or 
ISP's will not do anything about security problems on their networks as 
long as other ISP's will still accept packets from them.

It is routine to find mail server operators complaining about unfair 
blocking and insisting on the following:

      1. There is no spam coming from their network, the I.P. addresses 
in the reports are spoofed.

      2. They have configured a spam filter to remove all spam and virus 
complaints from their abuse mail box.

      3. In violation of the RFCs they do not have the required abuse 
and postmaster mailboxes, or they have configured them to automatically 
delete all incoming e-mail.

Now of the content filtering method, the only one that the spammers have 
not figured out how to easily bypass and still advertise their stuff is 
one that looks up the URLs or links in the spam, and checks them against 
the same DNSbls that the mail server refuses to accept e-mail from.

Even this can have false positives if you are receiving spam samples to 
assist in the decoding of spam.  But if you limit the content check to 
I.P. addresses in that are in aggressive DNSbls or have less than 
perfect rDNS, it is unlikely that you will find a false positive.

Since you can usually use the conservative DNSbls to safely remove 
between 80% and 95% of the spam, only a small amount of e-mail would 
actually need running through the content filter for a final check.

Currently SpamAssasin 3.0 appears to be the only content filter that 
knows how to do this test.  It does not appear to be available in any 
commercial content filter in spite of that algorithm being well known 
for over a year.

-John
[email protected]
Personal Opinion Only