Body filtering and delays

Kurt mentioned:

#The limitation in my method is just that, I need to have the domain in my 
#database, before it works, but how do I catch the first one, if the ISPs 
#filter doesn't. I'll think some other methods have the same problem. When 
#reading the papers in link Yakov sent, I got a possible attack method, the  
#question is how to utilize it without having every MTA changed? For it need 
#to be on the MTA level.

One thing I've long believed is that we need to get past the notion that
once articles getspooled to a user's mailbox, whether they're spam or not, 
those articles are untouchable. Bad articles need to be de-deliverable
(deleteable or quarantineable or able to be moved to a likely-to-be-spam
folder, whatever). 

Consider Usenet News article spools: if problematic content is received,
it may be filtered up front, much as mail currently is, or it may be deleted 
post hoc via cancel control messages or cryptographically signed NoCeM 
messages. I believe the next step for mail filtering will be to also apply 
post hoc filtering to mailboxes. 

Doing this efficiently will require creation and maintenance of a structure
similar to Usenet overviews files, not containing *everything* about each
message, but probably including at least:

-- message ID
-- mailbox filespec
-- Return-Path:
-- From:
-- To:
-- Subject:
-- Date:
-- Received: header host chain
-- attachment names and sizes (or some sort of attachment fingerprint)
-- body URLs

Why would one want a structure of this sort? Well, consider the following 
scenario:

-- spammer starts hammering your users with spam

-- there is some delta between the time the spamming starts and the time
   the spamming is identified, during which time some spam *does* get
   delivered

-- after the spamming is identified, future spam from that source gets
   blocked, but what of the spam that's already come in? Currently it is 
   allowed to stay in the user's mailbox.

-- I suggest:

   once you know that mail from 123.45.67.89 is mail from an open proxy or
   spam zombie (for example), you should be able to hit the "mail overview" 
   database and find all the *other* recent mail from that same dotted quad 
   and then retroactively "de-deliver" ALL of the messages from that spam 
   run which made it in to your user's mailboxes. 

   the keys to this working are:

   -- disk has to be cheap, and cpu's have to be fast, so as to allow the
      required data to be stripped from the article and crammed into a mail
      overviews database (I'd argue that if we can do it for current Usenet
      news volumes, we can do it for the far smaller volumes of SMTP traffic)

   -- there needs to be a delta between the time mail is received and the 
      time users actually read it (the degenerate case here is the POP user
      who downloads new messages ever minute; the best case here is spam
      delivery during the middle of the night, when in most cases you'll
      have hours to clean things up)

   -- messages remain in a host mailbox for a period of time (again, the
      degenerate case is the POP user who downloads messages on a frenetic
      basis; the best case is the command line mail user)

   -- system administrators who are currently comfortable blocking spam 
      need to make the leap to being comfortable de-delivering spam that
      has actually been stuck in a user's inbox; for some reason, I believe
      most users will feel differently about the two cases. 

   -- you may (or may not) want to "exempt" read messages from cleanup.

#Some have therefore looked at spam in the sender end, by delaying large 
#volume transfers, to make the spammers loose time. And as some in the 
#analyze-group did look at, you can view it also on the receiving side. Those 
#that worked with it, did look though at the connection state, the number of 
#active connections - a lot, spam.

No need to delay accepting message when "accepting" a message is a revocable
decision. If there's essentially zero cost to accepting a message and then
subsequently purging it, you actually WANT to accept the messages as fast
as possible because that will give you a greater volume of messages to use
for classification.

#In order to kill such a filter, the spammers need 10-20 domains for every 
#send-out, see to that routing is done through a number of places, before 
#arriving to the same target domain, so there is no more that 3-4 in queue at 
#the same time. Then it really will cost infrastructure for them and if they 
#spread them out, the chance of a domain to be reported gets bigger and can 
#be effectively blocked in the normal spam system before there is a volume. 
#And the spammers get a lesser ROI.

Domains are cheap; we're routinely seeing spammers using dozens per spam 
run, and I see no reason why they wouldn't increase the number of domains
if need be. That's okay, however; those domains all have to live somewhere,
and if you track both the FQDN and the dotted quad and the ASN routing that
dotted quad, life lived under a rock becomes significantly harder.

Regards,

Joe