Some thoughts from a non-ASK user

Hello there,

I'm looking to implement a server-side spam killing solution, along the
lines of ASK/TMDA etc. I don't think ASK implements things the way I need,
but I have some ideas I'd like to raise here and see what people think.

1. This is for a closed virtual-mailbox system, i.e. users do not have home
directories or shell access. Users retrieve mail using POP3/IMAP/Webmail
only.

So ideally, mail which arrives from a non-whitelist source should be
delivered into a Queue folder, and then automatically moved to Inbox when
the sender returns their confirmation mail. This would allow users to check
the Queue from time to time using IMAP or Webmail, and manually retrieve
messages which they wanted. Messages which remain in the Queue for over 14
days, say, can be purged automatically.

2. The system uses Maildir for NFS-stored mailboxes, and must be NFS-safe.
(I'm using courier-imap and sqwebmail, but there are other similar
solutions)

This means that any updating of the whitelist/blacklist file must be done in
an NFS-safe way: e.g. using lockfiles.

Also, the message itself may reside in .Queue/new/somelongfilename or
.Queue/cur/somelongfilename:flags [if the user viewed it manually before the
confirmation mail was received], so upon receipt of the confirmation the
message must be found in either of these places. But then actually moving
it into the Inbox is a trivial operation.

3. The system obviously has to update the whitelist itself when receiving a
valid confirmation mail, but there also needs to be some way for users to
manage it themselves - a web interface probably.

Ideally, if someone manually moves a message out of the Queue (but not into
Trash) then that would whitelist the address too. That needs some mods to
the imap/webmail server of course. (For symmetry, I suppose you could have a
Junk folder, and moving a message in there would blacklist the address?
Actually that's probably not a good idea, since the address is probably
forged anyway.)

4. The whitelist/blacklist file should not grow without bounds. I think each
entry should have an LRU timestamp, to allow the size of the list to be
bounded and old addresses to be discarded if necessary. This means updating
the whitelist/blacklist file on every message delivery, to update the
timestamp against the address.

5. I want my MTA (exim) to be able to query the whitelist/blacklist in real
time. An efficient way to do that would be to have a manager daemon which
accepts queries over a socket. This saves having to fork off a process from
exim for every recipient, or link in Perl etc.

6. Handling bounces properly is I think the hardest part of this. Personally
I get 100-200 spams per day, and a large chunk is bounces to viruses/worms
which have my address forged as the sender.

I have thought about this quite a bit, and the best solution I can think of
is outlined here.

- 'Bounces' are defined as all messages with SMTP envelope MAIL FROM: <>
(which is Return-Path: <> by the time the message reaches a mailbox)

- Now, in addition to valid bounces to our messages, we may receive bounces
to messages which forged MAIL FROM:<[email protected]>, and we may also
receive direct spams with an empty return path.

The only way I can see to distinguish these is to add some sort of header
with a special string on outgoing mails, and check that the bounce includes
this special string somewhere. (This relies on the bouncing MTA quoting at
least the headers of the message it is bouncing)

e.g.

X-Bounce-Auth: 1076414938, cb739acd0034a89f

When receiving a mail with a null sender, you can then check for this,
validate the timestamp and crypto checksum, and allow the bounce through.
All others just get put into the Trash folder.

Achieving this relies on users sending their outgoing mails via the ISP's
smarthost, which in turn is configured to add this magic header.

However I think the only way that ASK achieves this is by relying on you
putting a magic string ('mailkey') in your mail signature, and allowing
through the mail if it includes that. This has the problems that:

a. Our clueless ISP users must be told to configure their mail client with a
signature, AND configure their mailbox on the server to know what the magic
string is (our users won't do that, and if they get it wrong then all their
bounces will be lost)

b. It relies on the bouncing MTA quoting the entire message, up to and
including the signature.

Actually, I notice on the ASK website:

       "... ASK will by default
       deliver mail coming from MAILER-DAEMON, unless it is an invalid
       email response to a confirmation message. If a spammer uses  
       MAILER-DAEMON in the "From:" field, he could defeat ASK's
       protection. If that becomes an issue, future versions of the  
       program will "Junk" all mail coming from MAILER-DAEMON, unless it
       also contains your mailkey (normally Mail Transfer Agents append
       the original message to the error message, thus triggering the
       mailkey mechanism)."

So presumably this means ASK checks the From: header rather than the return
path. This is very weak I believe. Lots of MTA's give different From:
addresses for bounces (e.g. admin@, postmaster@, mmdf@), and I think the
only way to safely check is the SMTP envelope return path being null.
Furthermore, as noted, it's easy for spammers to get round; but more
crucially, all those bounces I get to viruses and worms *are* indeed valid
bounces - but not to messages that I genuinely sent. So ASK in its current
form would not help me filter against these. Hence I think the mailkey is
the way to go, and automating this mechanism by adding a
cryptographically-signed mailkey header to outgoing messages would make life
easier for end-users.

Well, that's about it. Any comments or suggestions? Other projects I should
be aware of? Is modifying ASK to meet my needs going to be worthwhile, or
should I just start a new project from scratch?

Regards,

Brian Candler.

P.S. I suppose I should add that personally I don't like the fact that ASK
and TMDA are written in Python, just because it's more software to install
on a mail system. C would be my preference, for both portability and speed.


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn