Re: Some thoughts from a non-ASK user

Hi Brian,

On Tue, Feb 10, 2004 at 12:31:20PM +0000, Brian Candler wrote:

Here's a few comments. Sorry for snipping your original message. Here it
goes...

<snip>

> So ideally, mail which arrives from a non-whitelist source should be
> delivered into a Queue folder, and then automatically moved to Inbox when
> the sender returns their confirmation mail. This would allow users to check
> the Queue from time to time using IMAP or Webmail, and manually retrieve
> messages which they wanted. Messages which remain in the Queue for over 14
> days, say, can be purged automatically.

If your IMAP server supports Maildir, you can configure ASK to save the mail
queue in Maildir format. I use it daily with offlineimap to sync my IMAP
spools. Works beautifully and allows you to quickly delete spam and move the
good messages (if any) to your inbox.

> 2. The system uses Maildir for NFS-stored mailboxes, and must be NFS-safe.
> (I'm using courier-imap and sqwebmail, but there are other similar
> solutions)

Yes, ASK also supports Maildir. Maildir over NFS should not be a problem, as
Maildir does not require locking.

> This means that any updating of the whitelist/blacklist file must be done in
> an NFS-safe way: e.g. using lockfiles.

ASK currently uses flock to lock the lists before updates. If it becomes an
issue with NFS, you can also specify an "external" file to be used as the
lockfile. This file can reside inside your native filesystem.

> Also, the message itself may reside in .Queue/new/somelongfilename or
> .Queue/cur/somelongfilename:flags [if the user viewed it manually before the
> confirmation mail was received], so upon receipt of the confirmation the
> message must be found in either of these places. But then actually moving
> it into the Inbox is a trivial operation.

Currently ASK only looks in the "new" directory. I have plans to modify
it so it'll look in the "cur" directory as well. It's a simple change.

> 3. The system obviously has to update the whitelist itself when receiving a
> valid confirmation mail, but there also needs to be some way for users to
> manage it themselves - a web interface probably.

I agree. There are some for ASK. I did not write them, however.

> Ideally, if someone manually moves a message out of the Queue (but not into
> Trash) then that would whitelist the address too. That needs some mods to
> the imap/webmail server of course. (For symmetry, I suppose you could have a
> Junk folder, and moving a message in there would blacklist the address?
> Actually that's probably not a good idea, since the address is probably
> forged anyway.)

Requiring a modified IMAP server would be a major pain. It would be good if,
say, ASK could scan a designated mailbox for messages to be added to the
whitelist. Naturally, this is expen$ive when it comes to CPU cycles.

> 4. The whitelist/blacklist file should not grow without bounds. I think each
> entry should have an LRU timestamp, to allow the size of the list to be
> bounded and old addresses to be discarded if necessary. This means updating
> the whitelist/blacklist file on every message delivery, to update the
> timestamp against the address.

But how do you decide what's old? I have people that sent me undesirable
messages a long time ago. And I still don't want to receive messages
from them. :)

> 5. I want my MTA (exim) to be able to query the whitelist/blacklist in real
> time. An efficient way to do that would be to have a manager daemon which
> accepts queries over a socket. This saves having to fork off a process from
> exim for every recipient, or link in Perl etc.

I try to avoid that as much as I can. At first, it seems a simple task. but
then, someones needs RBL and there you go code RBL code in your front-end
smtp daemon. Then someone else wants virus checking and there you go... When
you realize, you're coding another MTA. :) I *think* this can be done in
Exim with filters. I'm not sure though.

> 6. Handling bounces properly is I think the hardest part of this. Personally
> I get 100-200 spams per day, and a large chunk is bounces to viruses/worms
> which have my address forged as the sender.

<snip>

The only way to properly handle bounces is to change the Envelope-From on
every sent message to include a token that can be verified upon receipt. This
token needs to "expire" in, say, seven days from the day it was sent. Bounces
coming to any email address that does not contain the token are automatically
discarded.

The problem here is the added complexity. Again, we fall into the original
problem of how to intercept SMTP connections and keep it simple. Perhaps
something like MimeDefang, which moves emails from the "incoming" queue into
the "outbound" queue. This, however, requires significant knowledge during
installation and the ability to receive all emails for a given domain. I
plan on adding this to ASK, but unfortunately it is not a viable solution to
everyone.

> The only way I can see to distinguish these is to add some sort of header
> with a special string on outgoing mails, and check that the bounce includes
> this special string somewhere. (This relies on the bouncing MTA quoting at
> least the headers of the message it is bouncing)
> 
> e.g.
> 
> X-Bounce-Auth: 1076414938, cb739acd0034a89f
> 
> When receiving a mail with a null sender, you can then check for this,
> validate the timestamp and crypto checksum, and allow the bounce through.
> All others just get put into the Trash folder.

That's almost what ASK does with the mailkey. Unfortunately, not all
MTAs keep the original headers.

> Achieving this relies on users sending their outgoing mails via the ISP's
> smarthost, which in turn is configured to add this magic header.
> 
> However I think the only way that ASK achieves this is by relying on you
> putting a magic string ('mailkey') in your mail signature, and allowing
> through the mail if it includes that. This has the problems that:
> 
> a. Our clueless ISP users must be told to configure their mail client with a
> signature, AND configure their mailbox on the server to know what the magic
> string is (our users won't do that, and if they get it wrong then all their
> bounces will be lost)

Agree. Not only that, but messages coming from themselves to themselves
will not be delivered, which causes confusion. You can use a few programs
to forcefully add a header to outgoing messages. This may or may not be an
option in your case.

> Well, that's about it. Any comments or suggestions? Other projects I should
> be aware of? Is modifying ASK to meet my needs going to be worthwhile, or
> should I just start a new project from scratch?

I don't think there's a lot to be changed. And Python is very easy to
understand and modify...

> P.S. I suppose I should add that personally I don't like the fact that ASK
> and TMDA are written in Python, just because it's more software to install
> on a mail system. C would be my preference, for both portability and speed.

One is fast. The other is flexible... Choices, choices... :)

Regards,
Paga

-- 
Marco Paganini          | UNIX / Linux / Networking
[email protected]   | PGP: http://www.paganini.net/pgp/
http://www.paganini.net | Magnus Frater te spectat...


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click