Re: Some thoughts from a non-ASK user

On Wed, Feb 11, 2004 at 06:14:45PM -0500, Marco Paganini wrote:
> If your IMAP server supports Maildir, you can configure ASK to save the mail
> queue in Maildir format. I use it daily with offlineimap to sync my IMAP
> spools. Works beautifully and allows you to quickly delete spam and move the
> good messages (if any) to your inbox.

Nice. When it receives the confirmation mail, does it then automatically
move the mail into a different Maildir (i.e. the Inbox one)?

> > This means that any updating of the whitelist/blacklist file must be done in
> > an NFS-safe way: e.g. using lockfiles.
> 
> ASK currently uses flock to lock the lists before updates. If it becomes an
> issue with NFS, you can also specify an "external" file to be used as the
> lockfile. This file can reside inside your native filesystem.

That's not possible, because there are multiple front-end boxes all talking
to the same NFS server for resilience and load sharing (i.e. a couple of MX
boxes running Exim to receive mail, several POP/IMAP boxes, several webmail
boxes etc).

So a local file with flock is no good; you'd have to create a dot-lock file
within the Maildir itself. Or else update the file in another relatively
NFS-safe way; for example, have fixed-size lines (40 bytes say), and then
append 40 bytes at a time. When the file gets above a certain size,
atomically move a new file into place. This risks occasional loss of
entries, but that may be acceptable - certainly more acceptable than a
corrupted file, anyway.

(Maildir++ uses this sort of mechanism for its 'maildirsize' file, which
keeps track of the quota usage)

> > 4. The whitelist/blacklist file should not grow without bounds. I think each
> > entry should have an LRU timestamp, to allow the size of the list to be
> > bounded and old addresses to be discarded if necessary. This means updating
> > the whitelist/blacklist file on every message delivery, to update the
> > timestamp against the address.
> 
> But how do you decide what's old? I have people that sent me undesirable
> messages a long time ago. And I still don't want to receive messages
> from them. :)

"Old" = "the oldest". You set a maximum size of the blacklist/whitelist,
let's say 1,000 or 10,000 entries. For most people that's perfectly fine.
The limit is really just there to prevent a denial-of-service by someone who
decides to fill their whitelist with 10GB of crap. In an ISP with hundreds
of thousands of users, there are always a few people who are either
extremely smart or extremely stupid and push the boundaries...

> > 5. I want my MTA (exim) to be able to query the whitelist/blacklist in real
> > time. An efficient way to do that would be to have a manager daemon which
> > accepts queries over a socket. This saves having to fork off a process from
> > exim for every recipient, or link in Perl etc.
> 
> I try to avoid that as much as I can. At first, it seems a simple task. but
> then, someones needs RBL and there you go code RBL code in your front-end
> smtp daemon. Then someone else wants virus checking and there you go... When
> you realize, you're coding another MTA. :) I *think* this can be done in
> Exim with filters. I'm not sure though.

All of it can be done in Exim no problem. But I definitely want to put in
there that Exim will query the blacklist at RCPT TO time (to bounce mails
with 550), and update the whitelist LRU timestamp. To do that, either:
1. exim has to run an external program which contains the logic for safely
   querying and updating the blacklist/whitelist file, as discussed above;
2. exim runs an embedded Perl program which performs the necessary
   operations;
3. exim can send a message down a socket to a daemon which does the request
   and sends a reply

(1) has the disadvantage of requiring a fork and exec for each recipient,
which is why the daemon approach is of interest to me. (2) requires ASK to
be rewritten in Perl. And I don't want to modify the Exim source code
itself.

> The only way to properly handle bounces is to change the Envelope-From on
> every sent message to include a token that can be verified upon receipt.

Ah yes, the so-called VERP (Variable Envelope Return Path) approach; I think
it's what TMDA does. I considered that, although that really does force all
our users to go through our smarthosts.

> That's almost what ASK does with the mailkey. Unfortunately, not all
> MTAs keep the original headers.

Yeah I know. But I'd be happy to lose bounces from, say, Microsoft Exchange
and Lotus Notes servers :-) [Actually I wouldn't lose them, but they'd go
into the Queue or Trash with no notification]

Incidentally, I have just enabled pobox.com's latest spam filtering, which
is mostly based on RBL lists (country lists, spamcop, spamhaus etc). It
works really well, but of course I still get large amounts of virus and spam
*bounces* because the sources of those messages are not RBL-listed. So this
is really the biggest hurdle left for me to overcome.

> Agree. Not only that, but messages coming from themselves to themselves
> will not be delivered, which causes confusion. You can use a few programs
> to forcefully add a header to outgoing messages. This may or may not be an
> option in your case.

We have control of the smarthosts, and configuring Exim to add headers is
straightforward. I guess if we do that then we could equally implement the
VERP approach.

Regards,

Brian.


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click