Re: Some thoughts from a non-ASK user

Hi Brian,

On Thu, Feb 12, 2004 at 12:49:50PM +0000, Brian Candler wrote:

>> If your IMAP server supports Maildir, you can configure ASK to save the mail
>> queue in Maildir format. I use it daily with offlineimap to sync my IMAP
>> spools. Works beautifully and allows you to quickly delete spam and move the
>> good messages (if any) to your inbox.
> 
> Nice. When it receives the confirmation mail, does it then automatically
> move the mail into a different Maildir (i.e. the Inbox one)?

Oh yes. ASK would be useless if it didn't do this.

> > > This means that any updating of the whitelist/blacklist file must be done in
> > > an NFS-safe way: e.g. using lockfiles.
> > 
> > ASK currently uses flock to lock the lists before updates. If it becomes an
> > issue with NFS, you can also specify an "external" file to be used as the
> > lockfile. This file can reside inside your native filesystem.
> 
> That's not possible, because there are multiple front-end boxes all talking
> to the same NFS server for resilience and load sharing (i.e. a couple of MX
> boxes running Exim to receive mail, several POP/IMAP boxes, several webmail
> boxes etc).

Ah, I understand. I'm changing ASK to use dotlock style locking as well.

> So a local file with flock is no good; you'd have to create a dot-lock file
> within the Maildir itself. Or else update the file in another relatively
> NFS-safe way; for example, have fixed-size lines (40 bytes say), and then
> append 40 bytes at a time. When the file gets above a certain size,
> atomically move a new file into place. This risks occasional loss of
> entries, but that may be acceptable - certainly more acceptable than a
> corrupted file, anyway.

I'm a little confused. I don't think locking is such a big issue with
Maildir as it is with mbox. Maildir uses unique filenames and thus it's
unlikely that we'll see locking issues. Am I missing something? What do
current IMAP servers use? dotlock? fcntl? Both?

> > > 4. The whitelist/blacklist file should not grow without bounds. I think each
> > > entry should have an LRU timestamp, to allow the size of the list to be
> > > bounded and old addresses to be discarded if necessary. This means updating
> > > the whitelist/blacklist file on every message delivery, to update the
> > > timestamp against the address.
> > 
> > But how do you decide what's old? I have people that sent me undesirable
> > messages a long time ago. And I still don't want to receive messages
> > from them. :)
> 
> "Old" = "the oldest". You set a maximum size of the blacklist/whitelist,
> let's say 1,000 or 10,000 entries. For most people that's perfectly fine.
> The limit is really just there to prevent a denial-of-service by someone who
> decides to fill their whitelist with 10GB of crap. In an ISP with hundreds
> of thousands of users, there are always a few people who are either
> extremely smart or extremely stupid and push the boundaries...

Hmmm... Interesting... But I have to admit you're the first one to
suggest this. :)

> All of it can be done in Exim no problem. But I definitely want to put in
> there that Exim will query the blacklist at RCPT TO time (to bounce mails
> with 550), and update the whitelist LRU timestamp. To do that, either:
> 1. exim has to run an external program which contains the logic for safely
>    querying and updating the blacklist/whitelist file, as discussed above;
> 2. exim runs an embedded Perl program which performs the necessary
>    operations;
> 3. exim can send a message down a socket to a daemon which does the request
>    and sends a reply
> 
> (1) has the disadvantage of requiring a fork and exec for each recipient,
> which is why the daemon approach is of interest to me. (2) requires ASK to
> be rewritten in Perl. And I don't want to modify the Exim source code
> itself.

Do you think the extra fork of (1) would be a big deal? It seems the
easiest option to do what you need.

>> The only way to properly handle bounces is to change the Envelope-From on
>> every sent message to include a token that can be verified upon receipt.
> 
> Ah yes, the so-called VERP (Variable Envelope Return Path) approach; I think
> it's what TMDA does. I considered that, although that really does force all
> our users to go through our smarthosts.

And brings some problems of its own. Some anti-spam filters will freak
out with VERPs and send the confirmation to /dev/null.

> > That's almost what ASK does with the mailkey. Unfortunately, not all
> > MTAs keep the original headers.
> 
> Yeah I know. But I'd be happy to lose bounces from, say, Microsoft Exchange
> and Lotus Notes servers :-) [Actually I wouldn't lose them, but they'd go
> into the Queue or Trash with no notification]
> 
> Incidentally, I have just enabled pobox.com's latest spam filtering, which
> is mostly based on RBL lists (country lists, spamcop, spamhaus etc). It
> works really well, but of course I still get large amounts of virus and spam
> *bounces* because the sources of those messages are not RBL-listed. So this
> is really the biggest hurdle left for me to overcome.

Life would be so easy if we had an SMTP cookie of sorts. I don't believe
there's a "proper" way of matching bounces, except by using VERPs. But
as we discussed, they bring other problems of their own...

> > Agree. Not only that, but messages coming from themselves to themselves
> > will not be delivered, which causes confusion. You can use a few programs
> > to forcefully add a header to outgoing messages. This may or may not be an
> > option in your case.
> 
> We have control of the smarthosts, and configuring Exim to add headers is
> straightforward. I guess if we do that then we could equally implement the
> VERP approach.

You can do a simplistic trick. Have your envelope change every month for all
users. Say, append the month+year to it. Reject any bounces older than two
months. It's ugly, but it works.

Regards,
Paga

> 
> Regards,
> 
> Brian.
> 
> 
> -------------------------------------------------------
> SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> Build and deploy apps & Web services for Linux with
> a free DVD software kit from IBM. Click Now!
> http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> _______________________________________________
> A-S-K-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/a-s-k-users
> 

-- 
Marco Paganini          | UNIX / Linux / Networking
[email protected]   | PGP: http://www.paganini.net/pgp/
http://www.paganini.net | Magnus Frater te spectat...


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click