Re: How getmail deals with delivery errors and with bounce e-mails

Langenxx Feld <[email protected]> wrote:
> 
> > 1. Many POP/IMAP mail providers use MTA/MDAs configured incorrectly or
> > [...]
> 
> I think I can rule out this kind of configuration problems. I have selected
> one random provider for test purposes, and I have set up a "catch all"
> address there. All e-mails get an extra header named X-Original-To injected

That's great; as I said, if your mail services provider gives you a proper
domain mailbox that records the envelope addresses correctly, you're in great
shape to do a lot of stuff that is impossible otherwise, like automatically
delivering mail to different local mailboxes, or even different local users,
using the %(recipient), %(local), or %(domain) substitutions.

> with the original envelope recipient address. This seems to be standard
> practice with Postfix, and it matches what other people report. Without such
> a header, a catch-all mailbox is next to useless, so I guess that most
> providers get this right.

*slaps knee* Oh, you kidder!

Seriously though, you're trying to think logically about this.  That won't
work.  Of course every provider of domain mailbox service *should* do this,
but internet email architecture goes back just about *50 years*, to when it
was still called the Arpanet.  Hands up everyone else who had a bang-path
email address decades ago?  I did.

There are lots...  many...  multitudes, even, of email systems out there that
are screwed-up in any possible way you can imagine.  It's like Rule 34 for
email; any possible illogical, function-impairing, interoperability-breaking
software configuration you can imagine, there are mail servers on the internet
that do it.

Mail services providers advertising domain mailboxes, but which service does
not actually record the envelope sender address into a new message header, are
HUGELY common.  That's why there's code in getmail specifically to detect when
a getmail user tries to use a domain mailbox retriever class with such a
service, to prevent delivering mail to the wrong address.

> If the provider's MTA+MDA is misconfigured, you can always migrate to a
> different provider.

Humans being what they are, you would be surprised how often I hear "no, I
can't/won't" even when I've just finished telling someone that all their
problems are caused by their lousy mail services provider, and that their
problems will magically go away if they just migrate to another provider.

I have literally had this conversation hundreds of times.  The number of times
people said "huh, I didn't realize they sucked, I'll change providers" is ...
maybe 3?  4?

> I do not have to write a script to cater for all providers. When I select a
> provider, I can enable the catch-all mailbox, send a couple of test e-mails
> and check the headers.

Not quite.  Like I said, maybe your provider already has this completely
right, but "just send a couple test emails" does NOT excercise the corner
cases that oh-so-many service providers fail to handle correctly.

To be positive, you have to test all regular mail functions and confirm that
the envelope sender and envelope recipient addresses are correctly recorded in
the message headers -- correctly meaning "sender/recipient always recorded in
a header field with a specified name, and which is always the topmost (or
2nd-topmost, or whatever, it just has to be static) for every message
regardless of other aspects of the mail".

Some corner cases you want to test when judging a provider's "domain mailbox"
service:

* send messages to it with the null envelope sender.  Some "domain mailboxes"
  will do one of the following:
  * fail to record the envelope sender at all
  * will record something other than the envelope sender address - frequently
    an address from some other mail header field - into the header you expect
    the envelope sender address to be in
  * record it correctly ("Return-path: <>" or similar), but further down in
    the message headers than normal, with or without another incorrect one 
    further up
  * record it correctly, but in a header field with a different name than
    usual.

* send messages from domains that use, and advertise the use of, SPF, DKIM,
  and other anti-UCE features.  Some providers will end up rewriting or 
  otherwise munging envelope sender addresses in these cases.

* send messages where the envelope sender address is valid but doesn't appear
  anywhere else in the message headers (not in From:, Reply-to:, etc).  Much
  mailing list mail is like this, and some providers do weird things.

* send messages where the envelope recipient address does not appear in the
  informational message headers (To:, Resent-to:, cc:, etc).  Again, mailing
  list mail and lots of other mass mail will be like this, and some providers
  do weird things, recording some incorrect value as the supposed envelope
  sender.

There's more, lots more.  You need to expect weirdness, and be able to deal
with it, or at least test for it in advance.

Note you probably need to have a good understanding of mail and access to a
commandline MTA to test the above; you won't be able to control some of these
aspects from a GUI or web-based mailer.  Writing tests with a scripting
language is great for this.  You'll need a good understanding of the types of
TXT etc DNS records used to advertise various domain capabilities.

> For similar reasons, I doubt that the Return-Path header would be wrong. I
> checked on my provider and it's fine.

You checked on your provider for the basic, simple case.  Your provider may be
fine - but have you really tested the edge cases?

Again, this is done incorrectly on many, many email service providers.

> >    e) you retrieve mail from imap.example.net and re-inject it into an MTA
> >       on your own mail-handling machine at other.domain.org.
> 
> I do not understand why I should re-inject mail into some internal MTA of
> mine. I am retrieving mail with getmail, and that would go to dovecot-lda
> (an MDA). There is no MTA involved at all.

Well, you talked about having dovecot-lda generate bounces.  That by
definition involves an MTA.  You're not using MTA injection as part of your
basic delivery setup; that's fine, my comment applies to more cases than just
yours.

> I also do not understand why I should use other.domain.org internally.

That was an example.  You can reread it with the domain MX as
"mail.example.com" and the machine you're running dovecot-lda on as
"internal-mail.example.com" if you want; the point is the same:  people are
watching their messages to you be accepted at the SMTP level by
mail.example.com on IP address 1.2.3.4, and are later receiving a bounce
message that originates from a different host on a different IP address, or
even completely different network.

And that's wrong, full stop.

> I purchased domain example.org , so I will be using it internally too. Why
> do you want to introduce other.domain.org in the equation?

You don't have to, by any means.  As others have noted, using your external
domain for hidden internal services as well can cause confusion and lots of
problems, so many people will use a different, non-public domain for internal
purposes.

But you don't have to; I don't.  I use split-horizon DNS and have internal
hostnames in my pyropus.ca domain that only resolve inside my local networks.

> >    g) sender receives a bounce message from a other.domain.org, on a
> >    completely different network than the MX they sent a message to, and
> >    says "wtf, this is junk"
> 
> That is not going to happen in that way. When my internal
> server/logic/script generates a bounce message, it will state that it comes
> from [email protected] .

No, please do not do this.  You aren't even clear above that you are talking
about the From: header address - you're aware bounce messages *must* have a
null envelope sender?

> I will have to relay it to my provider's outgoing (SMTP) server, because,
> due to SPF, it is in fact the only server allowed to send e-mails on my
> behalf. And that is fine by me.

Those aren't valid bounces.  Please don't try to say "well, but they'll look
the same as real bounces from the domain's MTA" or other such stuff.  Just
don't do it - I guarantee you will be causing problems for mail administrators
around the world, even if you're not aware of it.

Repeat after me:

  Bounce messages only at the MX.

  After retrieving messages from the MX via POP/IMAP, deliver or drop
  messages.  Do not try to generate these invalid bounce messages.

> That is what I still do not understand. [...]
> Or must the bounce e-mail come truly from exactly the MX server?

Yes - only from the MTA running on the MX.  Generating a bounce message
anywhere else further down the line is invalid, and WILL cause problems for
other mail administrators.

> Manual review is not a realistic option in my opinion.

Just drop the messages.  99.95% of people pay absolutely zero attention to
bounce messages; the vast majority of people never see them at all, because
they're sent to a spam/junk folder.

> I have better things to do than reading such e-mails.

Here's the critical point:  all the other email administrators on the net have
better things to do than try to reverse-engineer whatever screwed-up
configuration generated an invalid bounce message that ended up in their
to-review-and-diagnose pile.

> > The proper way is to generate local white- or black-lists of addresses -
> > this can usually be done automatically and periodically - and transfer
> > those to the upstream mail server so it can use them to accept or bounce
> > the messages.
> 
> That would be nice indeed, but is not realistic. Big providers do not listen
> to their customers and provide an API to upload such white- or black-lists
> for their mail servers.

And now, where I tell you there are providers out there who are fully capable
of doing this for you... are you going to do the work to figure out which one
you like, how to switch, and how to do this integration?

Or are you just going to say "no, I won't switch"?

:)

> > Also note that your example of a "vacation" response - or any other
> > autoresponder, as they're known generically - is *not* the same as a bounce
> > message.
> 
> That is an interesting subject. What is the real difference between a bounce
> message, and one that says "this is not a real bounce e-mail, according to
> some unwritten law, but you tried to send a message to [email protected], and
> that's not a valid address"?

autoresponders handle messages differently from bounce processing; they
generally have various logic to avoid a lot of common pitfalls with blindly
bouncing messages, like dealing with mailing list mail (for one example).
They also tend to do rate-limiting per address they send to.

If you want to read up to learn about the differences between autoresponders
and MTA bounce processing, I would suggest reading all of djb's site at
cr.yp.to, a whole ton of RFCs, a bunch of mail administration books, including
Dave Sill's "The qmail handbook" and the bat book and some Postfix books, the
entire archives of the qmail users' mailing list and the djbdns users' mailing
list, and J. de Boyne Pollard's FGAs, ... that would give you a start, and
pointers to about ten million words elsewhere that you should read.

Internet mail is *complicated*.  Charging in and saying "I don't understand
why everyone says don't do X, I'm just going to do X" makes life miserable for
all your fellow mail administrators.  Here's a secret: lots of mail
administrators, when they start receiving garbage messages because someone has
ignored best practices, will immediately killfile all email from the domain
where such message existed.  Voila, no more problem mail from *that* newbie
mail admin.

I've got several hundred domains killfiled that way, but I only run my own
personal mail.  I've known some mail admins that have more than a million
domains going straight to the round file.  If you start generating these
invalid, bogus bounce messages - well, you'll probably end up with at least
some of your outgoing mail going to /dev/null, and you'll never know it,
because they don't send bounce messages in response to invalid crap.

Sorry if this seems cruel to you.  I, and others here, have been trying to
educate you on best practices, but you don't seem willing to take that advice.
If you want complete, in-depth answers with full sourcing on the why behind
those best practices, you're going to have to do a lot (a *lot* - 50 years of
internet mail...) of research, or perhaps pay a consultant to write them for
you.  I and others on this list aren't going to put 50 hours into educating
you for free.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------