Re: SPF and bouncing

At 05:42 05/04/2012  Thursday, Michael Deutschmann wrote:
>On Thu, 5 Apr 2012, alan wrote:
>> you reject based on, spammyness, sender reputation, local policy,
>> content, spf-fail dkim-fail whatever ONLY
>
>Faced with an unbounceable incoming message about to cross the perimeter,

ok firstly "unbouncable"? i assume you mean "possibly non-existant address" as there is no way to determine this 

>you try *really, really hard* to find a good reason to reject it.

yes

>  But you often won't succeed, because a large proportion of *desired* mail has no
>SPF information.  (I see 72.9% year-to-date)

spf is only a tiny tiny factor in the accept/reject decision making process, more important is
IP>PTR>IP, helo parent domain=ptr parent domain, helo>ip, helo spf pass, helo csv pass, IP being on numerous blacklists, IP being on numerous whitelists, number of rcpt address's that failed, does the envelope-sender have an mx, does the envelope sender only have a A, then spamassasin score on data, whitelist/blacklist URLs in data, whitelist/blacklist From: in data, DKIM pass/fail, etc. etc.

>And of course that assumes it is reasonable to bounce SPF pass mail --
>which if you'll remember, was what I questioned in my original post.

again rejecting any mail is acceptable, bouncing (which is accepting and then later returning it to sender) is not ever acceptable, thus SPF isn't a factor in bounce/do not bounce decision, as bounce is never a valid option. 

(simply put receivers should never generate bounces, senders (who can react to the "dead letters" by checking who submitted the forged mail and tightening their own controls on "their" inputs (the cause of the issue) can/should/do generate bounces ONLY)

>> this is not recent thinking, this is more a reaction (backlash)
>
>Not in my experience.  Before spam boomed in the turn of the millenium, it
>was quite common for businesses to MX backup for each other, without
>sharing valid address lists.  That's a big backscatter magnet these days.

maybe back in the days when all ran open relays (thus you could make any of these a backup MX for yourself with/without permission simply by adding their servers to your MXs)

I will admit this may be before my time

>What changed was that some of those sloppy MXes started using "sender
>verify with callout" (itself an abusive practice, but that's another
>story) to protect themselves from deadletters piling up when they
>accepted a spam with a completely made up envelope sender.

no one used it as deadletter protection, (yes i drank that kool-aid for a while i was young/naive), all used it on the assumption that 90% of spam is forged a good 50% of that is non-existant thus early way to dump 50% without content checking

>  That also blocked the spammers from reaching users behind those MXes who actually
>existed, so the spammers responded by using real addresses from their
>"Millions CDs".

many used real forged address' even before sender-callout-verification, did this, most often viri would use the machines local address book and pick both from and to from same (based on the increased chance that they might know/trust one another and thus run the attachment)
but yes forged but existant envelope-sender was the beginning of the backscatter issue, firstly AV systems sending "notifiys/bounces" to the joe-jobbed address', later spammers doing bounce-spam* (just to clean the reputation of their IP)
(bounce-spam* == find a exchange box (little protection, default policy of accept all, then bounce (to stop address harvesting via which 5xx  and which dont they claim) then send spam from victim to known non-existant address, let the bad reciever create the bounce and send it to the victim (payload intact) from a good reputation IP

as people realised bounce-spam was as bad as open-relay if not worse (and started being blacklisted for both)
they re configured their exchanges to do reject at RCPT time.
then spammers started looking for alternatives and found the few complex multi-stage multi-policy relays to continue.

backscatter is admittedly only an issue because the sheer volume of invalid vs valid mail is staggering, 99% of connections here result in a reject at some point. vs the 1% that get accepted (and a good many are still spam)

but it is our job as good netizens to ensure we are not part of the issue and do not allow our systems to cause grief to others so once you have identified receiver side bounces are always an issue, you eliminate them.

this causes, a dramatic downscaling of bounces 
(as 60% of the connections inbound are from ratware (direct from infected pc to MX, thus no bounce generated (as the ratware knows it forged the address in envelope-from))
so thats 60% of bounce traffic gone immediatly
then of the 40% thats left it is up to the sender to generate a bounce to the user
(many of those will be ESPs and mailing-list-srvs and thus no-bounce just unsubscribes)
leaving 2 groups, 
well run senders who return the bounce/NDR to the validated user who appreciates the information about the undeliverable message
badly run senders who allow their users to forge envelope-from who will generate backscatter to innocent 3rd parties, who will in turn have "dead letter" queues , be complained to, be RBL'd and shunned/educated, till they hopefully eventually clean up their policys

this is the desired outcome at least, the issue is a sender side one, as receivers we should ensure that it remains a sender issue and not obscure or amplify the issue by generating our own NDRs to 3rd parties

and again SPF is simply a non-factor in any receiver generated NDR scenario as we should never generate them
(that said NDRs are generated if an "internal" mailbox server is down for extended period, but this is a vanishing small non statistically relevant number, and becoming less, but it is the one case where a well setup receiver can cause an issue for a forged from)

>> simply whitelist the domain from RCPT-reject, if from:(in data) differs
>> from whitelisted address then reject after data IF you would have
>> otherwise rejected at RCPT
>
>Looking at the RFC822 From: is a big can of worms.  You have to let the
>connection go to DATA, and work out how you'll deal with multi-RCPT mails.

multi rcpt mails are easy, i accept the first(valid) one (that passes the users spec thus far)
then defer(4xx) all others that would have passed but have different post data specs
thus post data accept/reject dosn't affect the other recipients still queued on the sender side

you have to let the connection go to data for DKIM, spamassasin, av scanning etc etc too,
dealing with multi RCPT mails has been no issue for any of those.