Re: Microsoft takes over British Telecom

Paul Smith wrote, On 10/23/11 8:19 PM:
> On 23/10/2011 22:43, Steve Atkins wrote:
>> On Oct 23, 2011, at 2:16 PM, Paul Smith wrote:
>>
>>> On 23/10/2011 17:09, Steve Atkins wrote:
>>>> On Oct 23, 2011, at 7:39 AM, Alessandro Vesely wrote:
>>>>
>>>>> On 21/Oct/11 19:07, Steve Atkins wrote:
>>>>>> The bigger issue is that you shouldn't care about SPF failing. An SPF
>>>>>> pass
>>>>>> provides somewhat useful data, an SPF fail means absolutely nothing.
>>>>> Wasn't it supposed to mean "reject"? IIRC, that was the difference
>>>>> between setting -all (as irtf.org does) rather than ~all or ?all.
>>>> That's what it's proponents used to claim, yes. But it's unfixably
>>>> broken for
>>>> that purpose. The only reason you don't see serious mail problems for
>>>> anyone unwise enough to use -all is that almost nobody pays much attention
>>>> to SPF for rejecting email.
>>>>
>>> How is it 'unfixably' broken? Just interested.
>>>
>>> it seems to me that the only reason it's 'unfixably broken' is that
>>> people are using it without really understanding what it's doing. If all
>>> MTAs rejected all mail that failed SPF checks, then they'd soon get the
>>> problems fixed. Remote users having to relay via a specified submission
>>> server is a trivial problem to fix (AFAICS), and incorrectly specified
>>> SPF records (as in the BT/MS case) deserve to cause messages to be
>>> blocked (IMHO)
>>>
>>> (SPF is certainly not that useful as an 'anti-spam' mechanism, but it
>>> seems to have the potential to work reasonably well as an anti-spoofing
>>> mechanism, which makes it considerably harder for many spammers)
>> Mail forwarding is one common cause of failure. If you're sending mail to
>> someone at @ieee.org and they forward it on to their @aol.com account, and
>> you and AOL decide between you to enforce SPF -all then that mail will be
>> undeliverable - and there's no way either you or AOL can predict that.
> I thought most people did return-path rewriting for this case. If you don't
> do that, then you get undesirable behaviour anyway - eg if the AOL mailbox
> is full, you get a bounce message from AOL, even though you didn't send any
> messages to AOL at all.

Yes, that happens. No, return-path rewriting is the exception, not the rule. 
People still use Sendmail aliases and ~/.forward mechanisms and functional 
work-alikes in other MTA's and it breaks the utility of explicitly 
derogatory SPF, both "-" and "~"

However, as often as the classical forwarding model is brought up as an 
example of SPF breakage, it really isn't the critical failure mode for 
simplistic SPF usage, since classical forwarding is mostly a tool of people 
who remember the days before the Eternal September. We're an influential 
bunch, yes, but there are not enough of us to break something that would 
otherwise work.

>> Mailing lists is another similar cause of SPF failures..
> Similarly, I thought return-path rewriting was standard for mailing lists.
> Especially ones which do VERP, but usually even in ones which don't - eg the
> return path on the message I received from you was
> "<[email protected]>", not your blighty.com address. Therefore, it
> correctly passed an SPF test for irtf.org.

Yes, any mailing lists using reasonable list management tools rewrite 
messages with their own return-path and cause no trouble for SPF. There are 
some things called mailing lists that use 'exploder' features of various 
MTA's which don't use their own return-paths or any distinct software, but 
those sorts of lists were basically dysfunctional before spam was a 
significant problem. Like classical forwarding, that is also not the hardest 
obstacle for clean and simple use of derogatory SPF results.


>> A third, which is a lot more common and complex to resolve than you'd
>> expect, is organizations simply not knowing or not controlling where their
>> mail is sent from. There've been cases both for SPF and ADSP ("SPF for
>> DKIM") where one group within a company put out a restrictive policy based
>> on their beliefs of where mail was sent from, and it all fell to pieces
>> when employees got kicked off mailing lists when mail appearing to come
>> from them was delivered from the mailing list manager and rejected due to
>> auth failure, or where internal monitoring email sent from servers (cron,
>> even) was silently discarded.
> This is the problem of people who should know better not understanding
> what's going on.

Yes. That is why this problem is the truly hard one for application of SPF 
in derogatory ways.

Deprecating technical mechanisms via "standards" (to whatever degree a 
low-grade RFC can be called that) and real-world bad outcomes has a 
successful track record of working to extinguish their use or encourage 
workarounds.

Deprecating human carelessness and ignorance has no successful track record.

NO SUCCESSFUL TRACK RECORD

In other words: THIS DOES NOT WORK!

> If the company had a policy that all mail from their domain
> had to go through certain servers, and employees were sending mail through
> other servers, then they deserved to get kicked off mailing lists... Just as
> they would deserve punishment for watching porn over the work Internet
> connection, etc. If they had a policy that mail had to go through certain
> servers, and configured SPF for different servers, then they shouldn't be
> managing mail systems.

That's nice. The fact is that sysadmins follow Sturgeon's Law just like any 
other large population. I'm very sorry about that. I really am.

Beyond that sad fact of life, there are other issues with "-all" (and even 
"~all") grounded in forms of dumb that are external to the administrators of 
any domain setting SPF records. For example, diverse online service 
providers including resume search systems, news clipping agencies, magazine 
publishers, and so on (and on and on) will send email on behalf of 
registered users with any envelope sender address that they have for the 
user. Whether it is a "mail a friend this article" link at the NYT or a 
"find me candidates for this job" form at a headhunter agency, there are a 
lot of ways that random 3rd parties can end up sending mail claiming to be 
from a user of a domain whose SPF maintainers cannot predict them. I don't 
like that, I think it is wrong and bad. However, it is my direct experience 
that fighting that practice is a consistently losing battle because the 
practice rarely fails, is easily accommodated, and is attractive to those 
who engage in it because it lets them completely abdicate responsibility for 
the mail they generate and send.

(I apologize for the overly complex sentences... )


> If 'SPF' and/or DKIM was standardised and rigourously enforced, then
> return-path rewriting would be de rigeuer as would people learning how to
> configure the mechanisms properly.

This would require universal deployment of people designing and running mail 
systems who are almost entirely not stupid, evil, irresponsible, lazy, or 
even overworked.

Good luck with that. If you figure out how to find clueful people in a 
reliable way, you can rule the world.

In the real world, the result is not (as some have claimed here) that SPF is 
completely useless. It's just not a tool you can use without paying 
attention to its many failure modes and particularly the ones where 
ignorance and hubris are likely to cause trouble. The canonical example is 
to counter phishing, where the sender domain is not used by random human 
users and legitimate sources are actually known. The next most commonly 
cited example is to recognize SPF records that consist of "-all" as the only 
element, stating that there is no such thing as legitimate mail from the 
domain. Stranger uses can be based on the recognition that there are 
spammers who know about SPF and about simple spam filtering techniques; 
being "too good" while being "very new" is a sign that a sender is a 
spammer, and an affirmative SPF result can be a "tell." (yes, I have SA 
meta-rules, no, I'm not sharing) If there is a general rule, it is that 
SPF's uses require a sender domain (good or bad) whose admins understand SPF 
well.

The bottom line is that SPF is actually useful as a small part of a spam
control system, but not in the simplistic way that it was designed to be
used or in ways that are practical for the giant providers of cheap/free
mailboxes. If you want to use SPF to control spam, then you need to have
someone skilled watching your mail stream. If you want to hand out cheap
mailboxes to anyone who wanders by, you will not find SPF to be useful.