Re: Comments on IM2000

On Fri, May 06, 2005 at 07:20:54PM -0000, James Craig Burley wrote:
> >If the majority of mail ends up in this state, then under your proposal the
> >sender of the mail will need to keep polling, just to prove that they are
> >'still interested'.
> 
> No.  That might *indicate* such interest.  But if the sender isn't
> genuinely interested in the email to *that* degree, there's no need to
> track the delivery, is there?  A quick answer to someone's question on
> an email list is really more of interest to the *recipient* than the
> sender, so the sender doesn't really care if it reaches the
> destination.

That I disagree with strongly. If I didn't care whether the recipient got
the response, I wouldn't have spent the time to compose it in the first
place.

(It actually annoys me when I send a mail in response to a mailing list, and
get the direct copy to the author bounced because they used a fake From:
address, or they use TMDA, or some other filter which rejects direct mail.
Posting a message in a public forum with a reply address IMO solicits a
response, and it is very impolite to reject that response.)

Anyway, I think the point is this: if I send a mail, I *do* want it
delivered. It's up to the mail system to do whatever is necessary to get it
delivered, or let me know if it cannot.

The SMTP world specifies various rules, or hoops the mail system must jump
through if you like, to perform a successful delivery. For example, if it
cannot connect to the remote SMTP server, or it can but it gets a 4xx
response code, then it must back off and try again later. Given that most
mail deliveries (perhaps over 99% ?) are immediately successful, then it's
true that some spam sending programs don't bother to implement jumping
through some of the less frequently-met hoops, and so greylisting was born.

But I think the fundamental problem with your proposal - if I understand it
properly, which I may not as it seems to be quite complex - is that you
would like to see a new set of hoops to jump through to ensure successful
delivery of mail. These hoops may involve being given a tracking number, and
having to poll status using that tracking number - potentially for a long
period of time after sending, if the recipient chooses not to confirm that
the message has been successfully delivered.

Now, the trouble is, everyone will have to write their mail sending system
to be able to jump through those hoops, because when I press "send" on my
mail client, I expect the mail system to do the *utmost* to get it
delivered. There's no point having another button which says "send but don't
try very hard". Why would I ever push that button when I could push "send"
to have a better chance of my message being delivered?

Now, because this new mail architecture provides more hoops to jump through
which are met more often, then people who write mail sending software
(including spammers) will be forced to make their code jump through those
hoops. That is, greylisting will become *less* effective, because the new
mail system puts a stronger requirement on senders to retry when the mail is
not completely delivered, and so spammers will do the same.

> >but spammers can easily match the raised bar too.
> 
> Addressed previously.  True for all designs; *inherently* least true
> for my proposal, since spammers have to send *all* message contents at
> least once (despite the vast majority of recipients being unlikely to
> ever read them), as with SMTP but not IM2000, and would have to then
> track *all* deliveries in a typical fashion to avoid recipient's
> detecting lack of interest in incoming email from a previously unknown
> source, which IM2000 provides in its own way, but not SMTP.

But that's not a very high bar, is it? I mean, it's the same as what we have
now (i.e. spammers have to send a copy of every mail to every recipient),
plus some subsequent probing to indicate "interest" in the message delivery,
which is easily added.

If this modified SMTP doesn't provide any better anti-forgery, or any better
tracking of spam to original source, then it's hard to see what benefits it
offers. And if it's SMTP just with added complexity, then the complexity
itself will become a problem as there are more corner-cases which may cause
failures; that is, if there are 10 different responses to receiving a
message which the recipient may give, and one of those is very rarely used,
then you may well find it doesn't interoperate very well. Even SMTP which
just has 2xx (OK), 5xx (fail) and 4xx (tempfail) has this type of problem;
you will find mail systems which do stupid things if you give a 4xx response
to a EHLO or a MAIL FROM, so in the end to maximise interoperability you
have to accept EHLO and MAIL FROM, and give 4xx/5xx responses to each RCPT
TO recipient (even if what you were trying to do was to reject the MAIL FROM
sender)

> (The email equivalent of this might be those advertisements I
> mentioned earlier, included in headers for emails sent via "free"
> providers.  If we move towards a model of having only a few
> well-trusted relays or mail stores, there'll be economic pressure for
> those stores to offer free accounts that result in such intermixed
> advertising, as in "Received: from brian by ... brought-to-you-by
> Pepsi <http://www.pepsi.com>".  I don't want this future.  ;-)

Absolutely. There's no such thing as a free lunch - and you only use
free hotmail-type services if you don't mind your correspondence being
polluted in this way.

> So, suppose sending UBE is essentially "free", in terms of iron and
> connectivity costs.

Agreed

> What's the *real* problem with that?  Two biggies:
> 
>   1. The infrastructure handling *receiving* emails might be unable to
>      cope with the load, so legit email wouldn't get through.
> 
>   2. Those reading email have trouble finding the few emails they
>      really care about among the huge amounts of UBE they (mostly)
>      don't, so legit email isn't always read.
> 
> Focusing on #1, I say let's think in terms of a system that makes
> *receiving* email as cheap as possible, at the (possible) expense of
> the sender.

Hmm, except (1) is OK at the present - that is, E-mail infrastructure copes,
even with >50% of all mail being spam. (1) is a problem for ISPs, but can be
solved by suitable spending of (their customers') cash. (2) is where the
real problem lies, for end-users at least.

> Now, the easiest and cheapest way I can think of to receive an email
> is to accept an *incoming* connection delivering it, after which I can
> take my own sweet time deciding what to do with it.
> 
> I don't want to have to look up any return address to provide status
> information (as in a bounce or DSN).  I don't want to have only a
> short window of time in which to respond definitively or risk
> duplicates or redelivery (as with the response to the SMTP DATA
> phase).  I don't want to have to respond *at all* to the delivery or
> to the sender's subsequent tracking requests.
> 
> How can it get any cheaper than that?  Yet, even if I don't do any of
> the things above, I can still receive a complete message, display it
> for an end user via an MUA, and the user can act on it as she sees
> fit.

OK. That's what we have now, minus the definite acceptance or rejection of a
mail at transfer time.

SMTP is already cheap to deliver to, and already people have been adding
various hoops to jump through (e.g. validating EHLO domain, validating MAIL
FROM) - some of which are arbitary, since the EHLO domain should not need
"validating" in the first place, and indeed the RFCs tell you that you must
not. Your proposal needs either to be able to convince people to drop their
arbitary hoops, or give them a new set of standardised hoops to jump through
- hoops which minimise the risk of rejecting legitimate mail, which
unfortunately current ones often do.

I see where you're coming from about leaving policy up to individual sites,
but unfortunately that leaves those sites free to choose very bad policies.
For example, as I said before, there are sites which reject all mail which
has an envelope-sender of [email protected] (and so my mail is blocked). That's
based on the stupid assertion that because they've seen some mail which had
MAIL FROM:<[email protected]>, then pobox.com is a spammer's domain.

In fact the opposite is true - pobox.com controls its users very tightly
indeed - but spammers have been forging mails with that domain.

The trouble is that the mail system purports to give some information about
a message (the sender/origin), when in fact that is NOT information, it's
mere heresay.

So more important to me is that any "information" which a receiving site
might use to validate or reject a message, is correct.

Now, even if your proposal removes the idea of a sender address and bounces
altogether, what you have instead is that the whole message headers and body
may be received before you analyse whether to accept or reject the message.
That means, for example, that people will do the same sort of stupid
filtering on the From: header that they used to do on the MAIL FROM
envelope. And of course, forged From: headers have another particular
problem with "phishing"-type attacks.

So I think it's essential that any proposal include ways to inherently
validate this information. IM2000 does because you *have* to use the sender
identity to collect the mail; if it's forged, mail collection doesn't work.
Something like DomainKeys on top of SMTP tries to do the same, although it's
heavyweight and take-up of anything relying on public-key cryptography has
historically been poor.

> As the expense of ordinary exchange of email drops across the board,

would it drop very much, compared to SMTP? The bandwidth and disk space for
receiving mail are the same, so you just lose the cost of being required to
send bounces after a message has been accepted.

Of course, most *recipient* E-mail systems avoid this already, by validating
the recipient at RCPT TO time and sending a 5xx response if the mailbox does
not exist or is otherwise unavailable. So the main cost here is at the
sending mail relays, who may be forced to try to return bounces to
(possibly) non-existent senders.

Eliminating that may reduce the cost of building outbound mail relays, a
bit.

> that frees up capital for other things, such as improved anti-UBE
> measures, improved vetting of new users by ISPs to be sure they
> haven't previously engaged in spamming, etc.

Ah, but if *I* save some money at my ISP, that doesn't free up money for
*other people's* ISPs to invest in controlling their users, unfortunately.

> Since my proposal obviates the need for bounces as does IM2000, joe
> job are eliminated, and each host's capacity for handling incoming and
> outgoing email can be speced based on *actual* expectations for
> *legitimate* outoing email plus legitimate incoming email plus a
> certain amount of incoming UBE.

Joe-jobs are a small subset of the problem, IMO. If that's your main
concern, then you can fix that using SES/SRS/BATV today. It would help if
there were a common agreed standardised format though, so that things like
mailing list software could recognise an encoded sender.

> (False positives aren't a problem with my proposal compared to SMTP,
> since a message is, presumably, never fully accepted and *then*
> discarded as spam, unless that's what a recipient *wants* to do.  SMTP
> servers accept and silently drop messages all the time these days.)

But with tons of spam hitting an inbox, how are you going to stop people
becoming overwhelmed with it - your problem (2) above? Surely people will
*demand* automatic filtering, and that in turn will involve accepting and
silently discarding mail as people do know? In my definition, automatically
receiving a mail into a 'spam' folder counts as 'silently discarding', since
people rarely look at such folders, and valid messages which end up there
are almost certainly lost anyway.

Regards,

Brian.