Re: IM2000: what it is and isn't

>On Mon, 8 Mar 2004 11:47, Jon Ribbens wrote:
>> Firstly, "cost to sender". UBE only exists because the cost to sender
>> is low. Clearly if the cost of sending an email were somehow increased
>> sufficiently, UBE would become unprofitable and the problem would be
>> essentially solved.
>
>This is all true, but I think you're following the wrong trail. The problem of 
>spam is "rule #0: spam is theft". So long as the only costs involved in spam 
>are applied to *willing* senders and *willing* recipients, it really doesn't 
>matter what the cost is.

I'm not sure I can agree with that: the costs to *agents* acting on
behalf of those senders and recipients *are* important.  Such agents
include outgoing messages servers and stores (for IM2000, and for SMTP
with upstream relays), incoming messages servers and stores, MUAs used
to send emails, MUAs used to receive emails, and the network itself
that connects all this up.

Though I don't think it's a crucial distinguishing factor -- besides,
IM2000 offers some opportunities for scalar reductions in costs for
certain common message-exchange operations -- we do have to be careful
to not discount the costs to such agents, which I'll call
"intermediaries".

After all, the reason we can have discussions like these is that the
costs to intermediaries is sufficiently low that they provide these
services for little or no directly billable price.

Think of what would happen if everyone switched to an email
architecture that required 20GB of data to be processed for every
single piece of email, no matter how small, as an extreme example:
even if senders and recipients were willing to bear that cost,
*intermediaries* would *not*.

(In fact, I gather this has been one of the reasons certain forms of
P2P haven't flourished as widely as email, news, and www serving;
though senders and recipients are highly *willing* to exchange data
via P2P, the intermediaries aren't happy about the costs, which, of
course, include risks of litigation over enabling sharing of illegally
copied materials.)

>So long as there are no unwilling participants, and 
>non-participants are not bearing any costs, cost is irrelevant.

That's another important point: some designs can impose unexpected
costs on non-participants (which include everyone who is not a sender,
a recipient, or an intermediary for a message).

Any design that creates tempting targets for attacks invites costs to
non-participants, as will any design that requires non-participants to
process and ignore, or especially respond to, components of the
message exchange.

So opt-out requires non-participants to respond, by opting out of
lists that they did not subscribe to in the first place.  C/R requires
non-participants to respond, by processing and ignoring Challenges
issued for joe-job messages that SMTP servers accepted and then
bounced in Challenge form.

And IM2000 potentially requires non-participants to respond, by
defending their outgoing message stores against attacks by spammers
who cannot convince those stores to hold *their* outgoing email, and
which have therefore become widely trusted by incoming message
notification agents.

>The problem 
>is not cost; it is cost-shifting from willing participants to unwilling 
>participants and non-participants. It is the cost-shifting that is the 
>primary concern, not the cost itself.

Yup.

>IM2000 potentially decreases all costs, since there are no bounces, and mail 
>which is known to be unwanted is not sent over the network.

How is it known that this mail is not wanted, though?  If it's based
on the sender and recipient addresses plus the originating message
store's identification, is that *enough* different from SMTP delivery,
which (compared to QMTP) sends the sender and recipient addresses
*first*, and includes the IP address of the *relaying* message store,
*before* transmitting the message, to justify rolling out IM2000?

Here's a clear advantage of IM2000: in the case of a trusted relay
(and IM2000 will obviously have to have relaying for message
*notifications*, right?), the final receiver of a notification will
not have to begin downloading the message content itself to learn of
the identity of the originating message store, in vanilla IM2000
anyway.  That'll be part of the "envelope", so to speak.

Whereas, with SMTP, if one trusts the IP address of the client as a
relay, one must then receive the message, parse the "Received:"
headers, and only *then* can one find the roughly-equivalent info on
the source of the message content -- a source that might well have
since packed up and moved on to greener pastures (which has both
positive and negative connotations for each of SMTP and IM2000).

However, IM2000 will, in practice, have similar problems, unless it's
designed to *not* have them.  In particular, lots of systems will be
configured to automatically replicate (cache, especially replace)
incoming message contents as the notifications are relayed, as in the
complete SMTP relay function.

Unless the *original* information on the outgoing message store from
whence that message was obtained is available along with the
*notification* of a message -- that is, available as part of the
envelope, not part of the message -- then the final recipient of the
notification will either not have access to that info or, if it's put
in the message contents a la "Received:" (a bad design IMO), it'll
have to be able to retrieve the message from the "message-store
relay", parse it, and *then* decide whether to treat it as coming from
an untrustworthy source.

>SMTP can be 
>operated in a somewhat similar manner if the MTA network is appropriately 
>configured, but it typically isn't. For example, the IM2000 "pull" approach 
>works better for forwarded mail. Forwarding in SMTP results in problems 
>unless the policy of the final recipient is applied at the first recipient, 
>and any system which retro-fitted such a policy-sharing mechanism would be 
>more complex than the IM2000 approach.

It's certainly true that retro-fitting such a mechanism onto SMTP
would be complicated.  But it's already being done; I've seen it
operate.  I've seen messages I tried injecting directly into an SMTP
server get rejected because of some policy that was temporarily broken
for some reason, then when I tried injecting them via comcast.net's
relay, I've seen them get rejected by that *relay* for the same
reason, except it was stated as "final destination system
requires...".

So either that relay had intuited that the final destination was using
a policy it might as well implement itself, or it was attempting an
immediate forwarding during the SMTP session and reporting on the
results.

I'm not sure which will happen, but IM2000 doesn't offer a clear,
total improvement over this either.  Either it won't allow relaying at
all (in which case it'll fail), it'll allow relaying of message
notifications (in which case the network will have to be up end-to-end
in order to implement policies when the notification finally reaches
the destination that knows about them), or it'll allow relaying of
message content (in which case we're right back to SMTP-land, albeit
with a cleaner protocol).

Here's a quick clue as to my *general* views on this issue: since
about the mid-1970s, I've believed, and continue to believe, that
propagation of such policies "upward and outward", generally through
upstream connections in a backbone-based Internet, but even via
ordinary connections in a peer-based network, will *ultimately* be
required in any sufficiently hostile network.  And the Internet is
more than sufficiently hostile.

So, though we can discuss the details, I don't believe IM2000 or any
other technology will make the need to propagate policy info go away.

>> Furthermore, since the spammer will clearly be operating their own
>> message store, they receive the great benefit of knowing who has
>> decided to read their messages.
>
>Slight correction: the sender knows which entities have *fetched* the mail, 
>not whether they read it. Whether or not the recipient also read it depends 
>on whether the fetch happened as the result of an attempt to read, or simply 
>as a matter of pre-fetching for some other reason (such as to run through a 
>spam filter). The spammer also knows which entities accept their mail under 
>SMTP, but not whether they read it. Information about reading is best 
>obtained by some other channel entirely, such as HTML mail and web bugs.

AFAIK, with IM2000, the sender who notifies a lot of dispersed
recipients about the availabilit of a single email (e.g. a spammer),
and thus isn't interested so much in whether "everyone" has read it
and thus he can delete it, since he plans to leave it up for only as
long as he can get away with it, is less likely to learn the
identities of those who have downloaded it, compared to SMTP.

That's because the request for a message in IM2000 comes separately
from the information on the recipient (which is part and parcel of the
SMTP stream).

IM2000 is an improvement in this regard.  Senders who want to know
exactly who is downloading messages they've sent will at least have to
go to the trouble of providing a unique ID for the message for each
recipient they notify about its availability.

>If an implementation of 
>IM2000 cut all mail-related bandwidth in half, then that would be a benefit 
>even if it had no impact on end-user-visible spam at all.

Yes, but I believe that's an insufficient benefit to justify the huge
"capital expenses" of rolling out IM2000 as a replacement for SMTP.

>> It seems me that IM2000 increases the cost to the recipient. The
>> recipient has less information about the message to make their
>> decision as to whether or not the message is something they want to
>> read.
>
>Despite thoughts to the contrary on this list, IM2000 is not a protocol: it is 
>a broad theory of mail technology based around the "pull" principle. "IM2000" 
>is just DJB's catchy brand name for pull-based mail. Be careful whether you 
>are arguing against IM2000 itself, or someone's attempt at implementing it. 
>What you're arguing against here is a particularly bad way in which one might 
>implement IM2000, rather than the principle of pull-based mail generally.
>
>Sure, if your implementation of IM2000 effectively meant "mail a URL to the 
>recipient using SMTP", then that would be a lousy implementation, although it 
>still obeys the overarching principle of IM2000, in which the recipient 
>domain pulls the message content from the sending domain. It's also possible 
>to turn SMTP into a pull-based system which is only marginally different to 
>push-based SMTP: see my BSc(Hons) thesis at the following URL.
>
>  http://www.comp.mq.edu.au/~brett/bschons/index.html

At this point my brain hurts.  *Until* you pull down the email, you
have precious little information on which to decide whether it's worth
pulling down.  But *once* you pull it down, you've now wasted
bandwidth for an unwanted message just like with SMTP-style delivery.

What am I missing here?  What is the magic bullet that allows me to
*know*, in advance, that a message is spam or vermin, without having
to pull it down?  A magic bullet that *cannot* feasibly be provided in
SMTP's existing ecosystem?

>> Finally, sender authentication. This is the problem that things like
>> SPF try to address. ... IM2000 does not address this issue in any way
>> so far as I can see.
>
>IM2000 does not address the issue directly, but it does offer a firmer 
>foundation than SMTP for an authentication layer. SPF is, necessarily, a 
>horrible ugly hack. It uses the envelope sender field as an identity, and the 
>envelope sender field was never intended to be an identity. The envelope 
>sender is just the address to which the originator wants bounce messages 
>delivered. A new mail protocol which uses IM2000 principles is in a very good 
>position to overcome this problem, since it can include a "message 
>originator" identity field, and authenticate that identity. (IM2000 has no 
>need for an envelope sender field, since there are no bounces.) An SPF-like 
>authentication system for IM2000 would restrict associations between message 
>originators and IM2000 message stores.

That would help prevent easy dDOS'ing of outgoing message stores, but
it would increase the costs associated with receiving message
notifications, because, presumably, one would not want to even
*bother* with a notification that forges an outgoing message store ID.

I'm not sure why that same kind of mechanism couldn't be applied
within SMTP to combat forged "Received:" headers, offhand....

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>