Re: Comments on IM2000

>It was thus said that the Great James Craig Burley once stated:
>> 
>> LKML then sends a tracking request out to each of the remaining
>> recipients about a minute later.  Assuming they work pretty much as
>> they do today, they'll have committed the messages to disk, done any
>> preliminary O(N)-type scanning of content, and accepted the messages,
>> so they'll respond "I've accepted responsibility".
>
>  See, this is where I don't understand the tracking mechanism.  Today, you
>(more or less) have
>
>	MUAs -> MTAs -> MTAr -> MUAr
>
>  I don't see any real difference with SMTP now and what you are proposing,
>except for the overhead of polling ("Did Bob get this yet?  Did Bob get
>this now?  What about now?  Now?  Did he just read his email?") and a
>possible annoyance factor [1].

It's beneath the radar for the recipient, and the *sender* isn't
likely to be annoyed by her *own* use of tracking!  (We probably have
two different ideas regarding what "tracking" means -- see below.)

But if you find today's bounces and joe jobs less annoying to deal
with than lightweight sender-initiated tracking requests and
responses, great.  I'm pretty sure I'd prefer a tracking-based system,
as either an end user *or* as a sysadmin for an MTA that sees a lot of
incoming UBE and joe-job bounces.

Just keep this in mind: a typical tracking request/response pair costs
little or no more than a typical DNS lookup, *assuming* the entry is
already cached in the upstream DNS cache.

In SMTP-speak, think of a tracking request this way.  An SMTP client,
MTAs, initiates a TCP connection to an SMTP server, MTAr, to use your
terminology.  It goes something like this:

  <-220 MTAr.example.net, pleased to meet you ESMTP
  ->EHLO MTAs.example.com
  <-250-MTAr.example.net, pleased to meet you ESMTP
  <-250 TRACKING
  ->MAIL FROM:<[email protected]>
  <-250 ok
  ->RCPT TO:<[email protected]>
  <-250 TRKID=0123456789 ok
  ->DATA
  <-354 go ahead
  ->[...message contents...]
  ->.
  [MTAr disconnects]

MTAs can't be sure whether the message was accepted or not, since it
never saw a response to the DATA phase.  It should therefore treat
that as a temporary failure (as if a 4xy response code was sent).

So, sometime later -- maybe a minute or two -- it throws a UDP packet
at MTAr (probably preferring the same IP address, if the latest MX
lookup returns that IP in its result set) saying little more than
this:

  Status of 0123456789?

MTAr might choose to response with the equivalent of little more than
this:

  250 TRKID=0123456789 ok

At that point, MTAs has achieved the equivalent of the SMTP
conversation ending with the following, prior to disconnection:

  <-250 ok
  ->QUIT
  <-221 quitting

Now, how many more packets would that final exchange, during the SMTP
session, have cost anyway?  Well, given the nature of SMTP, at least
as many as a single tracking request/response pair, is my
understanding.

Of course, MTAr might not respond to the tracking request at all,
which would cause MTAs to probably try again in another few minutes.

Or, MTAr could respond with a 4xy or 5xy code, or a new code, 6xy,
meaning, literally, "ask me again later" (though perhaps the "xy" in
"6xy", or the extended response code, would give MTAs enough of a
sense of resolution to stop tracking).  (Of course, 6xy codes wouldn't
be returned except to clients that somehow advertised they understood
them -- probably via new SMTP verbs or something similar.)

At some point, if MTAs isn't satisfied the message was received, it
could simply attempt to transfer the message again.  That conversation
might look like this:

  <-220 MTAr.example.net, pleased to meet you ESMTP
  ->EHLO MTAs.example.com
  <-250-MTAr.example.net, pleased to meet you ESMTP
  <-250 TRACKING
  ->MAIL FROM:<[email protected]>
  <-250 ok
  ->RCPT TO:<[email protected]> TRKID=0123456789

At that point, if MTAr sees it already has the message contents
available locally, it can tell MTAs to not bother resending that.  Or
it can explicitly reject the message without accepting the contents.

This is my rough outline of an SMTP++ that would implement my proposal
in the context of existing SMTP implementations.  It wouldn't be
*particularly* clean, but it could be quite effective anyway.

Under my proposal for a *replacement* for SMTP, the conversations
above would be much simpler, with hardly any handshaking at all.

And I think it'd be easy to make the new protocol naturally support
both push-style *and* pull-style delivery, so IMAP/POP3/IM2000-type
access could simply "fall out" of the protocol design.  (But that's
not terribly interesting if the basic premise of my system --
submission following by tracking -- is *itself* uninteresting, since
that would imply that SMTP with bounces will always be adequate, or
else IM2000's pull-style delivery will be necessary after all.)

>[1]	Years ago I worked at IBM and used its internal email system (not
>	SMTP based---some proprietary mainframe format that required users
>	to log *into* the mainframe before checking) and it had tracking
>	capabilities much like you mention and believe me, I *hated it*. 
>	Managers (and team leaders) would routinely mark their email with
>	return receipts indicating not only when I received it, but when I
>	*read* it.
>	
>	A friend of mine found a way around the tracking mechanism though (I
>	think by reading the spool file directly but it's been way too many
>	years).  Just because *you* want it doesn't mean *everybody* wants
>	it.

In other words, you dealt with a stereotypical big-iron imposition on
how you read email.  That's not the sort of tracking system I'm
talking about, though it can be *cooperatively* used in that fashion.

Just because that system used the word "tracking" and I'm using the
word "tracking" does *not* mean I'm proposing the same thing.

I do worry about the possible downsides of allowing a recipient to
specify that a message has actually been *read*, not just accepted
into a mailbox a la IMAP/POP3 (with responsibility being taken over
from the sender), because there will always be suits who will require
their employees to enable and dutifully obey that "option".

That's not really the fault of the protocol, though, and I think
enough people would *prefer* to have the choice of whether to request
and/or provide that level of detail regarding disposition of emails to
justify providing it (and providing it would be trivial in the context
of the protocol I'm designing).

By the way, one thing I did not like about IBM's old mainframe designs
was the fact they were built around polling, not interruption in a
pro-active sense, when it came to the CPU<->peripheral interface, and
that mentality seemed to creep into other areas that, from my Digital
and Pr1me background, simply bothered me.

Similarly, I've been very hesitant to propose this tracking concept
for ordinary email, because I just don't like polling, generally
speaking.

I've come to believe it's best, however, in an email environment.
Ultimately, it really *is* the sender that cares most about the status
of an outgoing message.  That's different from a CPU having to poll a
3270 terminal to find out whether its user has hit "Enter".

Still, I'd like to provide the option for highly efficient
bounce/DSN-style notifications in this protocol, maybe without even
requiring backwards DNS lookups most of the time.  (A sender could
provide an IP:port combo to the recipient for such notifications, and
even give a DNS-style expiration for that combo in case the sender
switches to another IP address.)

>I would also think this tracking mechanism
>would be quite expensive for a large mailing list.

"Profile, don't speculate."  As I said, it's on the scale of the
"expense" of DNS lookups that are performed when distributing an email
to each member of the mailing list.

And that sure beats the list manager having to overcome anti-UBE
mechanisms like greylisting, which require subsequent TCP connections
and re-transmission of entire message contents, or Challenge/Response,
which requires *multiple* TCP connections, as just two examples.

>Now, for each outgoing
>message you have *two* connections, one for the sending of the email, and
>then another one (or two, or three) constantly nagging the other end "Did
>Bob get this?" (okay, maybe not that often, but still ...).

What second *connection*??  It's one outgoing UDP packet, and
potentially one incoming UDP packet in response.  Like a DNS lookup in
terms of bandwidth and overhead, except there's rarely a need for the
sender to recurse or redirect to another entity, unless the recipient
responds in a way that suggests the sender do exactly that.

Compare that to the deluge of joe-job bounces, "weird" (or useless)
bounces, and the *lack* of bounces in cases where messages were
dropped, all of which involve TCP connections and multiple handshakes
*within* those connections (never mind the handshakes needed to set up
and tear down TCP connections *themselves*), thanks to SMTP's design,
not to mention the resources needed to simply *compose* a bounce in
the first place, for non-English-speaking people to *decode* its
meaning, etc.  (Some of this doesn't really argue against the
*concept* of a recipient-side notification -- just SMTP's
*implementation* of that concept.)

>> Only two more tracking request per remaining recipient would be sent,
>> assuming the recipients didn't respond.  The LKML server would,
>> presumably, treat inadequate response by any recipient after the final
>> tracking request as it would if it received a bounce in today's
>> system.
>
>  I understood the IM2k method as working like this:  LKML would send out a
>notification of email to each recipient:

That's a different issue -- it's not a *traditional* mailing list, in
terms of operating a la SMTP.  Since my proposal is much more
SMTP-like, I responded to the previous question about list management
in terms of an SMTP-ish mailing list engine.  (I haven't given list
management a lot of thought, frankly; partly because it seems to me
that it'd be a lot like it is under SMTP, since the basic push model
is preserved.)

But, if you study your own example, you'll see that each "notice" is
just like my tracking request in terms of resource utilization, except
it's sent *first*.  Then, your example shows an *incoming* TCP
connection being made to MTAs in order to fetch the body -- which,
aside from the well-known problem IM2000 will have with higher-latency
networks, means DNS must be working in the backwards[A] direction, and
MTAs *must* be available when the *reader* (via MUAr1) wants to read
the message.

Since my proposal involves MTAs submitting the entire message
contents, along with the notification, directly to MTAr1 (which in
turn can cheaply forward it -- that is, with no fsync()'ing to disk --
to MUAr1, if it happens to be online), and only *afterward* following
up that transmission with a tracking request (if necessary), the
*essential* bandwidth requirements are roughly similar, except your
IM2000-based example requires MTAr1 to do a backwards DNS lookup to
find MTAs, which is not necessary (under my proposal) under normal
circumstances.

[A] I realize I've used "reverse" in connection with DNS lookups, but
"backwards" is a better term in the context of this discussion.  A
"forwards DNS lookup" is what happens when you type http://google.com
into your browser -- it has to look up google.com to find to which
host to connect.  A "backwards DNS lookup" is what happens after
you've submitted a URL to, say, your own web site, to a mailing list
or blog or similar -- anyone clicking on that link must look up that
URL via their own DNS mechanism.  Most of the time, that's fine,
except it is more prone to abuse (by you and anyone else who is freely
able to submit URLs through such channels).  So, *generally*, it's
best to make sure there's a *need* for such a backwards lookup, rather
than simply expecting the "submitter" to directly inject the desired
content, instead of a backwards pointer to it, into the forum in
question.  The "slashdot effect" is, conceptually, an example of the
problem of backward (not just DNS) lookups being substituted for
direct injection of content -- it's pretty much a combination of
convenience and copyright law that prevents the direct use of content
in the first place.

>  Going back to individuals sending email, it works simularly.  Alice sends
>Bob an email [2]:
>
>	Alice -send-> 	Iago
>			Iago -notice-> 	Ibby
>	Alice <-stored- Iago
>
>					Ibby -notice-> 	Bob
>					Ibby <-getbody-	Bob
>			Iago <-getbody-	Ibby
>			Iago -body->	Ibby
>					Ibby -body->	Bob
>	Alice <-recv-	Iago
>
>  (that last step assumes Alice is online at the time---otherwise, the
>notification can happen the next time she checks her email).

That seems a tad complicated and wasteful.  Compare it to this:

	Alice -send->	Iago
			Iago -send->	Ibby
					Ibby -send->	Bob
					Ibby <-thanks   Bob
			Iago <-thanks   Ibby
	Alice <-thanks  Iago

Now, each send/thanks pair *can* happen within a *single*
rightward-moving TCP connection, in which case neither Iago nor Ibby
needs to commit to storing the message contents on disk at all.

If the TCP connections don't stay up long enough to accommodate that,
or if Ibby simply doesn't want to confirm receipt to the degree Alice
wants to see, then it might look like this:

	Alice -send->	Iago
			Iago -send->	Ibby
					Ibby -send->	Bob
					Ibby <-thanks   Bob
[...later that day...]
	Alice -track->  Iago
			Iago -track->   Ibby
			Iago <-thanks   Ibby
	Alice <-thanks  Iago

("Thanks" could mean just "responsibility for message accepted" a la
an SMTP response code of 250, but Alice might want to be sure it goes
further -- until it means "message stored in Bob's mailbox", "message
given priority higher than spam by Bob's MUA", or "message actually
read by Bob", depending on how important the message is to Alice.  If
it doesn't reach the desired state quickly enough for her, she can
pick up the phone and call Bob.  Alice would therefore love tracking,
since it'd potentially keep her from having to pick up the phone and
pester Bob regarding each "important" email she sent.  Coincidentally,
I used to work for someone named Alice!)

Now, in this sample use of tracking, though Alice has to look up Bob's
MX (yielding Ibby's IP address) again via DNS, it's quite possible the
answer will be in a nearby DNS cache, since Alice *already* looked up
Bob's MX to send the message in the first place.  But neither Ibby nor
Bob have to look up Iago nor Alice, since they're not trying to send a
bounce or DSN, so *their* DNS cache isn't ever polluted as a result of
*receiving* incoming email or tracking requests.  In both cases, the
cost to outside third parties is lower -- about as low as possible, in
fact, when it comes to exchanging an email message.

Further, those track/thanks pairs are about as expensive, in terms of
bandwidth, as simple DNS lookups.

>> Since handling an incoming bounce is inherently much more expensive
>> than sending a tracking request and receiving a response (or waiting
>> for one that doesn't arrive), the question is how does the ratio of
>> all those tracking requests and responses compare to the (presumably
>> smaller) number of actual bounces?
>> 
>> The third-party costs are lowered as well, because sending bounces or
>> DSNs requires DNS lookups in the *reverse* direction.  That's
>> conceptually a waste of resources (DNS caches), since bounces/DSNs
>> aren't really *original* communications, they are *responses* to
>> communications.
>
>  Sending the intial email requires DNS lookups to begin with---actually, a
>minimum of two (one for the MX record, then for the A record of the MX
>host) and possibly more.  I don't see this as being a major issue.

That misses the point entirely.  It's the lookup in the *backwards*
(left-moving) direction that is, strictly speaking, unnecessary.  With
vanilla SMTP, it occurs only when a bounce/DSN must be sent.  With
IM2000, it occurs every time an email is exchanged.  (Besides, A
records might accompany MX records in the "additional" section of a
response to a DNS lookup, though that's not particularly pertinent.)

As I've said before, when you design a protocol (such as SMTP or
IM2000) to rely on a backwards-initiated communication that requires a
DNS lookup in order to fully exchange an email, you give *anyone* the
ability to pollute a recipient's DNS cache with arbitrary (and
generally useless) information.

Seems like everyone who thinks they can design, improve on, or
criticize an email-exchange protocol trivializes or ignores the
burdens their ideas place on third parties, including distributed data
bases like DNS.

But those are *not* infinite resources; if they were, we could burden
them even *more* by simply sending *all* of our email through them,
and whitelisting them for incoming email -- problem solved.

Since they're not infinite resources, and since they are not able to
*directly* stem the flow of invalid requests for their services, they
should not be designated as critical paths for something as crucial as
email exchange.

For an information-exchange system in a hostile environment, then, the
key is to keep the exposure of a recipient's mechanisms to outside,
untrusted entities as limited as possible in the general case.  To put
it bluntly, that means a protocol should lean towards require the
*sender* to "expose" itself moreso than the recipient.

>> Tracking requests and responses would not be DNS-addressed -- they'd
>> be more ephemeral than bounces. 
>
>  Um ... how else do you know who to send the tracking requests to? 
>Granted, IP addresses may not change that often, but they do change
>(renumbering, moving to a new provider, etc).

The same way TCP on the receiving end of an incoming connection knows
to whom to send responses to that connection, data from the receiver
to the sender, etc.  (Tracking requests are ephemeral, like DNS
lookups or TCP connections; they are not queued.)

But if you mean how does a *sender* know to whom to send the tracking
request -- well, to whatever MX is currently advertised for the
destination domain name, assuming the recipient didn't already provide
a list of IP addresses to use in response to the original message
submission or a previous tracking request.

(Yes, that MX lookup is another DNS lookup.  But it's not a
*backwards* lookup, and it's quite possible the information will be in
a local DNS cache, since the same lookup was recently done to submit
or track the message.)

>> >  -spc (And what's with the tracking anyway?)
>> 
>> Not sure what you mean by that.  Don't you track important packages
>> you send via FedEx &c. using their online web sites?
>
>  Um ... nope.  Nor do I track regular snail mail (is that even possible?)

No.  So you don't know snail mail got lost until, sometimes, long
after you sent it, if ever, because you're relying on a *return*
mechanism that is basically no more reliable than the *sending*
mechanism.  Hence the need for certified mail and the like, since
reliability of regular snail mail is widely known to be inadequate for
many situations (and, yet, adequate for a huge % of outgoing mail).

At least with the US Post Office, you have a monolithic organization
that is "committed" to making the system work...at a substantial loss,
last I heard.

>> That's why I keep coming back to wanting direct MUA<->MUA
>> communications, which takes me away from *presuming* there'll be large
>> monolithic third-party systems, stores, data bases, etc. that enable
>> email exchange.  (Although, in practice, even under my proposal, such
>> entities would likely exist, though more to add value in various ways
>> than to merely *enable* email exchange, as IM2000 and, increasingly,
>> SMTP seem to do.)
>
>  Well, until IPv6 become common, don't expect full peer-to-peer
>communications on the Internet [3].  

>[3]	Yes, the Internet was designed to be peer-to-peer, but since 1994
>	it's become less and less peer-to-peer, what with firewalls, NAT and
>	the assumed scarcity of IP addresses.

Good point.  In the meantime, having lighter-weight "smarthosts" as I
described would, I think, greatly improve the picture and allow people
to exchange emails with much lower overhead and without having to rely
any more heavily on (disinterested, buggy, and/or hostile) third
parties than they do when they pull up each others' web sites in their
browsers.

In particular, MTAr's, to use your terminology, do not need to ever
commit (fsync()) incoming message contents to disk.  They *can* store
them locally as long as they have the space, and if they are
sufficiently convinced of their own prowess (reliability,
availability, etc.), they *can* certainly tell the sender that they're
taking full responsibility for further delivery.

But they can also just keep track of "logged-in" users (where their
MUAs are, in the Intranet for which they relay) and forward message
submissions and tracking requests to them, then forward the responses
back to the MTAs side of things.

Since MTAs, in this system, don't need to invest significant resources
into handling potential UBE, they can let individual MUAs and users
make finer-grained decisions regarding what constitutes Unsolicited
and/or Bulk Email -- decisions that spammers will have a much harder
time reverse-engineering, since they'll be more individualized,
heterogenous, and, often, more opaque to senders.

>  -spc (Who sadly remembers when he had 32 static IP addresses routed
>	to his home ... )

Heh.  I'd love to have just *one*.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>