Re: Comments on IM2000

> http://pobox.com/~b.candler/doc/misc/im2000.html

Excellent document!  Here's my take on it.

Item 2i, on content filtering possibly becoming unnecessary:

  One person's UBE is another's WBE (Wanted Bulk Email).  The
  *pertinent* definition of "unwanted" email ultimately comes down to
  a recipient's personal choices, and the basis of *most* (not all)
  such choices revolve around *content*, not identity of originator.

  As an exercise, imagine we all had unlimited bandwidth, storage, and
  infinitesimal latencies in our communications with each other.
  (It's not *that* difficult to imagine this; it's the direction in
  which technology has been heading for centuries, even millenia.)

  And, imagine the population was constantly changing -- not just new
  people coming "on line", but existing people changing their ways
  (going from being spammers to "solid citizens", etc.).

  Here, technology is *clearly* not the limiting factor; our
  individual capacity to cope with the influx of information, and sift
  the chaff from the wheat, becomes more evidently the nub of the
  problem.  Any "trust system" that depends on a single worldwide
  "root" is fatally flawed; we don't depend on one in real life
  anyway.

  In that scenario, the problem of "too much spam" is really one of
  "too much information, not usefully prioritized for *me*", since, as
  everyone who has received emails from friends about the threat to
  the Brazilian rainforest or similar knows, what's important (and not
  spam) to a sender is not always *that* interesting to a recipient,
  and vice versa (a sender might easily offer advice that is
  tremendously valuable to a recipient).

  (This ideal world can't avoid the problem of viruses and the like,
  since their equivalents exist in *our* real world.  Even if we
  assume there will be perfect security across all computer systems,
  this ideal world will still be tempting for someone to *try* to
  target by sending tons of "useless" emails trying to get people to
  do something stupid.  In practice, therefore, even ideal
  technologies will require filtering based on *content*, not just
  identity or history of the sender.)

  Accordingly, especially as technology advances, but even now, the
  ultimate solution includes giving end users an easy way to
  prioritize their incoming email so UBE *tends* to be easy for users
  to recognize, skim through, and filter out.  That will probably mean
  using AI-like technologies to serve as a per-user agent to do the
  content filtering and prioritization automatically, though info on
  the sender/originator can be very helpful as well.  (E.g. one's
  email-reading agent can say "this email appears to solicit a unique
  partnership, but the sender is known to have solicited a similarly
  unique partnership with about 200 million other people in the past
  three days".)

  The problem with *that* viewpoint is that it doesn't really justify
  switching to IM2000, since spam is ultimately defined based on
  *content*, not on the (label worn by or given to the) originator.
  But analyzing *content* requires retrieving the message anyway, in
  order to analyze it, apparently defeating one of the big wins of
  IM2000.

Given that we'll never see this ideal world, and that latencies (as
well as outages) *are* a huge problem, we then move on to the One Big
Problem with IM2000 -- that of the end user seeing substantial delays
when trying to pull up any given message after clicking "read" on a
notification panel.  (You address this in "Points for discussion and
further thought".)

To work around that, and to work around the fact that blacklists will
never be perfect, whitelists will never be easily maintainable, laws
will never be vigorously and uniformly enforced, and so on -- else we
could probably just stick with SMTP, thankyouverymuch -- the reality
is that the vast majority of *real* end users' mail-reading systems
*will* retrieve message contents immediately, or nearly so, after a
notification is retrieved -- to cache it (a form of prefetching), to
scan it for spam/viruses, or, most likely, both.

This leaves, for me, the "big win" for IM2000 -- that of being able to
retrieve a message without unpinning it.  But that is not, in SMTP,
terribly unlike returning a temporary failure response after receiving
the message contents, yet still "delivering" the entire message, after
a fashion, for subsequent processing, analysis, etc., in order that a
subsequent delivery attempt by the upstream SMTP client might be
"rewarded" with a success response, a permanent failure response, or
just another temporary failure response (either because scanning has
been inconclusive or because the SMTP server has decided to simply
irritate/tarpit a "known" spammer).

This form of receipt, where the sender is told "I'm not accepting
*responsibility* for delivery, but I might look over the entire
payload and accept or permanently reject it later on if, or when, you
retry", is somewhat like today's greylisting.  But since it accepts
transmission of the message and (potentially) does something with that
message, I might call it "beigelisting" or "ecrulisting".  ;-]

Given this perspective, I wonder why we need to move to IM2000 at all,
when the delivery of the message *contents* is already handled fairly
efficiently and inline (without a reverse-DNS lookup) by SMTP.  Adding
some protocol goo to enable tracking of once-delivered messages could
avoid the necessity of always transmitting an ecrulisted message a
subsequent time.

(We tend to think and talk as if message transmission involves the
sender giving *two* things to the receiver -- the envelope and the
message contents.  I believe a *third* thing is involved: that of
*responsibility* for the message, or at least its contents.  Spammers
get their big bang for the buck not just by transferring message
contents, but by transferring *responsibility* for those contents.
Any system that can accept contents while deferring accepting
responsibility includes a substantial potential built-in cost for
senders of UBE that most ordinary users will not find difficult to
bear.)

Item 8d, parenthetical comment on privacy problems, etc.:

  IM2000 definitely makes for a "cleaner" separation between notifying
  a recipient of the availability of a message and sending the message
  along, since, with SMTP, you pretty much *have* to send the message
  in order to make a sufficiently useful notification (taking into
  account relaying, the recipient's desire to filter/prioritize based
  on info in the header of the email that should really be in, or
  "on", the envelope, and so on).

  I don't worry too much about the implications of this, however.

  First, just because an IM2000 message has been retrieved, or even
  unpinned, does *not* mean any end user has viewed it.  That could
  have been done solely to apply a filter/prioritization to the
  message, to assure that the sender isn't just sending out arbitrary
  notifications, to "cache" a message that isn't ever actually read by
  a real person (such as for an abandoned account), etc.

  Second, the reality of any two-party exchange of information is that
  the sending party can never safely *assume* the receiving party saw
  an entire message *unless* the receiving party specifies it did so
  via a confirmation.

  Therefore, a receiving party can always choose to not confirm it
  received the message, even though it did.  (The sender can insist on
  breaking the message into a series of distinct message chunk,
  sending each chunk only after receiving confirmation that a previous
  chunk has been read; this doesn't really solve the problem for the
  final chunk, and makes the transmission protocol much slower, as
  latencies become amplified in their effect.)

Item 9, "Retransmission of notifications"

  What I think keeps getting overlooked in discussions about UBE,
  SMTP, IM2000, and so on, is that it's not just the *recipient's* job
  to somehow decide whether some message is desired.

  It's the *sender's* job to decide how important successful delivery
  of a given message is to that sender.  As the end-to-end principle
  implies, since email delivery can *always* ultimately fail (and,
  even with SMTP, this can include the possibility of failure to
  deliver a bounce back to the sender), it is up to the *sender* to
  expend resources to assure successful delivery via the email system
  or, if that fails (or might be deemed by the sender to fail, e.g.
  no confirmation within the sender's desired timeframe), by some
  other means.

  These sorts of discussions take this *implicitly* into account,
  especially when people talk about Challenge/Response -- "if I send
  an email to you giving you advice on something you ask about in a
  public forum, why should *I* waste time responding to *your* email
  system's challenge?" -- but I think it should be made both more
  explicit and fine-grained, on a per-delivery basis.

  Therefore, a *sender* should be able to control just how
  "persistent" his outgoing MUA is in terms of notifying the recipient
  about the message being available, checking on the status of
  delivery, and so on.

  IM2000, as presently formulated, really doesn't offer much more
  flexibility in this regard than does SMTP.  (One could argue that
  SMTP's flexibility in this regard hasn't been exploited, suggesting
  that indicates the capability isn't desired.  My counterargument
  includes the fact that *spammers* already exploit this flexibility
  by running "less persistent" SMTP clients; hence the apparent
  "success" of anti-UBE schemes like greylisting.  I've certainly made
  some use of my primitive abilities to determine the relative
  importance of outgoing messages, since, as I run my own MTA, I can
  see whether, when, and sometimes how, certain important messages are
  successfully delivered.  Ideally, MUAs should make this easy by
  reporting even temporary SMTP delivery failures back to the user in
  some fashion, logging all delivery attempts, and so on, on a
  per-message basis.)

  An advantage to giving senders a wider range of options when it
  comes to notifying and inquiring about outgoing emails is that it
  allows *recipient* systems to actually "bias" their prioritization
  of incoming emails based on the apparent persistence of the sender.

  Rolling this sort of system out from the beginning would offer great
  assurance that the retransmit features of IM2000 *would*, in fact,
  be properly and thoroughly tested.

Item 10, "Dynamic equilibrium?"

  Great point.  Since one person's UBE is another's WBE (Wanted Bulk
  Email), spammers will always have some market out there.

  Therefore, I don't think "eliminating" spam is possible.  I think
  the solution lies in giving end users an easy way to prioritize
  their incoming email so spam *tends* to be easy to for users to
  recognize, skim through, and filter out.

  My question is (and has been, for some time), will the expense of
  moving to IM2000 justify the *relative* reduction in spam that we'd
  actually see, in practice?

I believe any new email protocol -- IM2000 or an SMTP upgrade -- gives
us the biggest bang for the buck by replacing the bounce concept with
the tracking concept (widely used by delivery services such as FedEx
and UPS).

Accordingly, the response to a tracking request (issued, presumably,
by a sender) might, if the recipient so chooses, include more than
just "message contents have been retrieved" -- it might include
"message has been placed in recipient's in-box at [low/medium/high]
priority", "message has actually been read by recipient", "message has
been [archived/printed/trashed/shredded] by recipient", and so on.

Without this facility, many users on "friendly" terms will try to
present and/or obtain the information via some other means.  So it
might as well be included as an option in protocol.

Since, in reality, IM2000 would *have* to be a store-and-forward
protocol to meet widespread acceptance (other IM2000 advocacy appears
to deny this, but your own description of how it would be useful
essentially *assumes* some capability equivalent to the
store-and-forward capability of SMTP, especially with POP3 and/or IMAP
added), issues revolving around transferring, redirecting, and/or
forward responsibility for message notification and content delivery
would presumably exist for message *tracking* as well.


All in all, I've come to believe the "email problem" is becoming just
an increasingly large subset of the general problem of sharing,
transmitting, authenticating, storing, mirroring, revising, and
end-of-lifeing (and so on), *information*.

That is, it seems to me that, between our increasing awareness of the
need for a great deal more flexibility in our email systems and of
that for other related systems (version control; website handling,
including RSS; blogging; software distribution; and so on), it might
be useful to begin thinking in terms of a "grand unified principle" of
information exchange, reduced to a coherent architecture that can be
designed and implemented as a general-purpose information broker on
top of an unreliable, heterogenous network like the Internet.

(E.g. sending an email is not terribly unlike uploading a file,
except, for one obvious difference, the sender/uploader doesn't care
what name is given to the file by the recipient -- unless the sender
wants to refer to it later, such as in subsequent requests for
tracking, deletion, revision, etc.  Along this line of thought,
imagine how many security holes and other expenses would have been
avoided from the outset had Unix offered, instead of the /tmp
heirarchy and similar customs, a variant of fopen() that allowed the
caller to say "Lemme start writing to a private file, but *you*, the
OS, tell *me* how I can refer to it later via some handle".)

Your document really makes this point in many ways, when it talks
about IM2000 as a potential replacement for IMAP (and hence POP3),
which is, IMO, a very pertinent insight (since those are *already*
"pull" protocols), among other things you mention.

If we could pull *that* off -- design and build a "grand unified
information exchange protocol" (GUIXP? ;-) -- then layering an
email-like system on top of it should be a piece of cake, regardless
of whether it's IM2000-like (pull-based), SMTP-like (push-based), or
something in between.

After all, UBE is really just a subset of UBI (Unsolicited Bulk
Information), in the sense that UBI includes spam posted to blog and
other web sites allowing posting of arbitrary anonymous content, as
well as to USENET.  So the fight is really not *just* in the email
arena, and many of the techniques used in that arena have plenty of
applicability elsewhere (and, presumably, vice versa).

And, personally, despite my server being targeted heavily by spammers
(I'm a joe-job victim as well), my "spam problem" doesn't really
strike me as being as difficult to handle as many of my *other*
information-management problems, many of which strike me as being
elegantly solveable by a more-general GUIXP-type system.

If such a system is conceivable, and doable within, say, 20 years, it
might not be worth rolling out IM2000 in its presently-proposed form.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>