Re: Comments on IM2000

>On Thu, May 05, 2005 at 11:27:33PM -0000, James Craig Burley wrote:
>> >> On the SMTP side, it's not all *that* hard to prioritize incoming
>> >> messages based on a recipient's perception of the trustability of
>> >> upstream relays, including whether SMTP AUTH was used.  (But in cases
>> >> where use of AUTH isn't reliably reported in "Received:" headers,
>> >> either the relay in question always requires AUTH and thus acquires,
>> >> over time, lots of trust, or will have to find other ways to be sure
>> >> it avoids the problem of being a mixed source of UBE and desireable
>> >> email and, thus, untrusted.)
>> >
>> >The trouble is you can't rely on the sender *telling* you that the message
>> >was submitted via SMTP AUTH, and that therefore you should trust it. If so,
>> >spammers would just make their mail systems make the same declaration.
>> >
>> >With SMTP, you need to verify independently whether the mail server in
>> >question is actually trustworthy - for example by having trusted agents on
>> >that ISP's network performing the testing for you. Ick.
>> 
>> I'm saying, why care whether a third-party relay uses AUTH or any
>> other *particular* mechanism?
>> 
>> All you *really* care about is whether that relay is well-run enough
>> to avoid accepting more UBE, likely destined for your system, than you
>> feel comfortable with.
>
>OK, sure. But then the question is - how do you *evaluate* that?

That's the central question: how do you evaluate whether a third party
(a relay or mailstore) is *really* doing a good job of preventing UBE
from being intermixed with legit email?

I don't believe there's much point in doing it by analyzing the
specific tactics used or choices made by that third party, because
that amounts to micromanaging its operation.

>It's kind-of done now, but it works best for objective tests which can be
>made from outside - e.g. "is this mail server an open relay?" If the test
>fails you blacklist them, and if the IP owners wants to be removed, you can
>easily re-test.

Open relay tests had their short-term utility in the past, but are,
conceptually, of little use.  It doesn't *really* matter whether a
third party is an "open relay" in the technical sense; what matters is
whether it openly relays a substantial % (versus all email it relays)
of UBE.

Pro-actively searching for, and then blacklisting, open relays wasn't,
IMO, fruitful because they were open relays -- it was fruitful because
the *fact* that they were open relays strongly suggested that their
admins were lax about securing their systems against abuse by internal
or external entities, so blocking an open relay served to get the
admin's attention to a growing problem.

From a reliability point of view, the fact that an important email can
be freely and quickly routed through one or more immediately available
(and thus perhaps "open") relays is a *plus*.

And with a better-designed email transport protocol, a relay could be
truly open and yet still have plenty of incentive to be sure it isn't
exploited to relay UBE.  (For one thing, as more anti-UBE tactics such
as blocking based on source IP are able to look "through" the IP of
the connection client, which might pertain to a reasonably trusted
open relay, to the IP of its upstream injector in the topmost
pertinent "Received:" header, and block the message based on whether
*that* IP is in the blacklist.)

>Things like "does this mail server owner manage their users properly?" is a
>lot more woolly. Some blacklists will just blacklist any IP even if only one
>or two pieces of spam are seen from it (regardless of whether millions of
>non-spams are sent). But establishing "well run" involves either talking to
>the users of that ISP, or going to that ISP and performing tests, or talking
>to the ISP themselves, or otherwise gathering indirect evidence and weighing
>up the probabilities.

Right.  So I don't see SMTP AUTH as an especially persuasive solution,
nor its slow uptake as a particular problem.

>The sort of things we're talking about are:
>
>1. The mailserver operator knows the customer identity of every piece of
>mail submitted into the system.
[...]
>2. This customer identity is carried forward *in the SMTP session* when
>relaying to another host (e.g. as AUTH= parameter)
[...]
>3. The mailserver limits customers to sending a small number of messages per
>day, unless the customer has established themselves as a bona fide sender of
>large amounts of mail. [Same here; both SMTP and IM2000 could be extended to
>indicate the number of messages sent recently by the same customer]
>
>4. The ISP takes measures to limit the number of free signups [same]
>
>Looking at this list, I think you're right that most of these weigh
>similarly for IM2000 as for SMTP.

Right.  More generically, the issue boils down to, do you trust an
immediately upstream (but external) relay/mailstore to give you
reliable information (even if rendered opaque via a hash or unique ID)
identifying the entity immediately further upstream?

If you do, you can be more generous about accepting incoming email
from that entity even if you might believe some of it could be UBE, as
long as *you* can learn to distinguish among *its* upstream sources
with regard to whether *they* send UBE and/or you can, via some
reliable means, notify the immediate upstream source that its own
upstream sources are sending UBE (or even ham, since positive feedback
is helpful too).

>> So, let the sender track *all* those messages, as you suggest.  Now
>> they're expending much more, in the way of their resources, per
>> *important* outgoing message than the typical sender of legitimate
>> email, *both* of whom get to shovel the *content* across the wire the
>> first time they try.
>
>I still don't much buy "expending _much_ more"
>
>I thought a bit more about this last night. In order to bypass 4xx
>greylisting systems, I only need to store one *bit* of information for each
>recipient!

Yup.  But storage isn't the issue; it's cheap as beans.

>With a little more care I could arrange for my resends to occur at set
>intervals which more accurately mimick a real mailserver (e.g. spend 15
>minutes delivering messages, then rewind and attempt to re-deliver the
>failed ones from the first block, before moving on). It would then become
>very difficult to 'fingerprint' a spam sending program based on its retry
>intervals.

*If* you assume that the typical spammer will have sufficient
incentive to write and/or buy specialized software to inject huge
numbers of emails into the system, then you're effectively assuming
that sending spam *can* be cheaper than sending ham:

  - It's somewhat cheaper to send a message and not track delivery at
    all; but recipients can easily notice that.

  - It's a bit cheaper to trivially track delivery than to do so in
    the usual fashion; but recipients might notice that pattern as
    well.

I don't disagree that *some* spammers would use such tactics.

I'm saying, I'm assuming spammers can, ultimately, easily afford to
use whatever off-the-shelf software implements my proposal.  In fact,
it's inherent in my proposal that an implementation *will* offer a
"blast-it-once-and-forget-it" mode to cater to senders of SBE
(Solicited Bulk Email) who are sending messages that, if they aren't
accepted and read after one delivery, no biggie (maybe they are simply
notifications of temporary conditions).  So spammers will have a range
of tools to use, from the beginning, without having to pay $$ for them
up front.  (A controversial idea, to be sure, but, again, it's
inherent, and it helps me avoid the dangers associated with assuming
anything akin to Security Through Obscurity in developing my
proposal.)

That is, they can easily afford, simply because of economics, to buy
big-enough iron to send, say, 1G emails a day, without needing special
spam software, because the vanilla email software works just fine and,
naturally, from the point of view of the recipient, does not appear to
exhibit any suspicious behavior on a per-piece basis.

What I think spammers *can't* afford is to keep this software running
for days or even weeks as it continues to track the status of
deliveries, while the authorities are hunting them down as a result of
so many of those 1G emails making it into spamtraps, and while so many
admins have already updated their *local* IP blacklists so that
messages coming from them are *never* confirmed as having reached the
recipients' eyes.

So, spammers will still be in a hit-and-run business, and *if* they
use specialized software to do el-cheapo delivery, that'll be easy
enough for ordinary receiving software to notice.

But after they've hit and run, the fact that they "ran" will be almost
as evident as it would be with IM2000, except they'll have already
made copies of outgoing messages for recipients to review, to submit
to content analysis, to submit to authorities, etc.

If spammers can flourish despite all that, the *only* way I can see
that IM2000 would stop them is if it *required* the sender provide a
domain name for callback *and* if domain names were locked down even
more tightly than the IPv4 address space.

The former makes IM2000 less reliable as a mail protocol than either
SMTP or my proposal.  The latter ain't gonna happen; in fact, it is
more likely that the IPv6 address space, or something similarly large
and intractable, will ultimately replace the IPv4 space (or,
equivalently, that the IPv4 space will become more complicated,
insofar as more and more addresses will be NATs and thus "mixed
blessings").

>> With IM2000, if that email has not actually read by a real person, it
>> cannot be read at all, as the sender has moved on.  Once a few MUAs
>> detect that, they can notify the "collective" that *all* pending
>> notifications from that source are suspect.  Here, responsibility has
>> not been accepted, but it's not clear whether false positives are
>> involved (the "moving on" might have been a legit laptop sending legit
>> email being disconnected or given a new IP address), so there's a
>> problem with the fact that the messages aren't actually available to
>> the reader (or a content-analysis engine).
>
>Yes, this is a problem with IM2000, and it quite strongly discourages the
>use of mailstores on dynamic IP addresses. It could be achieved with dynamic
>DNS, but that's unlikely to be a reliable solution. IM2000 mailstores *need*
>to be online whenever someone tries to fetch mail from them, in the same way
>that currently a POP3 server *needs* to be online when someone tries to read
>their mail from it.

My IM2000 mailstore would be on a dynIP address, if it existed today,
since that's how my system presently is hosted.  I've had three, maybe
four, distinct IP addresses over the past two or more years.  So it
isn't a *big* additional point of failure, but you're right that it
*is* one.

In "my" world, my dynIP host is a fine candidate for sending outgoing
email to other hosts.  Even AOL might someday accept it directly,
under my proposal, since it'll be easier to distinguish my hosts'
"mail-care behavior" from those of 0wned machines, which necessarily
run specialized software that tries to stay under the radar of the
machines' true owners.

>But it would be hard to infer from the non-availability of a mailstore that
>it's not legitimate. There are plenty of legitimate but poorly-run
>mailservers and networks out there already.

True, and that statement is, IMO, the kiss of death for IM2000, since
non-availability of a mailstore means non-readability of mail!

>> With my system, until that email is *fully* delivered, the sender
>> retains responsibility for it.  "Moving on" can therefore be
>> legitimately interpreted as no longer demonstrating interest in it.
>
>I think that's the same. If you care about your message, you'll deposit it
>in a well-connected mail relay on a fixed IP address.

Not necessarily.  *I* will try to inject the messages I care most
about directly to the SMTP server listed in the MX record for the
receiving domain.

Why?  Because I trust *my* system setup more than I trust it *plus* my
provider's SMTP relay, because I can more easily track the progress of
outgoing SMTP sessions, and because I can therefore more quickly know
when to try another method to get my message to its destination.

Remember, a third party necessarily knows *less* about the "profile"
of a message's importance.  So the frequency of its tracking requests,
the choices it makes regarding how quickly to try alternate routes,
and so on, is unlikely to agree with the choices *I* might make for a
given outgoing message.

Yes, a protocol could allow such a profile to be relayed along with
the message itself, in the hopes that the third party (your
"well-connected mail relay") will honor it.  But it might not be
willing or able to do that, even if all the protocol goo is correctly
designed and implemented, which is not a trivial task in the first
place.  (Essentially, such a protocol ends up being little different
from "here's a bash script to have crond run every N minutes
until...".)

Whereas, as long as the outgoing message is being delivered and
tracked *locally*, there's little difficulty in an MUA providing
buttons such as "Track", "Redeliver", "Deliver Via Alternate Route",
and so on, so its user can *personally* care for an outgoing message,
if the default tracking policy for that MUA isn't suitable for that
particular message.

(That is, ultimately, what *I* want as an end user sending email.  I
want buttons and displays so I can directly track and muck with my
*personal* outgoing messages -- I'm not speaking as a sysadmin here --
so I want my *MUA* injecting messages directly to the SMTP server
listed as an MX, even if it simultaneously injects them to a local or
remote, e.g.  upstream, MTA, should direct delivery fail before I
disconnect my laptop.  And I want a similar degree of control over
incoming messages as a reader -- I want to be able to decide whether
to accept or reject messages and on what basis, and "bounce" them or
whatever my upstream MTA can do on its own.)

>> In the extreme (but likely frequent, in today's world) case, if an MTA
>> simply decides to drop *all* pending email from a given source, it can
>> do so in a fashion that either lets the sender discover the fact by
>> issuing new tracking requests (which either fail or yield "discarded
>> by server") or resending (which the MTA might /dev/null but not
>> disclose accordingly).  Such mail is never "lost", however, since it
>> was never truly *sent*, in that responsibility was never
>> *transferred*, in the first place.
>
>That means the recipient can see status as:
>( ) Not transferred
>(*) Transferred but not accepted (yet)
>( ) Rejected
>( ) Accepted

Yes.

>Effectively in SMTP, you have 1 (your SMTP relay may send you a warning if
>the message is still on its queue), 3 (a bounce), and 4 (successful
>delivery). State 2 is new, and a bit more woolly. The message contents have
>been transferred - but if I'm genuinely interested in delivery, should I
>attempt to transfer them again? Or is it sufficient just to keep polling for
>status given a tracking ID?

The former in current SMTP, the latter in a new protocol (SMTP++ or
something entirely new), for as long as the sender is satisfied with
the progress of the delivery overall.

Since the system relies more on dup detection and elimination (which
becomes nearly trivial under an entirely new protocol), the sender
*could* decide to attempt to transfer the message again, but via a
different path (say, to a backup MX, even to a different recipient,
such as to a user's home address instead of a work address, in case
they are working at home).

While it's tempting to frown on multiple deliveries using different
paths, anyone interested in *reliable* and *immediate* delivery of
emails should welcome a system that *allows* that sort of thing from
the get-go, and leaves it to market forces to decide under what
circumstances it's used.

>If the majority of mail ends up in this state, then under your proposal the
>sender of the mail will need to keep polling, just to prove that they are
>'still interested'.

No.  That might *indicate* such interest.  But if the sender isn't
genuinely interested in the email to *that* degree, there's no need to
track the delivery, is there?  A quick answer to someone's question on
an email list is really more of interest to the *recipient* than the
sender, so the sender doesn't really care if it reaches the
destination.

And, to be clear: with my ecrulisting proposal, only ecrulisted email
ends up in this state, but with my *full* proposal, email will
*usually* end up in this state, unless it's coming from whitelisted
senders, which I'm trying to not rely on to assume a workable system
(though it's trivial to add).

>That will be a big workload increase for all legitimate
>mail

I've already addressed that in detail.  In short, it is *hardly* a big
workload increase, if compared to coping with the uncertainty of not
knowing whether delivered email reached its destination, with bounces
coming back, and especially with joe-job bounces.

Also, my new proposal can be implemented less expensively by making it
behave a lot more like vanilla SMTP, if desired, on a per-site basis.
(So, a message is deemed "accepted" once it reaches the equivalent of
a POP3 or IMAP box, even though the MUA or user who fetches it might
later decide it's spam.)

But I'm trying to think in terms of the messaging facility we
*ideally* want to use, as end users, for the next several decades, as
millions more new users come online, as existing users improve their
behavior, and so on.

>but spammers can easily match the raised bar too.

Addressed previously.  True for all designs; *inherently* least true
for my proposal, since spammers have to send *all* message contents at
least once (despite the vast majority of recipients being unlikely to
ever read them), as with SMTP but not IM2000, and would have to then
track *all* deliveries in a typical fashion to avoid recipient's
detecting lack of interest in incoming email from a previously unknown
source, which IM2000 provides in its own way, but not SMTP.

In short, I believe my proposal actually *lowers* the bar for senders
and recipients of legitimate email compared to today's SMTP (vanilla
SMTP + all the $#@!% that's going on to try to stop UBE), and spammers
have to work harder to match that *lowered* bar -- and they won't be
able to do so as economically as they've been with SMTP and as they
probably would with IM2000 (which raises the bar for *everyone*, due
mainly to requiring third-party mailstores, besides requiring all-new
software).

>I might need to
>allocate *two* bits in my memory bitmap to properly record the state of each
>message! However, if I have to remember a tracking-ID which was allocated by
>the recipient, then that's some more work. I might end up keeping a spool
>file on disk containing the state of each message. At 100 bytes each, my
>100M messages might require 10GB of disk space to track (but only if they
>all end up in this state; not-yet-transferred, rejected and accepted
>messages don't require it)

Again, this might all be true, but I'm not basing my proposal on the
assumption that spammers can't afford the iron and connectivity to
send *their* payloads via my proposal.

>That does however suggest a sort of super-greylisting: instead of sending
>back a tracking ID, if the recipient is suspect I send them back a huge
>cookie (say 4KB of random data) and ask them to retry at least 2 hours
>later. When they retry, they must return the same cookie. I don't waste
>storage, because I just keep a cryptographic hash of the data, but the
>sender is required to hold on to 4KB of crud for that period.

Yes, recipients can play all sorts of fun games like that, especially
with a new protocol, which I have partially designed in my head with
an eye towards just that sort of fun.  ;-)

For one thing, legitimate responses to a tracking request might
include:

  - No response (always an option for recipient)

  - Not yet read (an obvious one)

  - Read (another obvious one)

  - Responsibility accepted by recipient (obvious)

  - Lost contents, please resend (of course, but will sender resend?)

  - Never heard of it (maybe never sent?)

  - Send future tracking requests to [set of hosts...]

  - Please resend message through one or more of [set of hosts...]
    (like a per-delivery, or even per-tracking, MX)

  - Send SHA1 of message (to double-check local contents are correct)

With responses such as these, which senders can legitimately ignore or
honor with less worry over complications of the sort we got into
discussing IM2000, recipients -- especially multiple independent, but
cooperating, recipients -- can play all sorts of games with senders to
make sure they *really* want to get their messages through.

Now, off-the-shelf implementations would probably implement senders so
they automatically honored such responses to all such tracking
requests, and *possibly* in a way to allow some degree of local
(sender) control over the extent to which such responses are indeed
honored (e.g. "don't bother resending this message, I don't want to
chew up my outgoing bandwidth with these pics").

So spammers would have to either tune these implementations so they
kept their resource utilization to a minimum as recipients began
figuring out they could try making the spammers' systems dance for
them, in which case they'd risk more recipients quickly figuring out
that they *are* sending spam (for one thing, they don't dance!)...

...or they'd run them full-blown, in which case they'd give recipient
systems much more control over their systems' resource utilization
than SMTP presently allows.  (It'd be in IM2000 territory.)

And this is all without letting *senders* control *recipients* any
more than they do with SMTP (and even a bit less) or as much as they
might be able to do with IM2000 (since mailstores, including those run
by spammers, can twiddle their thumbs and the like in response to
requests to the stores).

That is, *none* of the responses listed above require the recipient to
keep much of anything in the way of state around for the message.  No
open TCP connection is required, for example, until a sender responds
to, say, a "send SHA1 of message" response with a new connection
saying "here's the SHA1 of message #dshjk453dsk, which you asked for".

However, the design could certainly allow for *all* such exchanges to
take place within a single TCP session, and I'd probably design it
that way, as well as to allow a hit-and-run-style delivery where the
entire message, including a *sender-provided* ID, are thrown down a
newly opened connection and then the connection is closed by either
party.  (That would be super-fast delivery indeed, AFAIK.  Could even
be done via a UDP packet in many cases.  Interested senders woud
presumably resend the same message once or twice more, and/or send a
tracking request, though not necessarily.)

>It's still not expensive enough though - 4KB times 1M recipients is only
>4GB - and yet the extra sending and receiving of 4KB of data is costly to
>the recipient as well as the sender.

Ah, but the recipient can decide they have excess bandwidth and CPU,
perhaps even to pretend they have many more users than they do.  *I*
certainly do that, basically without trying -- you should see my email
logs -- yet my old Pentium II 233MHz is keeping up with it without
breaking a sweat.

We do this with SMTP and would presumably do it with IM2000 as well.

But with my proposal, receiving email is cheapest of all, so it's more
"rewarding" for sites to have spam traps, to have those traps behave a
lot like real users, etc., in terms of the low cost of doing this
*versus* the higher cost to spammers.

>> There are difficult aspects to my proposal.  With a new, clean design,
>> the challenges are mostly coming up with a design that won't have
>> aggravating aspects down the road; with SMTP, it'd be annoying to have
>> to take into account problematic aspects of that infrastructure,
>> including a biggie, the difficulty of uniquely identifying any message
>> *transmission* regardless of the *path* that message takes to reach a
>> given MTA.
>
>You can get the recipient to allocate a unique ID, which is unique locally
>to them, such that [recipient,ID] is globally unique. When the message is
>relayed you'll get another pair; the sender then has to associate the
>recipient's ID with their own ID, potentially forming a chain. This would be
>what happened if you received a packet by FedEx and resent it using DHL :-)
>
>Or you can just get senders to allocate IDs, so that [sender,ID] is a fixed
>unique key. That's what Message-ID: is supposed to do, although guaranteeing
>uniqueness is hard. <Message-ID:, submitter-IP-address, submitter-identity>
>should be enough, and if kept in this form, would give useful information
>for evaluating the source too.

Right.  Again, that's in the details.

I think I should restate my essential *economic* point here, which was
buried in my previous email.

In my view, the right approach is *not* just making it inherently more
expensive to send UBE.  If that was the right approach, then the
solutions would be obvious, and in fact are often put forward: require
AUTH, require $$ to send email, require using one of a few
well-trusted relays/stores, etc.

In fact, I don't really care whether it becomes *super-cheap* to send
UBE.  As far as I can tell, it's *already* so cheap that it'd be
nearly impossible to *practically* raise the cost by any factor to
make a useful difference.

The economic reality is that senders of UBE depend on having lots of
recipients willing, eager, and *able to afford* to buy products
advertised by UBE.  So, while senders want sending UBE to be as cheap
as possible for *them*, they also have an interest in making sure that
*receiving* email is cheap enough for their target market, and in not
themselves being drowned out by others offering apparently identical
services.  (This almost perversely argues *for* IM2000, in that
spammers won't find it a particularly rewarding environment -- but
only because few *legitimate* users will find it so.  ;-)

Accordingly, they really don't care all that much about the
"cheapness" of sending UBE.  What they do, as a *group*, is simply
select the least expensive way they believe will reach the widest
possible target audience of *buyers*.  If that's running some
specialized software via 0wned machines, they'll do that; if it's
running ads in newspapers, they'll do that.

And, naturally, if sending UBE is free, then sending non-UBE is likely
to be free as well.  It's important, for spammers, to be sure their
messages are *intermixed* with "desireable" messages, just as
advertisers prefer putting ads in newspapers over distributing
standalone flyers, putting commercials in popular TV shows over
running infomercials on obscure cable-TV channels, and so on, all else
being equal.

(The email equivalent of this might be those advertisements I
mentioned earlier, included in headers for emails sent via "free"
providers.  If we move towards a model of having only a few
well-trusted relays or mail stores, there'll be economic pressure for
those stores to offer free accounts that result in such intermixed
advertising, as in "Received: from brian by ... brought-to-you-by
Pepsi <http://www.pepsi.com>".  I don't want this future.  ;-)

So, suppose sending UBE is essentially "free", in terms of iron and
connectivity costs.  The real expense comes with running the business,
owning *useful* domain names or other points of presence to sell the
goods being advertised, and avoiding legal trouble.  (For legitimate
businesses, the latter is not necessarily less of an expense, sadly;
but that's another issue.  The main thing is, senders of legit email
don't have to worry about having to constantly relocate in IP space as
well as meatspace.)

In a free-to-send-UBE world, spammers can send, say, 1T (1K*1G) pieces
of UBE each and every hour if they like.

What's the *real* problem with that?  Two biggies:

  1. The infrastructure handling *receiving* emails might be unable to
     cope with the load, so legit email wouldn't get through.

  2. Those reading email have trouble finding the few emails they
     really care about among the huge amounts of UBE they (mostly)
     don't, so legit email isn't always read.

Focusing on #1, I say let's think in terms of a system that makes
*receiving* email as cheap as possible, at the (possible) expense of
the sender.

That changes the balance of economics back to being in favor of
recipients, but it also potentially changes it in favor of senders of
legitimate email.

Why?  Because senders of legitimate email already have a demonstrable
interest in a recipient reading their email (in most cases), otherwise
they wouldn't send it or consider it "legitimate".  So their
expenditure-to-desire ratio ends up being smaller, in relation to
spammers, with my proposal than with SMTP or IM2000.

Now, the easiest and cheapest way I can think of to receive an email
is to accept an *incoming* connection delivering it, after which I can
take my own sweet time deciding what to do with it.

I don't want to have to look up any return address to provide status
information (as in a bounce or DSN).  I don't want to have only a
short window of time in which to respond definitively or risk
duplicates or redelivery (as with the response to the SMTP DATA
phase).  I don't want to have to respond *at all* to the delivery or
to the sender's subsequent tracking requests.

How can it get any cheaper than that?  Yet, even if I don't do any of
the things above, I can still receive a complete message, display it
for an end user via an MUA, and the user can act on it as she sees
fit.

Now, SMTP doesn't quite fit the bill, but comes reasonably close.
IM2000 doesn't fit it all that well either, but at least I can choose
to not even receive a particular message if I don't want (though
there's some question whether this is practically different from
SMTP), the downside being that, if I *do* want it, *I* have to go to
the trouble of looking up the "mail store" in my phone book,
connecting to it, etc.

My proposal, especially as a brand-new protocol, would make
*receiving* email fundamentally as cheap and easy as possible,
assuming a deterministic universe (no mind-reading allowed ;-).

On top of that *foundation*, all the stuff we presently add, or might
add with IM2000 -- possibly even the option of a recipient of a
message notification saying "don't send me the message, tell me where
I can retrieve it [a la an IM2000 message store]" -- can be added, as
desired.

But, the foundation is now tilted as much as possible in favor of the
"recipient class", without in any way deliberately hurting the
"sending class" (although recipients might choose to make senders jump
through various sorts of hoops, of course, as they have more control
under this system than under SMTP).

Now, here's the *crucial* advantage of the economics of my proposal.

If it's cheap as possible to be a member of a recipient class, and if
it remains reasonably cheap to be a member of a *sending* class, then
market forces will continue to favor making email an inexpensive, yet
reliable, form of nearly-immediate communication...

...but the most expensive "membership" in such an environment would be
to that of the *bulk* sending class.

This is the central insight that led djb to propose IM2000 (and others
to propose nearly-identical systems) -- that bulk sending needs to be
inherently more expensive, *relative* to sending and receiving
non-bulk email -- but it makes the relationship between sending UBE
and sending or receiving *all* other kinds of email even *more* tilted
against the sender of UBE.

As the expense of ordinary exchange of email drops across the board,
that frees up capital for other things, such as improved anti-UBE
measures, improved vetting of new users by ISPs to be sure they
haven't previously engaged in spamming, etc.

Meanwhile, the expense of sending (or relaying) UBE goes *up* with my
proposal, because either more and more outgoing messages must be
redelivered more times (SMTP with ecrulisting, greylisting, etc.) or
must be tracked to try and convince recipients they really are
important (with my proposal), either of which implies a *longer-term*
commitment to sending email (a commitment which a sender of legit
email inherently makes nearly 100% of the time).

Under that design, senders of UBE find themselves expending more
resources to be sure their able to send to only "live" people, not
spamtraps; to demonstrate long-term interest in their outgoing email,
so daemons monitoring mailboxes on behalf of users who only
occasionally check their mail won't tell MUAs for other, more active,
users that there appears to be an overall lack of interest
demonstrated for email sent from IP address x.x.x.x; and to avoid
being arrested for spamming by virtue of the fact that they're having
to stay online longer in order to demonstrate all that interest for
all that outgoing email.

This won't eliminate UBE.  Nothing will -- or, if it can, it can with
my proposal in place at least as well as with SMTP and probably if
IM2000 was in place.

Instead, UBE will be forced, over time, to go more "upscale", in that
we might still see Unsolicited Email that is less "Bulky" (it's more
targeted) advertising higher-margin products (real Rolex watches
instead of fake ones ;-), etc.

Such economic pressures should make UBE, as a whole, less prevalent in
legitimate users' mailboxes -- which, besides lowering the costs to
run legitimate email operations, is the whole point.

So, when I think in terms of the three fundamental technologies under
discussion -- SMTP, IM2000, and my design (which evolved purely from
these lines of thought; I didn't just "invent" it and then try to
figure out how to justify it, believe me!) -- I try to picture them
both "naked" and "clothed" with similar anti-UBE measures, in order to
compare apples to apples.

By "naked" I mean no blacklists, no spamtraps, etc., a world not too
different from Internet email circa, say, 1987.

But I add, to the MUA, the ability for a recipient to indicate "I read
this message, [save/discard/print] it, it's legit" versus "I haven't
really read this message yet, even though it's staring at me on the
screen", versus "I've decided this message was Unsolicited [and is
maybe the 100th message I've seen just like it, so it's probably
Bulk]".  This ability is becoming more important anyway; reasonable
defaults would be provided, and might even change based on "regions"
of an in-box (lower-priority regions, being mostly UBE, would perhaps
default to "I haven't really read this message").

And I assume it's generally important for a *sender* to know whether a
recipient has actually had an opportunity to see the message, so the
sender can decide whether to take some other approach to being sure it
gets to the recipient (phone call, avian carrier, etc.).

SMTP doesn't fare well here due to opaqueness to the sender and lack
of control by the recipient.  Only some traditional guarantees
regarding bounces mitigate that somewhat; but *recipients* aren't
required to "bounce" messages they don't like or don't get around to
actually reading, even though they're sitting in their in-box.

IM2000 fares pretty well.  It also pretty much demands this extra
level of sophistication for MUAs and end users (or at least enough of
them to make IM2000's assumptions concerning anti-UBE measures work).

But, in terms of transport *quality*, my proposal gives the sender and
the recipient end-to-end communication regarding the progress of the
message, and the recipient *full* control over such communication.

Neither SMTP nor IM2000 come close to that, though in certain corner
cases, each offers a superior benefit (which my proposal can obviously
offer as extensions to its foundation; they're just not
*foundational*, or *inherent*, to it).  For examples, SMTP offers the
sender the prospect of a rapid bounce in certain useful cases; IM2000
offers the sender the prospect of being pro-actively notified, by the
recipient, that it will accept responsibility for a message
("unpinning" the message).  In my proposal, both notifications don't
occur until the *sender* sends its next tracking request, in terms of
the foundation of the system.

And, in terms of transport *costs*, my proposal is cheapest for the
recipient, by a fairly substantial margin; for messages that go unread
for a time, potentially much less expensive for *senders* as well
(since they need only track, not re-send, messages).

(I'm relying an important distinction between SMTP and my proposal
here.  With SMTP, you usually don't get a bounce if a message makes it
into a POP3 or IMAP mailbox, even if the users' MUA decides it's spam
or the user never gets around to reading it.  Such bounces would
improve the transport *quality* of SMTP but, if widely-enough
implemented to do that, would greatly increase the *costs* of SMTP.
So, comparing apples to apples, SMTP would have to provide such
bounces across the board to equal my proposal, even though it doesn't
do so today and is thus "cheaper".)

Since my proposal obviates the need for bounces as does IM2000, joe
job are eliminated, and each host's capacity for handling incoming and
outgoing email can be speced based on *actual* expectations for
*legitimate* outoing email plus legitimate incoming email plus a
certain amount of incoming UBE.

There's no need to spec for receiving joe-job bounces; no need to spec
for hosting a mail store; no need to spec for sending lots of bounces
after accepting too much email when blacklists are unreachable; etc.

Just spec for (on top of merely sending and receiving legit message
contents) tracking legit outgoing email and for accepting and
processing tracking messages for incoming email.  And tracking
requests are small, cheap, and not usefully forged in either direction
(or so I hope!).

Of course, since there are no bounces, the overall expense of the
average message remains more in line with its *existence*, not its
*size*, as there's never a need to *return* the entire message, or
even a portion of it, to a sender, as bounces are designed to do.

Most of all, though my proposal still *allows* for "hit-and-run"
deliveries, which are the norm for SMTP but don't work for IM2000,
they *aren't* the norm, and senders are in fact given, and encouraged
to use, tools to demonstrate ongoing interest in outgoing emails,
which will tend to separate the spammers from the rest of the senders
unless the spammers decide to "normalize" in terms of technology and
presence on the Internet.

Remember, "demonstrating interest" is not intended by me in the sense
of hashcash or similar ideas -- it is not a way to *artificially*
increase sender costs.  It is part and parcel of a sender's legitimate
interest in tracking the progress of an outgoing message, assuming the
sender really has such an interest.

If the sender has no such interest, no such cost is paid, and
legitimate messages from popular senders (offering advice, or
responding to previously sent emails) will tend to fare well in
delivering their messages, while unknown or unpopular senders (such as
spammers) will not.

(False positives aren't a problem with my proposal compared to SMTP,
since a message is, presumably, never fully accepted and *then*
discarded as spam, unless that's what a recipient *wants* to do.  SMTP
servers accept and silently drop messages all the time these days.)

But if the sender has a legitimate interest, demonstrating that
interest by tracking the message is a reasonable, low-overhead way
versus being prepared to receive a bounce.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>