Re: Comments on IM2000

(Sorry for the long delay...been very busy!)

>On Fri, Apr 15, 2005 at 10:49:39PM -0000, James Craig Burley wrote:
>> Well, of course.  As I said, I don't have any data on how much of the
>> UBE I *don't* get is blocked by this mechanism.
>
>You might have some indication in your mail logs, if you have a good MTA.

As I said, I haven't patched it accordingly.  Silly me.  I have a new
server on order, and have long put off doing software upgrades as the
current one is a Pentium II running RedHat 7.3, etc.

>I would propose outlawing [IP addresses in mail-store pointers] too,
>for reasons described in my document (section 3). Essentially if you
>see an incoming mail notification saying to retrieve mail for
>'[email protected]' at message store 1.2.3.4, then this is too
>prone to phishing.

I'd prefer the *protocol* not disallow it, rather, that the protocol
allow a message-notification recipient to reject or ignore a
notification due to implementing a policy of not allowing IP addresses
in notifications.

Overall, I believe any *new* email system must be useful in intranet
setups as well as on the Internet itself, and therefore should be as
immune as feasible from, e.g., DNS outages, since email is itself used
to help admins diagnose such things.  (It has sometimes been useful
for me to send emails from postmaster@[192.168.0.1] to
craig@[192.168.0.72] or similar.)

Accordingly, I believe it's unwise to require any running DNS (or
/etc/hosts-style domain->IP-addr mapping) for IM2000 to operate at a
basic level.  (I realize this puts IM2000 at a disadvantage from the
very start, since a third party of some sort -- a message store -- is
required for it to work.)

Since anti-phishing measures necessarily must combat things IM2000
can't inherently combat, they will be implemented in the end-user's
MUA (if not the end-user's brain ;-) and, accordingly, can easily
include checking for suspiciously-used IP addresses.

I say, put complexity where it belongs, and nowhere else.

>It's a weakness of SPF that even if it were widely deployed, at very best it
>would just validate domain names - and you can sign up for as many of those
>as you wish.

Exactly.

>So the designers propose some bizarre distributed reputation
>system which means that new domains have to "earn" credit somehow. It seems
>pointless to me when you might as well just look at the IP address being
>used, which (unlike domain names) is a finite resource and can't be created
>at will.

Bingo.  I'm heartened to see I'm not the only one who sees things that
way!

>> Put another way: if you *aren't* using such a data base to block email
>> sent by known spammers, in an IM2000 world, you *will* be using it to
>> do exactly that.
>
>I think you'll be using a blacklist which looks up [IP address,msg store
>account]. Or, if IM2000 is modified so that the sender message store account
>*is* the sender's E-mail address, then you could look up the E-mail address
>(for accounts belonging to spammers), and separately the IP address (for
>message stores belonging to spammers).

This is item 2a on your web page, correct?

I think this presents a *huge* problem for blacklists, because
spammers can easily dDOS them by sending out bazillions of
notifications containing arbitrary sender-account IDs "tupled" with
otherwise-legit mail stores.

In my view, *any* system that depends heavily on looking up data in a
shared DB (such as DNS or a blacklist), where the "keys" for the data
are provided by an untrusted party (such as a potential spammer), is
highly prone to abuse.

There are all sorts of ways to *mitigate* the potential of abuse, of
course.  But all such ways seem, to me, to be more directly and
usefully applicable to a "vanilla" SMTP or IM2000 setup.

For example, my patched qmail-smtpd server, though it doesn't log
badmailfrom hits, does happen to log any HELO/EHLO commands that
specify a host different from the reverse-DNS lookup (and since I
don't enable rDNS lookups, that means all connections incoming from
the Internet that say HELO or EHLO!).

Interestingly, I see runs like this fairly often:

  2005-04-20 11:15:13.607144500 qmail-smtpd: 30603 @unknown[222.47.69.69] HELO slidemail.com
  2005-04-20 11:15:13.689209500 qmail-smtpd: 30601 @unknown[222.47.69.69] HELO goatlantaga.com
  2005-04-20 11:15:14.819248500 qmail-smtpd: 30605 @unknown[222.47.69.69] HELO aberystwyth.com
  2005-04-20 11:15:16.853260500 qmail-smtpd: 30599 @unknown[222.47.69.69] HELO mystarship.com

Sometimes I'll get 20 or so incoming, nearly-simultaneous, connections
from a given IP address, each claiming to be acting on behalf of a
different, perhaps entirely legitimate, domain name.  They're all
sending spam, of course (as verified by my logs, among other things).

Why would a spammer do this?  Well, I can only speculate, but I assume
there are plenty of SMTP servers out there that treat HELO as
providing a FQDN and do lookups to be sure there's some legitimacy to
the domain name.

Assuming those lookups are done on external shared DBs, that means
those SMTP servers are giving spammers an easy way to launch a
coordinated distributed DOS attack on those DBs via those servers.

Once those servers fail to respond to their clients within a certain
timeframe, the clients must, since they are acting on behalf of
connections involving timeouts, make decisions regarding how to
proceed.

So, such an attack can potentially convince some servers to falsely
classify incoming connections -- accept spam or reject legit email,
depending on how they react to external DBs being unreachable at
crucial moments.

My SMTP server does nothing but log those, and does only *internal*
lookups on domain names in envelope senders.  (I use no RBLs,
amazingly enough, out of a sorta-principled desire to experiment with
anti-UBE measures that don't let untrusted SMTP clients use my server
as a proxy to beat up external shared DBs.  Pretty much the only
useful "key" that a spammer can't forge is the incoming IP address of
the SMTP client -- but I don't do any external lookups on that either,
at the moment.)

Now, scale that up.  Imagine an IM2000 world, where message
notifications are *much* less expensive for spammers to send, where
they can point to legit message stores with impunity, and where they
can make up sender addresses regardless of their legitimacy, as long
as they aren't expecting to actually *deliver* spam that way.

What prevents spammers from gumming up the works?  Even aside from the
fact that spammers will acquire and distribute data bases of "legit"
[mailstore, senderID] tuples -- just as they widely distribute my
legit [email protected] address as a "useful" envelope sender for their
spam, thus drowning me in joe jobs and/or getting me blocked by
(theoretically clueless but practically effective) admins -- they can
make up any number of tuples that would need to be blacklisted or
otherwise identified as illegitimate.

As they succeed in gumming up the works -- by discombobulating shared
(community) DBs that blacklist all the arbitrary [mailstore, senderID]
tuples spammers can come up with -- those "works" will either be
disabled so people can receive email (opening the floodgates to UBE)
or will keep mail from being successfully and quickly delivered.

It's also not clear exactly how IM2000 will prevent spamming by
sending notifications such as:

  [im2000.comcast.net, [email protected]]

Seems to me such spam might be fairly successful in an IM2000 world,
unless sender addresses are normally hidden from users, which is not
what you appear to be advocating.

With SMTP, it's easy for a spammer to have his SMTP client do:

  MAIL FROM:<[email protected]>
  RCPT TO:<[email protected]>
  DATA
  [...]
  .

But, in addition to having to go through a DATA phase and hope the
message is accepted, at least with SMTP there's no assurance, to the
spammer, that example.com will be beaten up as an innocent third party
by recipients trying to pull down message contents, inquire about
senderID legitimacy, etc.

>> The biggest problem with this line of thought, as illustrated by my
>> expanding on your mention of web sites, is that spammers for
>> *products* available online (versus, say, Nigerian spam) are highly
>> incentivized to provide easily reachable contact points in their
>> spams.
>
>OK. But:
>(1) hardly any law-enforcement agencies seem to be interested in chasing
>them up at the moment, even given easy contact addresses;
>(2) it's obvious, but seems to have escaped the SPF brigade, that the
>envelope sender (MAIL FROM) plays *no* role at all in E-mail delivery,
>*except* when a delivery failure occurs and therefore a bounce needs to be
>returned to sender.

That'll be true in an IM2000 world, I assume.  After all, the more
"obvious" a role that a senderID, especially a [mailstore, senderID]
tuple plays in email delivery -- in terms of user visibility -- the
greater the necessity for that information to be useful by being
easily memorized and, thus, prone to forgery and abuse.

So, say spammers learn that my (legit) mailstore tuple is
[im2000.comcast.net, [email protected]].

Now, they can't *send* IM2000 spam via that tuple, because they
(presumably) can't convince the im2000.comcast.net mailstore to serve
their spam under my senderID.

But that doesn't prevent them from sending out a bazillion spurious
notifications to everyone, claiming that a bunch of emails are at that
tuple, thus wasting everyone's time as they try to read those emails.

(That'll swamp im2000.comcast.net, but only with requests coming from
*real* people -- not necessarily as a result of notifications being
sent to nonexistent addresses, spamtraps, etc.)

Some people will respond by "blacklisting" my tuple in their personal
lists.  Fine, that means they won't read any *legit* email I send them
either -- false-positive potential, just as with SMTP today.  At least
these people won't beat up im2000.comcast.net so much once they
blacklist my tuple.

Or, they will respond by telling the notification *reception* agent
that acts on their behalf to first try verifying that such a message
exists, which requires contacting im2000.comcast.net.  More likely,
such an agent will automatically do this for *any* incoming untrusted
email, regardless of how their users have configured their MUAs, and
maybe even for notifications sent to nonexistent users.

In any case, that'll swamp im2000.comcast.net such that it can't serve
*any* email, and there'll be nothing Comcast, nor recipients, can do
about it.

So, the "win" for IM2000 here is that spam can't be forged as coming
from someone else and still be delivered as spam.

The "loss" is that IM2000 pretty much requires the same sort of
"doublecheck" that people used to do with SMTP, when a server would
try to contact the envelope sender for untrusted email and see if
*his* SMTP server would VRFY that the address existed or accept a
bounce sent to that address (going as far as MAIL FROM:<> and RCPT
TO:<address> and checking the response, before discontinuing the
conversation).

Though IM2000 gives us the opportunity to improve the performance of
that sort of doublecheck, it does so only on a scalar basis (let's say
it makes it 10x more efficient), whereas spammers will have plenty of
incentive to ramp up attacks on certain popular, trusted mailstores at
a much higher rate than the performance is improved by IM2000.

Just as admins of SMTP servers "beaten" on by such doublechecks got
pretty angry at the admins who had their servers engage in such
"beatings", IM2000 mailstore admins will find themselves frustrated at
the increasing load of "doublecheck", or "prefetch" or "verification"
requests, on behalf of huge numbers of message notifications made by
spammers.

It's a vector of attack on both SMTP and IM2000, but at least with
SMTP it was fairly easy to disable the doublechecks and still receive
email.

With IM2000, the need for doublechecks is pretty much impossible to
turn off.  One can't even receive message *contents* without doing
these sorts of "reverse" lookups, or "callbacks".

This returns us to filtering *notifications* based on IP address of
the *notifying* parties, to avoid flood-style attacks by spammers and
zombies.

IMO, as long as we're going to have to do that anyway, SMTP remains an
adequate protocol, as it requires no callback to any message store at
all, even for legit messages.

>But even then, I can send my mail From: [email protected], but ask the
>person to reply to [email protected] (typically in the body of the mail).

Yes, that probably will always be with us, until humanity is replaced,
nearly wholesale, with robots that can't be fooled by such tactics.
(But that'll probably eliminate the entire pr0n industry, at which
point further development of technology will presumably cease, leaving
the our new robot overloads with nothing to look forward to in their
miserable lives.  ;-)

>But I think this is probably off the track somewhat.

I'm not sure about that.  It's important for us to look at the "big
picture" of UBE, remembering that spam and vermin target two entirely
different audiences, and avoid falling into the trap of expending vast
resources to combat what might turn out to be only a narrow, if
"obvious", means by which UBE meets its objectives, only to see
spammers expend comparatively small resources to work around the
problem.  (In general, one wants to choose tactics and strategies that
encourage or compel the enemy to expend greater resources than one
needs to implement those tactics or strategies.)

Amusingly, I've recently started receiving spam that advertises its
products via ASCII art, including the phrase "BEST PRICES"!  ;-)

>>   (Imagine if
>>   AOL occasionally sent adverts for its services to all known non-AOL
>>   users on the Internet, from [email protected].  Nothing
>>   *technical* can stop this, if the message is designed to get past
>>   the content filters of the day, because nobody can really block all
>>   email from aol.com and still be said to be "using" email.
>
>They could, however, block all mail from [email protected] (again, this is
>much easier under IM2000 than SMTP, unless you are able to assert that all
>mail coming from aol.com's servers has a genuine MAIL FROM address. If the
>spammer were to send using a series of different <account>@aol.com
>addresses, then you wouldn't be able to do that).

That was my point; I used salesguy237 as a "userID of the day" kind of
userID.

Just as AOL can gin up arbitrary userIDs that, necessarily, start out
as "legit", spammers can do the same for any legit mailstores they
know about, and of course they can make even "better" use of any
illegitimate mailstores they create, for as long as those stores are
viewed, by enough of the population, as legit.

>> It does raise the issue of, how serious is the spam problem *now*,
>> *really*, such that we contemplate moving to a whole new system that
>> will still apparently require many of the same "augmentations" (RBLs,
>> whitelists, blacklists, etc.) that we are already using with SMTP in
>> order for IM2000 to actually stop spam?
>
>I think the problem is serious, because every remote site I talk to
>implements spam filtering differently. You may implement a reasonable set,
>but many sites implement stupid sets. Sometimes my mail is bounced;
>sometimes it is blackholed. The spam problem is causing a crisis of
>confidence in E-mail; it is simply becoming less and less reliable as a
>medium of communication.

That may be true.  I'm not sure why I'm not seeing that on my own
personal end of things, but I'm not a typical "remote site".

I tend to think the biggest problem with SMTP is really the fact that
it (normally) prevents a server from accepting a message without also
accepting *responsibility* for that message -- which translates into
the need for generating a bounce.

I believe we can reduce or even eliminate the need for bounces, over
time, by introducing IM2000-like concepts into SMTP, but *without*
introducing (except as a per-message-negotiation option, I suppose)
the "pull" and "separate mailstore" concepts.

>> That is, we can theoretically eliminate X% of spam if we know exactly
>> *who* spams and *reliably* identify any incoming email (via SMTP or
>> IM2000) as coming from a known spammer.
>> 
>> Now, the pertinent question becomes, does requiring a sender to
>> provide a message store make *that* much difference, in terms of our
>> ability to reach the theoretical maximum for X, compared to what SMTP
>> is evolving to, in terms of putting practical requirements on senders
>> to inject a message from a source that is not immediately identifiable
>> as a known source of spam?
>
>I think it does.

The industry as a whole would need a better answer than that before it
spends $$$ to deploy IM2000 as an SMTP replacement.

>Right now if I receive a mail via SMTP then only thing I *know* for sure
>about the mail is the source IP address of the SMTP sender. And of course,
>the RCPT TO address must be valid, otherwise I wouldn't get it.
>
>But *everything* else in the conversation is not trustworthy; that is, the
>spammer can choose to put whatever they like; and if I implement a filtering
>policy which is widely used, they can put whatever is necessary to bypass my
>filter. This includes the envelope sender, all the message headers and body.
>
>But if I can only reliably blacklist on IP address, I am limited when it
>comes to shared mail relays at ISPs. If 1 million users are all relaying
>through one ISP's mail relay (a common case), plus 10 spammers, I either
>have to blacklist the entire mail relay and suffer collateral damage, or I
>have to accept the spam.

Right, I understand all that.  But that is simply a Very Hard Problem
anyway; as I pointed out earlier, even the "hotpop"-like ads in the
headers of otherwise-legit email could be defined as "spam".

And, as I point out above, if you resort to blacklisting [mailstore,
senderID] tuples of "known spammers" at otherwise-legit mailstores,
you either:

  - Overload the blacklist(s) you're using with all the spurious
    senderID's that spammers can make up, or

  - Overload the mailstore(s) with requests to validate whether any
    given senderID a spammer might invent actually has a legit account
    there (equivalent to SMTP VRFY or RCPT TO)

This problem could be mitigated by insisting that outgoing message
notifications go *through* (and thus be vetted by) mail stores, so
only "legit" email addresses are exported.

But that can be done for SMTP anyway, and in fact is done, sorta
kinda, in various ways.  E.g. some ISPs deny ordinary users outgoing
TCP connections over port 25; filter incoming connections to their
internal SMTP servers to ensure MAIL FROM:<whatever> identifies an
addy the customer (identified by incoming internal IP address)
actually owns; provide AUTH access for external use by legit
customers; etc.

In essence, an IM2000 mail store is a third party required for all
email transactions.  Either the sending party is somehow required to
first obtain permission from the third party prior to notifying a
recipient of message availability, or the sending party can cause a
receiving party to flood the third party with spurious requests.

In the former scenario -- sending party obtains permission from third
party, which is more than just whatever handshaking is needed to
actually store a message on that message store -- either the recipient
needs to be able to confirm that such permission was granted without
necessarily having to make a request from the third party (which is
technically possible, by the third party mailstore signing something
that the receiving party can check against a distributed DB like DNS),
or the recipient might as well be designed (by IM2000) to accept
notifications from *only* that third party anyway.

In that latter case -- accept notifications from only the third party,
not from the sending party -- we're mostly back to an SMTP relaying
world anyway.  Certainly the sending party can just use SMTP to inject
the message into the third party.  From there, it doesn't make all
that much difference, that I can see, whether the third party and the
recipient use SMTP or IM2000 to handle the message.

>Note: it should be clear that SPF doesn't help at all here.

Agreed.  SPF is, to me, like DomainKeys, of interest only after a
message has already been "vetted" by a human, or nearly-human, user
such that the "next step" is to try to figure out whether it was
forged.

Stopping incoming forged communications "at the door" is an extremely
expensive proposition, and gives the "enemy" an easy weapon to use
against you -- it's an overreaction to worry about whether each and
every communication might be "forged", in reality as well as on the
Internet.

E.g. I don't really care whether you actually are Brian Candler,
because I'm not investing anything in the proposition that you are who
you say you are.  So resources I might otherwise expend to validate
your identity are spent on things I think are more worthwhile, such as
considering your viewpoints, responding to them, etc.  You could be a
very intelligent German Shepherd, for all I know (but, on the
Internet, nobody knows etc. ;-).

>IM2000 also wins because it's easy to build message stores which rate-limit
>the number of messages sent per day from a particular account; it could be
>enforced as good practice (i.e. if you offer free signups for new accounts
>over the web, then you get your message store blacklisted if you don't
>follow this principle)

I'm not sure why that's hard for SMTP -- aren't some ISPs doing that
already?

>These things *could* be implemented using SMTP:
>
>- All ISPs could disable outbound relaying by IP address. All customers
>would be required to use SMTP AUTH to use the mail relay. This would require
>all customers to change their MUA configuration, and a flag day at each ISP
>where relaying by source IP address alone is disabled.

I don't see how this is different in an IM2000 world.  Only when one
customer "forges" a message as coming from another customer is
authentication presumably required as a countermeasure, and that
problem exists for both SMTP and IM2000, assuming IM2000 works by
users putting messages in a central message store managed by their
ISP.

>- All ISPs could carry forward the AUTHenticated user information in their
>SMTP sessions (RFC 2554):
>    MAIL FROM:<[email protected]> AUTH=wibble123
>
>- When you receive an incoming SMTP session, you can send the IP address and
>AUTH ID to a trusted third party (i.e. blacklist) to verify
>
>- If the IP address belongs to or is controlled by a spammer, then the
>message is marked as spam
>
>- If the IP address is a genuine mail relay at an ISP (which the TTP has to
>validate for themselves), and it provides AUTH information about the account
>which injected it, then only mail from certain AUTH accounts will be
>rejected.
>
>However, this ain't going to happen. The pain is huge, and the short-term
>gain is negligible.

How will the equivalent *not* have to happen with IM2000?  Seems to me
it'll have all the same problems, unless IM2000 means we'll move to a
sorta-centralized, few-but-highly-trusted-mailstore world, which is
not really different from everyone using a few highly-trusted SMTP
relays.

>However, an alternative mail architecture (e.g. IM2000, or something else),
>has a value proposition to offer:
>
>"Reconfigure your own MUA to use the IM2000 service, and immediately all
>mail you send to other IM2000 users will be delivered more reliably and in a
>way that the sender cannot be forged".
>
>That means, a business can get all its employees to move to the new
>architecture, or a group of friends can all change to the new architecture,
>and all get an immediate gain. Then lawyers and banks may start to use it.

Okay, then this is not a case of *technical* advocacy, more a case of
saying "it's a whole new system, we think it'll be fairly spam-free
compared to SMTP" -- and in fact it probably will for some time -- and
hoping to convince enough early adopters to jump on board, quickly
enough, that everybody else, except spammers, switches soon after.

I don't think that's anywhere near convincing enough.  I've had a
similar sort of mindset all my life, in that I used to discount the
"installed base" issue and other forms of institutional inertia.

Experience has *almost*, but not quite, beaten that out of me.

Hence, despite my serious distaste for SMTP, my greater distaste for
UBE impels me to ultimately frame all my "grand ideas" in terms of
"how can this idea be implemented in SMTP in a gradual fashion?", even
though I do also think them through in terms of "what if we were
starting from scratch?".

(You'd probably be amazed at the language and OS designs that are
floating around in my head these days.  In many ways, completely
unlike what we have today.  But hardly farfetched -- mind-numbingly
simple, in fact.  ;-)

>> And, especially with naive end users, the best thing you can do for
>> them, to convince them that "offers" are really scams and/or spams, is
>> to present those emails on their screens not intermingled with
>> otherwise-legitimate ones, but grouped together in a way that tells
>> them "this appears to be junk, and I [the content-analysis bot] have
>> put all the 419-ish stuff in one chunk, all the
>> enlarge-your-weiner-ish stuff in another, and all the notifications
>> that you've just won a lottery in yet another".
>
>That's essentially what SpamAssassin does. The problem is that the spammers
>have equal access to the SpamAssassin source code, and so customise their
>mails to bypass the rules.
>
>It is done more effectively by third-parties which maintain their own
>private rulesets. There are companies which offer this service: point your
>MX servers at us, we'll filter your mail for spam and viruses, and send it
>on to your mail server after cleaning. They work extremely well, but they
>are expensive. This is an example of the *direct* economic cost of spamming.
>If you want effective filtering, you either have to maintain your own
>private rulesets (cost in time and expertise), or you have to use a third
>party and pay them in cash.

In the long run, we'll need personal information agents acting in ways
like this on our behalf anyway.  It's inevitable, as technology
increasingly floods us with *legit* info, never mind the illegit
stuff.

In any case, IM2000 does not, in any way, shape, or form I can see,
dispense with the need for content analysis.  As an "illegit"
mailstore first comes online, or a [legit-mailstore,
evil-spammer-using-it] tuple springs into existence, *somebody* will
have to take the first steps towards discovering that email springing
from that source should have the Evil bit set.

And that requires retrieving the message contents and then analyzing
them somehow.

Further, in order to avoid everyone else having to do the same exact
thing, it requires notifying other white hats about the problematic
mailstore or tuple, so that they may be blacklisted.

IM2000 ultimately puts us in the position of hoping not just that this
will be required *less* often, as a % of overall email sent, but that
the economics will ultimate in spammers giving up and going away.

I'm not seeing a clear case being made for that scenario coming to
pass, certainly not clearly enough such that it justifies the
deployment of IM2000 as it now stands.

>> >They *will* happen, and soon (i.e. within months at most), as surely as the
>> >widespread implementation of MAIL FROM domain checks made spammers change to
>> >sending out mail with real (but forged) E-mail addresses as the envelope
>> >sender. The cost is minimal.
>> 
>> Right, it's an arms race, we all know that already.  None of what you
>> have said on that topic is news to me; I doubt it's news to anyone
>> else here.
>
>All I'm saying is, multiplied across the Internet, it's (a) a huge waste of
>time and effort, and (b) is contributing to the decreasing reliability of
>E-mail, which I consider to be a very serious problem.
>
>People think they are doing good, when they are doing harm.

But IM2000 *requires* it, in essence: envelope senders specify domain
names that *must* be looked up, and thus must have obtainable,
verifiable DNS information, must not be in various blacklists, etc.

So IM2000 gives spammers a *guaranteed* weapon to use against DNS and
blacklists that SMTP doesn't assure them, because not all SMTP servers
do doublechecks (various forms of reverse lookups on sender-supplied
data, which can be arbitrary) against external DBs.  Mine doesn't, for
example.

>Validating the EHLO domain name is another case in point. It's futile.

Pretty much.  Though, I really should get my server to double-check
EHLO domain names against my badmailfrom file and log matches, as
they'd indicate cases where the SMTP client actually *announces* that
it is acting on behalf of a known spammer (that is, the owner of the
domain name -- not just a domain name abused by spammers)!

I mean, as long as I've got the data base and use it for envelope
senders, might as well use it for EHLO, eh?  Wonder if it'd catch
anything....

>Admittedly the RFCs are badly worded in this area, but the whole point of
>EHLO is to give a sender-supplied cookie which can be included in the
>recipients log files for tracing. It need only have local significance.
>
>If I get a message saying "EHLO winsrv01" that gives me useful information,
>should I have to contact the site which is sending me the mail to try and
>iron out a problem. If you enforce that the EHLO name *must* be a DNS name
>which matches the IP address of the connection, then it has lost all
>usefulness; you can simply lookup the IP address yourself in the DNS.
>
>This sort of stupidity really bugs me, sorry :-)

IMO HELO/EHLO are kinda stupid anyway, and another win for IM2000 is
that they go away.  ;-)

I think IDENT is in the same boat.  It's simpler and more direct --
installed-base and badly-designed-OS issues aside -- for the upstream
system to do its own validation and logging of outgoing TCP
connections its users/clients initiate (ultimately) via its outgoing
network interface.

>> >> No, as is widely recognized.  It's a temporary tactic, one which
>> >> raises the bar in terms of increasing the expense of sending UBE (and
>> >> the visibility that one is doing so), which is kinda the whole point
>> >> of IM2000, right?
>> >
>> >No.
>> 
>> Then what *is* the point of IM2000, when the various web pages
>> promoting it proclaim that it puts more of an economic burden on
>> *senders* than does SMTP?
>
>I think they are making a wrong assertion. However there may be *other*
>advantages that a new mail architecture can have, which make an environment
>_permanently_ harder for spammers to work in.

It might also make an environment in which it is easier for spammers
to destroy the ability for the rest of us to work in, as I've
suggested above.

>Perhaps [adoption] could also be achieved by starting a new, parallel SMTP
>network, using port 26 say. The new network would *require* certain
>operational practices to be in place (like using SMTP AUTH to submit mail).
>At least you wouldn't have to rewrite any software, although you'd need some
>sort of MX record replacement to identify people who can use port 26 to
>receive mail.
>
>But it's still not going to be as strong. For example, counting the number
>of mails sent by a particular user is still quite hard. Consider an SMTP
>relay cluster of 10 machines; they would need to communicate with a central
>database to count the number of mails sent by one user over 24 hours.

How is that different from an IM2000 mailstore cluster of 10 machines?

>And you'll still need a new blacklist infrastructure for looking up
>[IP,auth-sender] instead of just IP address.

Sounds like item 2a on your web page!  Again, how is it different?

I think IM2000 is great as a proposal.  I don't think it wins as a
design.

What it suggests is that SMTP could evolve to incorporate IM2000
concepts in ways that *allow*, but don't *require*, clients and
servers to cooperatively use them.

A huge win of IM2000 is the (presumed) ability for a sender to
(repeatedly) notify a recipient of the availability of a message
without having to send that message with any particular notification.
This saves bandwidth in almost all cases of exchanges of legit email
messages, and of many involving UBE as well.

That doesn't necessarily *require* the recipient to be able to "pull"
the message from the sender (sender's message store).

Instead, the recipient could notify the sender that it was finally
ready for the message to be *pushed*, a la SMTP's DATA phase.

Similarly, an IM2000 sender is able to "remind" a recipient that a
message is still available, without having to send the message.

In SMTP, one can't do that, because all the recipient sees, before the
DATA phase, is the envelope sender/receiver pair.  There's no
transmission or message ID they can use to uniquely identify a
particular [envelope sender, envelope recipient, message] combination,
other than the message content.

That facility *could* be added to SMTP in an incremental fashion --
not requiring any particular ordering of upgrades, across the board or
vertically.

And, in SMTP, a recipient can't (normally) accept the contents of the
message without also accepting responsibility for the contents,
whereas IM2000 allows retrieving a message without "unpinning" it.

This is useful for subsequent anti-UBE analysis, for prioritization,
for a user to read and then decide how to dispose of the message, etc.
And IM2000's key advantage here is that, while such analysis goes on,
the *sender* (well, his mailstore) remains responsible for it and for
storing it, increasing the costs of senders of UBE.

But, in fact, an SMTP server can do this by convincing the SMTP client
to consider the delivery attempt to have failed due to temporary
error.

One way is for it to issue a 4xy response code after the end of the
DATA phase.  That arguably tells the client the message will *not* be
"delivered", but it doesn't actually mean the server can't do whatever
it wants with the contents of the message.

Another way is for the server to simply never issue a response to the
DATA phase, leaving the wave function uncollapsed, so to speak.  The
client can't assume the message was delivered, nor can it assume it
wasn't.  It'll timeout the connection, unless the server simply closes
it first.

Either way, the server is free to do whatever it likes with the
content, even though it has not accepted responsibility for it.

So, it can, as a result of downstream analysis/disposal, respond
similarly or differently to future delivery attempts -- or treat the
lack of a second, third, or subsequent delivery attempt as, itself,
signaling something "interesting" about the email.

This obviates the need for a server sending bounces to innocent third
parties in cases where the server simply doesn't ever accept
responsibility for an email until it is absolutely sure that it will
either deliver the message to a grateful recipient or drop it
entirely.  4xy and 5xy response codes leave responsibility for the
message with the injecting client, as does simply never responding to
the end of the DATA phase.

(This is not unlike IM2000's advantages that you identify as 2c and
2d, perhaps among others, on your web page.  E.g. by the time the
subsequent delivery attempts might have resulted in the server
accepting responsibility for the email, the server has potentially
learned that the source, or an upstream source, is a known spammer.)

A big problem with "abusing" SMTP this way is that downstream entities
(relays, IMAP/POP3 boxen, MUAs, end users) don't understand the
concept of a message being available but not the responsibility for
it, so they don't know how or why they should notify upstream entities
that they do or don't wish to accept responsibility.

IM2000 poses the same challenge to downstream entities, of course, so
we're going to have to upgrade those entities anyway if we intend to
deploy IM2000.

Another technical advantage of IM2000 is that a recipient can notify a
sender immediately upon the recipient accepting responsibility for the
message.  This is a nice feature that allows rapid reuse of storage
among cooperative agents, as well as rapid notification of sender that
the message has indeed been received.

SMTP doesn't allow this at all -- the server must wait for the client
to decide to connect again, and, even if it does, there's no elegant,
consistent means for a recipient to notify the sender that the message
has been accepted, read, whatever.

Generally, SMTP doesn't offer any benefits that require out-of-band
recipient->sender communications, though pertinent upgrades are of
course possible with SMTP (at substantial expense).  SMTP does have
DSNs; these don't reliably notify *senders*, given the forgery (or joe
job) problem, and they're not elegant, but they're vaguely like what
IM2000 would provide out of the box

On the other hand, IM2000 is less reliable to the extent it *requires*
such communications, because the communications path a recipient must
use to contact the sender might not be available at the time the
communication is required.

The most glaring example of this is exposed by the "vanilla" IM2000
message-pull model, in which the end user decides to read an IM2000
message and the system can't pull it up because it is sitting on an
unreachable message store -- one that might have been reachable when
the notification was first sent.

To the extent SMTP is extended to *allow* this sort of communication,
it exposes itself to similar sorts of failures.

But at least SMTP users will have more flexibility in deciding just
how reliably they want certain message contents to be made available
to them.  Whitelisted senders can be allowed to push messages via the
classic SMTP method, including the server accepting responsibility for
them, thus eliminating at least one, almost certainly two, potential
points of failure for each such delivery.

IM2000 users won't have the choice of making sure senders "push"
messages to recipients to ensure that subsequent reverse lookups
aren't needed and thus can't fail -- *unless* IM2000 is redesigned to
be a cleaner, more flexible SMTP.

And I am entirely in favor of a cleaner, more flexible SMTP, even if
it's called IM2000, and even if "uptake" remains a problem.

But I prefer to *conceptualize* in terms of a brand-new, clean system,
and then *actualize* in terms of incrementally improving SMTP in that
direction, in order to speed uptake of whatever I might think is
useful.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>