Re: Comments on IM2000

>On Wed, Apr 20, 2005 at 06:01:19PM -0000, James Craig Burley wrote:
>> I think this presents a *huge* problem for blacklists, because
>> spammers can easily dDOS them by sending out bazillions of
>> notifications containing arbitrary sender-account IDs "tupled" with
>> otherwise-legit mail stores.
>
>Sure, any blacklist is subject to that kind of attack. Blacklists which run
>on top of the DNS benefit from distributed caching.

I've previously posted, at some length, about the dangers of relying
on distributed (DNS-style) caching as an effective solution to the
problem of giving attackers direct ability to trigger database lookups
(and especially inserts) based on arbitrary keys.

(IMO, distributed DNS-style caching is to this sort of attack as more
CPU and RAM are to a brute-force implementation of an NP-complete
problem such as TSP as the problem sizes increase.)

>Dealing with junk notifications is one of the problems I have with the
>IM2000 approach. Even without blacklists, you will need to validate them
>somehow (e.g. by doing a callback to the originating message store, or by
>some cryptographic method). You don't want a blast of fake notifications to
>drown out legitimate ones.

Yup.

>With SMTP you can of course open large volumes of SMTP connections to hosts.
>This may trigger some sort of validation process (RBL lookup, sender
>verification callback etc). I think the trick is to keep to a minimum the
>amount of work which a random attacker can cause the system he/she is
>talking to (and any referenced third party) to perform.

Yup.  A key insight, I think, is that *most* attacks are designed to
cause a recipient to read spammer's mail.  That is, they are rarely
designed to cripple the ability of a recipient's system to *receive*
email.  Instead, attacks are directed at components of the system upon
which a recipient system might rely to distinguish welcome from
unwelcome email (that is, to designate UBE).

Therefore, it seems best to try to design any email system such that
the receiving system relies as little as possible on external
entities, especially third parties, to decide whether to accept, or
how to prioritize, incoming email.

IM2000 clearly violates this design goal.  That doesn't mean it's
fatally flawed; it just means a whole lot of thought and effort must
go into protecting those third parties (mail stores, blacklists,
etc.), since they'll be easy to attack without necessarily destroying
the ability of recipients to read email under normal circumstances.

>> My SMTP server does nothing but log those, and does only *internal*
>> lookups on domain names in envelope senders.  (I use no RBLs,
>> amazingly enough, out of a sorta-principled desire to experiment with
>> anti-UBE measures that don't let untrusted SMTP clients use my server
>> as a proxy to beat up external shared DBs.  Pretty much the only
>> useful "key" that a spammer can't forge is the incoming IP address of
>> the SMTP client -- but I don't do any external lookups on that either,
>> at the moment.)
>
>That's a shame. You ought to try RBLs, because they are surprisingly
>effective even in the current SMTP world, and to evaluate how blacklisting
>through DNS works in practice.

I don't question that.  "Everyone else" uses RBLs.  So I don't think I
have a whole lot more to learn, and therefore to add to the pool of
expertise, by using them.  (My priority isn't so much to *stop* UBE
coming into my system as to *study* it.  Secondarily, I want to slow
it spreading to other systems down by actually letting more of it
enter my system without bothering me, personally, or anyone else.)

>One notification generates one callback or lookup, so it's not a DoS
>amplification service. The spammer won't actually achieve anything, since
>the notification will be discarded having failed validation, so there isn't
>anything to be gained.

Yup.  If there's a protocol by which a 3rd party can say "you've sent
me too many misdirected callback/lookup requests, please throttle it
down", that might be helpful.

>> It's also not clear exactly how IM2000 will prevent spamming by
>> sending notifications such as:
>> 
>>   [im2000.comcast.net, [email protected]]
>> 
>> Seems to me such spam might be fairly successful in an IM2000 world,
>> unless sender addresses are normally hidden from users, which is not
>> what you appear to be advocating.
>
>I did suggest that (sender fingerprints), but I think it's better that the
>sender address *is* a genuine E-mail address, and that the notification is
>discarded unless it is proved that this notification did originate from that
>E-mail address. That gives phishing protection too.

I tend to agree.  In fact, *personally*, I really don't mind seeing
short spammy messages in notifications, as long as I can quickly skim
and skip them, and as long as I'm aware they didn't consume much in
the way of my system's resources to receive them.  I think most users
would also be willing to live with that.

(The spammier the notification, the more certainly I know to not
bother reading the message that accompanies it.)

This leaves the problem of phishing, but it might actually be easier
to teach people that "messages" themselves do *not* actually, or
legitimately, arrive as *notifications* themselves -- that
notifications should *only* tell them about the existence of messages.

(In short, there are to be no email equivalents to postcards, where
the message is written on the outside of the "envelope".  Or, perhaps
the equivalent to a postcard could be expressly designed into the
system, as it does have utility as a concept, as long as users are
easily able to understand the implications of sending and receiving
messages in that form.)

>> With SMTP, it's easy for a spammer to have his SMTP client do:
>> 
>>   MAIL FROM:<[email protected]>
>>   RCPT TO:<[email protected]>
>>   DATA
>>   [...]
>>   .
>
>Errm, but the recipient address won't exist, so there will be a 550 error
>after the RCPT command, and the DATA command will be rejected.

Whoops, good catch!  Lemme try that again:

   MAIL FROM:<[email protected]>
   RCPT TO:<[email protected]>
   DATA
   [...]
   .

Does that make more sense?

I'm not sure how any useful email system blocks such messages, and I'm
not really sure *why* it should.  Especially as standalone
IM2000-style notifications, it seems like a case of diminishing
returns to try to erect a defense against ordinary users seeing ones
like this such that they actually try to visit the web site in
question.

("Overhead" and "bandwidth" cease to be valid concerns when such
notifications are so cheap and easy to send to legit mailboxes.  If
too many spammers send them at once, end users will quickly tire of
them, never do the hand-copy-paste-edit thing into their browser, and
the market value of such spam will plummet.  If spammers send such
spam to too many spamtraps, their source IPs will be easily blocked
and pending notifications from them safely and easily dropped.  All of
this applies to SMTP email as well, except that unless something like
greylisting is used, the recipient's system has to receive the message
contents even if the end user would choose to skip reading it.)

>> Some people will respond by "blacklisting" my tuple in their personal
>> lists.
>
>No, clearly you cannot blacklist a notification based only on the
>(potentially forged) information it contains.

Whew, okay, but I hope *everyone* (who might implement or admin
IM2000) gets that point!

>You could however blacklist notification sources based on their IP address.

Which takes us back to where we are right now, since we can (and do)
do that now...?

>There may be people sending notifications from dynamic IP addresses, but
>that's unlikely; legitimate IM2000 message stores will be on fixed IPs, and
>the notifications will *normally* come from the message store itself.

Right.  I'm still wrapping my head around IM2000's potential
architectures and designs.

>However, the IM2000 protocol explicitly allows notifications to be
>forwarded, which is a useful capability.

More than useful: critical.  It's not clear IM2000 will fly if it
doesn't do full-blown relaying a la SMTP, but certainly the
notifications themselves need to be relayable.

>That's why I think it's probably better to validate notifications to prove
>they came from the claimed message store (and discard them if not), rather
>than maintain a separate IP-based blacklist of spurious notification
>senders.

Ah, okay.

>Spurious notifications are unlikely to be sent often, since they don't
>achieve anything apart from a waste of resources, so I think it's unwise to
>build a separate infrastructure to filter them out, one which won't be
>tested very often.

In essence, a notification becomes a handshake between a message store
and a recipient (agent).  That could be handled solely via a TCP
connection a la SMTP, in which case the client's IP address can't be
(easily) forged.

>> In any case, that'll swamp im2000.comcast.net such that it can't serve
>> *any* email, and there'll be nothing Comcast, nor recipients, can do
>> about it.
>
>For each one notification message sent (TCP connection), the recipient will
>open one TCP connection to im2000.comcast.net

Might as well have im2000.comcast.net itself notify via TCP connection
to the recipient?  What is gained by waiting for the recipient to
decide to try to validate the sender, if we're assuming recipients
will always want to do so?

Oh, well, one thing: when recipients trust the sending IP, for
example, then there's no need for a validation, so no need for
anything other than an incoming UDP packet containing a notification.
(And maybe a UDP packet in return saying "got it", so no further
notifications need be sent?)

This is getting down below the level of detail we need to understand
the ramifications, I tend to think.  You seem to understand the need
to minimize amplification, so I'm less worried about it being a
problem in any system you design (or vet).

>The attacker could just open those connections herself, and apply a similar
>kind of DoS anyway. However, this approach *does* hide the source of the
>attacker.
>
>Perhaps, when contacting im2000.comcast.net, the validation request should
>include the purported source IP address of the notification? Clearly this
>could be forged too, but if you get a lot of notifications from people all
>over the Internet and they all claim to have received notifications from the
>same IP, you have good evidence of the source of the attack.

Yup.  I've long (and I mean for *decades*) assumed this kind of
automated determination of trustability based on sufficient numbers of
*reasonably* trusted neighbors would be necessary for goodness to
thrive and evil to be vanquished on a hostile global network.

>As I say, this attack won't achieve anything (such as the delivery of a
>spam), only wasted resources - so eventually the attacker is likely to get
>tired. But I do agree with you that this needs to be considered carefully,
>and the best solution arrived at such that this approach is at least no more
>efficient than DoS'ing the message store directly.

We're definitely on the same page here.

>> So, the "win" for IM2000 here is that spam can't be forged as coming
>> from someone else and still be delivered as spam.
>> 
>> The "loss" is that IM2000 pretty much requires the same sort of
>> "doublecheck" that people used to do with SMTP, when a server would
>> try to contact the envelope sender for untrusted email and see if
>> *his* SMTP server would VRFY that the address existed or accept a
>> bounce sent to that address (going as far as MAIL FROM:<> and RCPT
>> TO:<address> and checking the response, before discontinuing the
>> conversation).
>
>Except that at the moment, an SMTP callback only verifies that the address
>exists, not that the mail you're trying to receive was sent by that person.

Well, yes, but what I was referring to was that such callbacks were
considered highly annoying by (sysadmins for) victims of joe jobs,
weren't they?

IM2000 can (and IMO will) make them *cheaper*, but will they be cheap
*enough* to be sure *all* sysadmins will find such callbacks, which
appear to be intrinsic to a deployable IM2000 (*replacing* SMTP),
acceptable?

(I have to phrase these as *questions*, since that episode apparently
predates my awareness of how SMTP works, my experience as an SMTP
server admin, etc.  I've only read some short descriptions of it.)

>> Though IM2000 gives us the opportunity to improve the performance of
>> that sort of doublecheck, it does so only on a scalar basis (let's say
>> it makes it 10x more efficient), whereas spammers will have plenty of
>> incentive to ramp up attacks on certain popular, trusted mailstores at
>> a much higher rate than the performance is improved by IM2000.
>
>Maybe. People do DDoS attacks on all kind of Internet infrastructure, for
>all kinds of reasons. It's a pain, but you're not going to fix that without
>fixing the Internet itself.

The main problem, besides widespread lack of upstream propagation of
filtering rules in our infrastructure, is when packets coming in from
a given IP address represent an arbitrary *mix* of "good" and "bad".

If IM2000 implies callbacks on all message notifications, then it
implies spammers can attack the callback mechanism *directly* by
sending spurious notifications to a large-bandwidth site, with such
notifications purporting to be from a small-bandwidth site.

Once the small site is drowned by the large site's overwhelming stream
of callbacks, such that it either ignores (or responds negatively or
positively to) *all* callback notifications or tells the large site to
stop sending notifications for awhile, the spammers have either
succeeding at DDoS'ing all IM2000 email sent from the small to the
large site, or at getting the large site to accept *any* IM2000 email
sent to the large site and *purporting* to be from the small site.

This is why I actually *want* an infrastructure that encourages email
to be sent directly from even dynamic-IP hosts (like laptops) to their
recipients, with as few intermediaries as possible, and without
relying on callbacks to validate email.

(When I think "callback" I think "potential blowback".  I want to be
*sure* that the former can't be turned into the latter.)

>> Amusingly, I've recently started receiving spam that advertises its
>> products via ASCII art, including the phrase "BEST PRICES"!  ;-)
>
>Cool. You can't accuse them of lack of imagination. It's another reason why
>(IMHO) content analysis will always be doomed, until we have genuinely
>intelligent robots. And if they are that intelligent, they will wonder why
>they are doing this job in the first place. "Filter my spam, Marvin" -
>"Brain the size of a planet and they ask me to filter out spam..."

Heh.  Or, content analysis can decide it can't understand the content
sufficiently and return it to sender (somehow) with a message "please
express this succinctly, in plaintext [English|French|Klingon]".

>> I tend to think the biggest problem with SMTP is really the fact that
>> it (normally) prevents a server from accepting a message without also
>> accepting *responsibility* for that message -- which translates into
>> the need for generating a bounce.
>
>That's pretty much essential in a store-and-forward model though. In the
>current world, all MUAs are set up to send outbound mail via a
>store-and-forward server (and indeed, blacklists are set up to *refuse* mail
>sent directly from end-user IP addresses, so replacing MUAs with something
>cleverer which followed MX records in the DNS would not work)
>
>Once you've handed the message over to the S&F server for delivery, if it
>fails, it needs some way to tell you so. That need not be in another E-mail
>message, but if not then there's an out-of-band notification required; and
>to prevent this channel from being used for spamming, it needs to be
>unforgeable.

Right.  Though, my dynamic-IP host actually tries sending directly
first, and automatically falls back to my upstream ISP's SMTP server
only if that fails (in certain ways).  (I wrote the patch that makes
qmail-remote work that way.  It became pretty popular among some qmail
users in boats a lot like mine.  It's comforting to see most of my
outgoing messages go directly to the MX for a recipient, in my mail
logs, instead of always going upstream to a relay, at which point my
ability to track delivery status evaporates.)

Anyway, what if *responsibility* does not accompany relaying as you
describe?

In that case, the MUA can't expect a bounce or DSN to ever come back
in case of failure (or success).  Which is pretty much just as well,
from a design point of view, because DSNs on *successful* delivery
aren't really working well enough in SMTP to be useful; waiting as
much as two weeks for a bounce that might never arrive is not a
reliable way to be sure an important message actually reached the
destination; the infrastructure to allow bounces to be sent back is
expensive, as we've been discussing; and when that infrastructure
isn't working, even a successfully-delivered message can't result in a
DSN announcing that success.

Instead, what if the MUA periodically contacted the recipient (or the
same store-and-forward, or upstream, relay) and asked it "how is
delivery of that message progressing?".

It would have to allow not only for answers equivalent to the
2xy/4xy/5xy responses we presently have, but for "it appears to have
been lost, please resend", since the MUA would have responsibility for
that message.

That would *eliminate*, by design, the *need* for any sort of callback
system (although one could still be provided, and used on an
as-needed, optional basis).

Add (to a mythical new system a la IM2000) a more reliable way to
detect and eliminate duplicate message deliveries, and an MUA can
handle "important" deliveries by simultaneously delivering an email to
its upstream relay *and* the ultimate destination, in case the
connection to one or the other times out, or the delivery fails or is
rejected on one side or the other, before the laptop running the MUA
is disconnected or powered off.

On the other end, as the duplicate is detected and eliminated, it
could be noted that it appeared to have been *legitimately* duplicated
in order to indicate importance to the *sender* and, perhaps,
prioritized accordingly for the recipient.  (If you receive the same
message via two routes, one of which is direct from a dynamic IP host
and the other is via a trusted relay, you can be pretty sure it's a
legit message even if you've never heard from the sender before.)

On the other hand, a site-wide receiving agent might notice a ton of
*similar* messages sent to all sorts of addresses for which it is
responsible -- including spamtraps -- all containing URLs like
enl4rg3y3r0rg4n.example.com.

In that case, it could assign all those messages very *low* priority
for all its recipients.  Or it could just delete the messages and
remember that the originator sent what appears to be UBE.  (Remember,
it wouldn't have accepted *responsibility* for those messages; just
the notifications and the contents.  So it can just "drop" messages.)

Meanwhile, without accepting responsibility, all incoming-email
servers are able to accept messages without ever sync()'ing them to
disk, among other way cool stuff.

There would need to be substantial changes in email infrastructure
throughout to properly support such a system.

As it happens, many similar improvements have to be made to properly
support IM2000.

>> >But if I can only reliably blacklist on IP address, I am limited when it
>> >comes to shared mail relays at ISPs. If 1 million users are all relaying
>> >through one ISP's mail relay (a common case), plus 10 spammers, I either
>> >have to blacklist the entire mail relay and suffer collateral damage, or I
>> >have to accept the spam.
>> 
>> Right, I understand all that.  But that is simply a Very Hard Problem
>> anyway; as I pointed out earlier, even the "hotpop"-like ads in the
>> headers of otherwise-legit email could be defined as "spam".
>
>Ah well, then you have a single E-mail with a split personality: part
>legitimate, part spam. If you dislike it sufficiently, then you can refuse
>to communicate with that person, or ask them to use a less obnoxious E-mail
>service.

It's an extreme case of the mixed-source problem I mentioned earlier,
so it is illustrative, yet insoluble in the general case.  (Whether a
reader considers the non-spammy part of the message sufficiently
worthwhile to have to see the spammy part is *purely* an end-user
decision.  Nothing short of pure AI can reliably and correctly make
that decision; even human beings can't make it, in some cases, until
long after they've read the message.)

That is, to the extent we (say, IM2000 enthusiasts) assume we're going
to move to a few, "trusted", intermediaries for exchanging email --
whether those intermediaries are IM2000 mailstores or SMTP relays --
we have to accept that there *will* be a mix, tolerable to some but
not all, of advertising accompanying messages coming from some or all
of those intermediaries.

It helps me remember to doubt any "promise" that anyone makes that
some new system, filter, whatever, will "eliminate spam", because I'll
always remember to say "what, exactly, *is* spam, and how is that
system or filter sure it'll eliminate only what *I* consider to be
spam?".

IM2000 and, I believe, ecrulisting or something along those lines,
both really promise only to make the "B" part of the "UBE"
sufficiently more costly, relative to the "E" part, that genuine spam
becomes much less prevalent.  (No proposals, of which I'm aware, truly
deal with the "U" aspect of "UBE", even though techniques like
whitelisting and greylisting touch on it.)

>> And, as I point out above, if you resort to blacklisting [mailstore,
>> senderID] tuples of "known spammers" at otherwise-legit mailstores,
>> you either:
>> 
>>   - Overload the blacklist(s) you're using with all the spurious
>>     senderID's that spammers can make up, or
>> 
>>   - Overload the mailstore(s) with requests to validate whether any
>>     given senderID a spammer might invent actually has a legit account
>>     there (equivalent to SMTP VRFY or RCPT TO)
>
>I don't see that *necessarily* follows, nor that the overloading you're
>proposing is any worse than a direct DoS attack from the spammer to the
>message store or to the blacklist.

A direct DoS attack is *also* a way to overload any central blacklist.
That's why it's unwise for a SMTP server to *rely* on any external
blacklist to accept incoming email, unless it can afford to not
receive incoming email.

But with a traditional DoS, a blacklist server can presumably block
all packets coming from a particular IP or IP range for the duration
of the DoS, and still serve requests from other portions of the 'net.

To really do damage, a spammer needs a DDoS -- an attack coming from
all sorts of 0wned machines all over the 'net.

But even *these* machines can be detected and blocked by the blacklist
owner as, over time, the pattern of the DDoS becomes distinguishable
from the pattern of requests from legitimate sources.

Requests regarding blacklisting from *legitimate* sources, however,
can be used to DDoS the blacklist provider in ways that *it* cannot
possibly resist, without becoming useless *as* a blacklist provider.

For IP-based BLs, this is somewhat mitigated by the fact that the
space of potential IPs is limited, and once a blacklist knows an IP
address is a source of bad requests, it can tell its clients to
blacklist that source *entirely*, meaning it won't even see incoming
*notifications* from such sources.

IM2000-based blacklisting would have to get pretty involved to be able
to do stuff like that.  E.g.:

  - 0wned machine stuffs spam into message store

  - Store notifies recipient

  - Recipient asks blacklist about [store, sender] combo

  - Blacklist detects attempt to DDoS it via such requests, so
    gets IP address of 0wned machine from recipient somehow

  - Blacklist suggests store not accept further notifications
    from recipient for a period of time

  - In meantime, blacklist stops adding anything to its data
    base regarding [store, *] notifications originating from
    0wned machine

I'm not saying it ain't possible.

I *am* seriously suggesting it's *so* intricate that we might as well
do *half* this much work to make SMTP work just as well, if not
better, than IM2000 might work even in a "perfect world" of 100%
deployment without worrying about coexisting with SMTP.

Keep in mind that once you design callbacks into a system, you
essentially turn *every* system that sends email into a sort of
blacklist server, since it must accept and deal with callbacks, which
in turn are used to deny acceptance of forged email from spammers.

Therefore, *any* such system is potentially a target for attack just
as would be any of *today's* blacklist servers.

I am really, really worried about rolling out any design that makes
ordinary servers (like mine) targets of DDoS-like attacks by spammers
via *legitimate* hosts.

>> This problem could be mitigated by insisting that outgoing message
>> notifications go *through* (and thus be vetted by) mail stores, so
>> only "legit" email addresses are exported.
>> 
>> But that can be done for SMTP anyway, and in fact is done, sorta
>> kinda, in various ways.  E.g. some ISPs deny ordinary users outgoing
>> TCP connections over port 25; filter incoming connections to their
>> internal SMTP servers to ensure MAIL FROM:<whatever> identifies an
>> addy the customer (identified by incoming internal IP address)
>> actually owns; provide AUTH access for external use by legit
>> customers; etc.
>
>It can be done, but in general it isn't.

This is the line of reasoning you use to which I most object.

"Can be done but isn't" => "*IM2000* can be done but isn't".

Sysadmins and the suits who make the decisions aren't looking for a
whole new system to roll out.  They're looking for *incremental*
improvements to the current one.

What is it about IM2000 that makes any of us think it'll be the
"killer app" that replaces SMTP?  That's what I think this discussion
really boils down to, beyond the (IMO highly useful) overview of
various email issues.

(This discussion almost belongs on a metalist discussing various
potential and existing messaging systems, but I don't know of any
offhand.)

>Outgoing relays would have to
>become more complex to perform AUTH database lookups, and huge customer
>bases would have to be migrated over to using SMTP AUTH. Only when the last
>customer has switched to using SMTP AUTH, and relaying by source IP address
>only is disabled, can the source be considered 'trustworthy'.
>
>And then there's the problem of which E-mail addresses a particular user is
>allowed to use; either many local databases will have to be built, or one
>distributed one.
>
>As an example, I login to smtp.example.net as user "foo123", and then I want
>to send outgoing mail with MAIL FROM:<[email protected]>.
>
>Either:
>(1) The ISP maintains a local list which ties foo123 <=> [email protected]
>Every time a new user comes along, they will have to state which E-mail
>addresses their account is allowed to use, *and* prove to the ISP's
>satisfaction that they do in fact own those accounts.

This isn't a problem if the envelope sender is required to be
[email protected].

If it can be anything arbitrary, then IM2000 will have similar
problems, correct?

>(2) There is a global database. As owner of [email protected], I ask the
>pobox.com domain owner to publish information in the DNS which ties
>  [foo123, smtp.example.net] to my address, SPF-style.
>There are problems with this if the ISP decides to change their
>infrastructure, such that outgoing mail starts appearing from a different
>machine or IP address. This would invalidate the policy.

I don't quite see how IM2000 makes this any easier.

>(3) The ISP rejects all mail from me unless it has
>MAIL FROM:<[email protected]>. This is unacceptable for lots of reasons:
>- This may be a temporary account, e.g. at a cybercafe; I will need to
>  receive bounces after I have left that location.

IM2000 eliminates bounces, but only *after* SMTP is basically
eliminated from the Internet.  Until then, IM2000 will have the same
basic problem: a message store will have to validate whatever
*arbitrary* sender address is assigned to a user's outgoing message,
or worry that the user might send joe jobs.

Even once bounces are eliminated, how does an IM2000 *sender* actually
know whether an outgoing message has been received, if she's moved on
from her cybercafe?  Same problem: either use a consistent address, or
"log on" to the same mail store via something akin to AUTH and require
the message store to validate against a list of legitimate sending
addresses.

>- I may be sending a bounce (MAIL FROM:<>)

This gets into the whole issue of whether it's acceptable for IM2000
to not provide a way for a recipient to send back a corresponding
automatic notification of some sort to a sender.

But I believe that bounces are the biggest problem with SMTP anyway,
which is why I suggest we move away from them, one way or another (by
moving to IM2000, by slowly moving towards something like ecrulisting
if it's feasible, or by moving to a new push-based, bounce-free
system).

>- There are many legitimate reasons for using variable envelope senders;
>  e.g. I may be sending using VERP, SES, SRS or BATV envelopes.

I don't know BATV offhand, but the rest strike me as kludges to work
around SMTP's built-in limitations.  IM2000 doesn't appear to solve
the problems requiring VERP, though it certainly changes the landscape
(possibly so much that we can't recognize the new problems it'd
introduce).  SES and SRS are, I believe, obviated by IM2000, at least
in an SMTP-free world.  I don't know what BATV is offhand -- "Bounce
And Tell Vinnie"?.  ;-)

>Anyone who did this would break all their users, and yet there would be
>little benefit unless the *whole* Internet did it (which will never happen
>even if this were decided to be 'best practice')

I don't quite buy that argument.  The first part of it is true for
adopting IM2000 anyway.  With regard to the second, it seems to me
that it can be up to each "trusted" relay to decide whether and how
best to validate users that submit messages to/through it.

For example, there's not really *that* much difficulty, as far as I
can see, in having a relay handling 1M users not only validate their
outgoing envelope senders against some list, but allow those senders
to be VERP-encoded to boot.

>Seriously, changing to something different (maybe IM2000, maybe not) would
>be easier. You can have a phased migration. Over time, you would give more
>credence to your new mail and less to SMTP mail.

I'm not convinced it'd be easier to change to IM2000, though perhaps
to something different.

As to giving "more credence" to messages sent via the new system, that
general capability (of prioritizing messages for recipients based on
various criteria) is necessary for my ecrulisting proposal.  It's
going to be necessary for IM2000 anyway, so we might as well take
advantage of it and try it out within the current SMTP infrastructure
(which ecrulisting exploits, without really committing any particular
portion of the SMTP world to use or accommodate it, as it coexists
with existing standards and *most* practices).

>> >IM2000 also wins because it's easy to build message stores which rate-limit
>> >the number of messages sent per day from a particular account; it could be
>> >enforced as good practice (i.e. if you offer free signups for new accounts
>> >over the web, then you get your message store blacklisted if you don't
>> >follow this principle)
>> 
>> I'm not sure why that's hard for SMTP -- aren't some ISPs doing that
>> already?
>
>Because
>- mail relays generally don't have an unambiguous indication of *who* the
>  person is sending through them (without SMTP AUTH, or some sort of
>  callback into a RADIUS accounting system)

Huh?  Source IP isn't enough to disambiguate a paying customer?  Or do
you want to include roaming users?

What does IM2000, or any new system, offer that makes this a
non-issue?

>- mail relays are typically built in clusters, so in order to count the
>  number of messages sent by a particular user in a particular period of
>  time, you would need to use a central database

See below, where this is revisited.

>...[forcing the world to adopt SMTP AUTH]...
>> >However, this ain't going to happen. The pain is huge, and the short-term
>> >gain is negligible.
>> 
>> How will the equivalent *not* have to happen with IM2000?
>
>You can start setting people up on IM2000, and include gateways to SMTP
>(inbound and outbound).
>
>People can immediately see which mail has come in through IM2000, and which
>through SMTP. They can use this to make value judgements on their mail. It
>would particularly benefit closed user groups (e.g. companies, groups of
>friends) where they know the other party is on IM2000, as it means these
>identities would be forged if they come in via SMTP.
>
>You can apply more stringent filters on SMTP mail. As more and more of your
>correspondents are on IM2000, the less worried you are about false positives
>on SMTP. For granny who only wants to communicate with grandchildren, SMTP
>could be disabled entirely for that account.
>
>Over time, though, IM2000 becomes less and less of a "closed user group" of
>course. Actually, I think a mail replacement architecture should explicitly
>support the idea of closed user groups. And possibly instant messaging too.

Wait, wasn't SMTP AUTH described here as a way for a legit client to
deposit an email on a widely trusted relay?

What I'm asking is, how does a legit IM2000 client deposit an email on
a widely trusted mailstore with something a lot like AUTH?

>> Okay, then this is not a case of *technical* advocacy, more a case of
>> saying "it's a whole new system, we think it'll be fairly spam-free
>> compared to SMTP" -- and in fact it probably will for some time -- and
>> hoping to convince enough early adopters to jump on board, quickly
>> enough, that everybody else, except spammers, switches soon after.
>
>No, if I believed that then I wouldn't suggest it.
>
>Any new solution has to be a *permanently* harder environment for spammers
>and fraudsters to work in. I think there are some strong arguments why
>IM2000 could have an anti-spam infrastructure which works much better and at
>much lower cost (of complexity and false positives) than SMTP, which I
>documented in my comments on the web.

Indeed.  That's what got this discussion started.

My concern is that IM2000 isn't *enough* permanently harder, and has
an excessively high built-in failure rate for *legit* email due to its
additional points of failure, to justify the expense of rollout.

I've already identified a few ways in which SMTP can (and in some ways
already does) offer what you identify as IM2000's advantages.

One clear example where SMTP does *not* do so is your item 2f (IIRC).

In particular, there's no way in SMTP for a server to respond to a
notification of a message with something meaning "please relay that
through one of the following servers: ...".

That would be quite useful.  The fact that an IM2000 recipient can
simply toss a notification to a trusted 3rd party so *it* can pull up
the message is, IMO, a *big* advantage to IM2000.

I haven't yet gone over all your items in such detail.  So I'm not
convinced either way; IM2000 might indeed have just enough in its
advantage to justify rollout.

>Furthermore, if there is better evidence of where the spam came from, then
>anti-spamming laws might be more effective.

Yup, this is why item 2f is such a big win for me.  Instead of
recipients of spam forwarding (possibly forged) spam to a government
agency, imagine how much cleaner the system would be if (IM2000)
recipients simply forwarded *notifications*, allowing the agency to
retrieve the *contents*, which, of course, couldn't be claimed to be
forged without implication the agency *itself*.

(Of course, said government agency would employ its own army of 0wned
machines, or their equivalent, so a spammer's mail store would have a
much harder time responding differently to a knock on its door by the
email equivalent of a 'narc.  ;-)

>> IM2000 ultimately puts us in the position of hoping not just that this
>> will be required *less* often, as a % of overall email sent, but that
>> the economics will ultimate in spammers giving up and going away.
>> 
>> I'm not seeing a clear case being made for that scenario coming to
>> pass, certainly not clearly enough such that it justifies the
>> deployment of IM2000 as it now stands.
>
>Well, maybe we have to build it and see. As well as writing IM2000, we write
>an IM2000 spam-sending toolset and publish it too, to check that things work
>as we expect.

Hey, I was hesitating about bringing that up anywhere, but it might
actually be worth deploying *free* spam-sending software widely and
publically, in order to help knock the profitability out from under
the UBE industry.

Of course, it would be a risk taken under the assumption that the
world will be able to deal with the results.  Kinda like governments
giving away various drugs in order to knock the profitability out from
under the illegal drug trade.

>> So IM2000 gives spammers a *guaranteed* weapon to use against DNS and
>> blacklists that SMTP doesn't assure them
>
>It's the same point - DoS is DoS. It doesn't really gain them anything more
>than a direct DoS assault on whatever it is they're trying to attack.

Again, I'm not so sure.  I *think* it's easier to block a direct DoS
than a DDoS, and, in turn, easier to block a DDoS than a coordinated
attack where all the requests are coming from legit sources.

And, again, I hesitate to fundamentally denote each and every host
that sends legitimate email as a key component in a global blacklist
and, therefore, a target for spammers.

>> >But it's still not going to be as strong. For example, counting the number
>> >of mails sent by a particular user is still quite hard. Consider an SMTP
>> >relay cluster of 10 machines; they would need to communicate with a central
>> >database to count the number of mails sent by one user over 24 hours.
>> 
>> How is that different from an IM2000 mailstore cluster of 10 machines?
>
>Most likely each user would have an account on a single machine in the
>cluster, and you'd spread your users across them. Then it's easy.
>
>But even if you decided to have a cluster with a shared NFS backend, and
>users could connect to any of those 10 machines to submit a new mail: all
>you need to do is maintain a state file within the users' mailstore area
>giving a history of sent mail. Each new submission adds to that history, and
>may cause a threshold to be reached.
>
>This is unlike SMTP, where there is no permanent record of mails passing
>through the system, and all outbound mails are lumped into a single queue,
>not a separate queue for each user.

I'm not sure how this differs *fundamentally* from IM2000, except the
ISP has to expect to hold lots more outgoing messages in its queue
than otherwise.

After all, an outgoing SMTP queue *can* be structured so it is
per-user.

(I'm not sure an IM2000 mail store has to be so structured, offhand,
by the way.)

I still think this is a minor implementation issue on the SMTP side,
but maybe someone who *knows* big-iron SMTP-relay implementations can
chime in.

>> >And you'll still need a new blacklist infrastructure for looking up
>> >[IP,auth-sender] instead of just IP address.
>> 
>> Sounds like item 2a on your web page!  Again, how is it different?
>
>That's the point. Moving to a parallel SMTP world would be pretty much as
>difficult as switching to IM2000.

Um...except we already *have* SMTP AUTH client and server software
deployed, etc....?

>> What it suggests is that SMTP could evolve to incorporate IM2000
>> concepts in ways that *allow*, but don't *require*, clients and
>> servers to cooperatively use them.
>
>I'm happy to see specific proposals here, and as you're no doubt aware, a
>lot of work is going on already.
>
>The trouble with greylisting by giving a 4xx response after the DATA phase
>(or unceremoniously dropping the connection) is that there are plenty of
>broken MTAs out there which are likely to treat this is a permanent
>rejection.
>
>Like SPF, you may end up breaking more than you fix.

Indeed.  IM2000 (or any similarly "new" system) has the advantage of
unceremoniously dumping all *previously* broken software.

Hopefully, it won't come with lots of *new* broken implementations.
;-/

I also wonder if maybe it's better to just continue to incrementally
improve SMTP and, just as you suggest recipients might "favor" IM2000
email over SMTP email, they would favor newer, conforming SMTP clients
over broken ones, while still allowing (most) email to get through
from broken ones via whitelisting, specialized bridges, etc.

>> But I prefer to *conceptualize* in terms of a brand-new, clean system,
>> and then *actualize* in terms of incrementally improving SMTP in that
>> direction, in order to speed uptake of whatever I might think is
>> useful.
>
>Well, that's reasonable enough. But the extended SMTP may look not very much
>like SMTP by the time the job's done :-)

Right.

>Ultimately, I believe a new messages store with an SMTP gateway would
>probably end up being easier to build *and* easier to interwork with the
>rest of the Internet... and it could deliver tangible benefits immediately.

I tend to think so as well, especially if we are indeed heading
towards a world with fewer, more-trusted, relays/stores.

On the other side of the equation, I personally hold out hope for a
more distributed, jungle-like world of clients injecting email
directly into recipient's systems (via their servers), since I believe
that, in the long run, that's a more robust model for a mode of
communication that *has* to be robust to be useful.

There's no question SMTP's long history of bodges makes my hoped-for
outcome much more problematic under the current SMTP model, so the
best course of action to try it out might be to offer free,
incremental improvements to the SMTP world in the form of new, better
implementations that recognize each others' capabilities in the form
of protocol improvements (the usual EHLO and new-verb combo, mainly).

Or, the mess of SMTP bodges might justify designing and rolling out a
system that is completely new, like IM2000, but is push-based, like
SMTP, yet bounce-free, like IM2000, because it employs tracking, like
FedEx, yet leaves responsibility delivery with the sender a la the
end-to-end principle, like TCP.

Whew.  Maybe I ought to just stop talking and design either or both of
the systems *I* envision, and offer it as a sort of low-level,
nuts-and-bolts way for sysadmins to communicate with each other about
problems with higher-level layers of the 'net (including, potentially,
IM2000 ;-).

Then, leave it up to fate to decide whether "sysadmins" come to
include ordinary people using home PCs, blackberries, etc., and
whether it's worth bridging it with classic SMTP.

Whaddya think?

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>