Re: Comments on IM2000

On Thu, May 05, 2005 at 06:46:50AM -0000, James Craig Burley wrote:
> >Sure, any blacklist is subject to that kind of attack. Blacklists which run
> >on top of the DNS benefit from distributed caching.
> 
> I've previously posted, at some length, about the dangers of relying
> on distributed (DNS-style) caching as an effective solution to the
> problem of giving attackers direct ability to trigger database lookups
> (and especially inserts) based on arbitrary keys.
> 
> (IMO, distributed DNS-style caching is to this sort of attack as more
> CPU and RAM are to a brute-force implementation of an NP-complete
> problem such as TSP as the problem sizes increase.)

My theoretical counter-argument to this is that if you can cripple the DNS
using this approach, then you're saying that the DNS itself is open to a DoS
attack. That may be true, but then that is an entirely separate problem and
exists whether or not it's used for blacklist lookups.

My pragmatic counter-argument is that DNS-based blacklists exist already,
and work. Spammers *do* attempt DoS operations periodically against those
blacklists - against their DNS infrastructure but also against their web and
mail infrastructure and anything else they can attack. These attacks are no
different to normal DoS attacks and are handled in the way such attacks are
always handled.

Anyway, I'm not wedded to the idea of using DNS as a blacklist lookup
mechanism, but I do think that it's a good idea to use a well tried and
tested mechanism.

> Whoops, good catch!  Lemme try that again:
> 
>    MAIL FROM:<[email protected]>
>    RCPT TO:<[email protected]>
>    DATA
>    [...]
>    .
> 
> Does that make more sense?

Yep, although this spurious info will be hidden in a Return-Path: header and
unlikely to trouble the recipient.

> >Spurious notifications are unlikely to be sent often, since they don't
> >achieve anything apart from a waste of resources, so I think it's unwise to
> >build a separate infrastructure to filter them out, one which won't be
> >tested very often.
> 
> In essence, a notification becomes a handshake between a message store
> and a recipient (agent).  That could be handled solely via a TCP
> connection a la SMTP, in which case the client's IP address can't be
> (easily) forged.

Yes. But in IM2000 the notification is "push", and it does not necessarily
originate from the message store. The sender of the notification can't forge
their source IP address, but that doesn't stop them sending a spurious
notification, and the notification doesn't have to come from the message
store itself. These two combined mean that you need to validate it.

But that does make me think of another option. Suppose all notifications
were forced to come from the originating message store. If you want to
forward a notification to someone else, you do it by asking the originating
message store to do it for you. Hmm... I'd have to think if there would be
any benefits in that.

The SMTP equivalent would be a kind of 'redirect' response to an incoming
message. I vaguely remember the protocol actually has a response code for
that, which nobody actually implements. Ah yes, RFC 821:

      S: RCPT TO:<[email protected]>
      R: 551 User not local; please try <[email protected]>

RFC 2821 (3.4) advises caution again this, and SMTP clients are allowed to
treat this as a bounce, which everyone does.

> >Except that at the moment, an SMTP callback only verifies that the address
> >exists, not that the mail you're trying to receive was sent by that person.
> 
> Well, yes, but what I was referring to was that such callbacks were
> considered highly annoying by (sysadmins for) victims of joe jobs,
> weren't they?

A callback to verify isn't annoying, since it doesn't deliver a mail - but
an actual delivered bounce is, of course.

> IM2000 can (and IMO will) make them *cheaper*, but will they be cheap
> *enough* to be sure *all* sysadmins will find such callbacks, which
> appear to be intrinsic to a deployable IM2000 (*replacing* SMTP),
> acceptable?

Some people argue that callbacks are "too expensive". I don't think there
are any hard economics to back that up; they just want to minimise their CPU
and network overhead, which is fine, but then the cost of handling all that
spam must be pretty high too. I don't think the overhead of a TCP socket
setup is particularly high.

> Instead, what if the MUA periodically contacted the recipient (or the
> same store-and-forward, or upstream, relay) and asked it "how is
> delivery of that message progressing?".

I think that's almost the same, except you've turn an active notification
into a passive polling.

If the client polls, then it will need some sort of message key to query the
state of a particular message. In that case, the server could have made an
active notification using the same message key.

> It would have to allow not only for answers equivalent to the
> 2xy/4xy/5xy responses we presently have, but for "it appears to have
> been lost, please resend", since the MUA would have responsibility for
> that message.

So, whilst the message itself may have been lost, the *state* of the message
must be retained for the client to poll at some later stage - and
potentially must be kept for a very long time.

> >It can be done, but in general it isn't.
> 
> This is the line of reasoning you use to which I most object.

I think it isn't done because the economic cost is extremely high, the
immediate gains are virtually zero, and the gains are only realised when
everybody *else* does it. In other words, there's no business case. Who
wants to spend money on a project which will only deliver benefits if
everybody else joins in as well - when we know perfectly well that there's a
large subset of people on the Internet who won't?

OTOH, implementing something like SES/BATV on your own server does have a
business case: you do some work, and you immediately stop joe-jobs coming
into your mailboxes. That's much easier to take effect. You can then
leverage other features on top of that (such as callbacks to validate
envelope senders)

> >(1) The ISP maintains a local list which ties foo123 <=> [email protected]
> >Every time a new user comes along, they will have to state which E-mail
> >addresses their account is allowed to use, *and* prove to the ISP's
> >satisfaction that they do in fact own those accounts.
> 
> This isn't a problem if the envelope sender is required to be
> [email protected].
> 
> If it can be anything arbitrary, then IM2000 will have similar
> problems, correct?

IM2000 (as in current document) requires that the 'envelope sender' be the
actual mailstore hostname and account ID on that mailstore - otherwise it
simply doesn't work.

My own suggestion is that the sender ID be an E-mail address tied by design
to the same mailstore account.

I was trying to get at the issues which would occur if you go down the route
of stay-with-SMTP-and-try-to-patch-it-up.

> IM2000 eliminates bounces, but only *after* SMTP is basically
> eliminated from the Internet.  Until then, IM2000 will have the same
> basic problem: a message store will have to validate whatever
> *arbitrary* sender address is assigned to a user's outgoing message,
> or worry that the user might send joe jobs.
> 
> Even once bounces are eliminated, how does an IM2000 *sender* actually
> know whether an outgoing message has been received, if she's moved on
> from her cybercafe?

Well, the cybercafe probably wouldn't give you a mailstore account (unless
you're a regular customer). You would use a mailstore at a third party (a la
"hotmail") or your home ISP.

Now, the same could apply today: the cybercafe could simply not install a
mail relay, and block port 25. That would force you to connect over port 587
to your home ISP, and use SMTP AUTH to submit a new mail.

There's a chicken-and-egg though: most ISPs don't support this service; so
cybercafes can't block port 25; and so ISPs have little incentive to add it.

If we wanted to move to a new SMTP world order, it's hard to see how to
force it to happen. You could take a radical approach: blacklist everyone
who doesn't implement the new world order policies. In that case, you will
find yourself unable to talk to almost anyone (in which case you might as
well set up in the IM2000 world instead).

But there's a more fundamental problem: how can you *tell* whether a
particular SMTP relay server complies with the "new world order" rules or
not?

The new world order says things like:
- you cannot submit mail on port 25 based on purely your IP address. You
  must always use SMTP AUTH when submitting new mail.

However, you can't actually *test* this without going to the site, setting
up a machine on one of *their* IP addresses, and testing it for yourself.

That's one reason why a completely new protocol actually makes sense, in my
opinion. Mail received via the new protocol must be complying with the
new rules, if the protocol itself requires those rules as part of its
fundamental mode of operation.

> >- There are many legitimate reasons for using variable envelope senders;
> >  e.g. I may be sending using VERP, SES, SRS or BATV envelopes.
> 
> I don't know BATV offhand, but the rest strike me as kludges to work
> around SMTP's built-in limitations.

Absolutely. And even if we design an SMTP new world order, then either those
kludges will still be needed, or a whole raft of extensions to SMTP will
need to be added such that they are no longer needed.

That would put another barrier to joining the NWO: you'd first have to
upgrade your mailserver to SMTP++.

> I don't know what BATV is offhand -- "Bounce
> And Tell Vinnie"?.  ;-)

"Batavia Access Television", according to Google. But the ninth hit is the
one I meant: "Bounce Address Tag Validation (BATV)"

> >> I'm not sure why that's hard for SMTP -- aren't some ISPs doing that
> >> already?
> >
> >Because
> >- mail relays generally don't have an unambiguous indication of *who* the
> >  person is sending through them (without SMTP AUTH, or some sort of
> >  callback into a RADIUS accounting system)
> 
> Huh?  Source IP isn't enough to disambiguate a paying customer?  Or do
> you want to include roaming users?

With dynamic IP pools, source IP doesn't tell you *which* paying customer,
only that they are *a* paying customer.

> What does IM2000, or any new system, offer that makes this a
> non-issue?

That you have to authenticate to the mailstore to submit a new message. It
doesn't work without it.

> Wait, wasn't SMTP AUTH described here as a way for a legit client to
> deposit an email on a widely trusted relay?
> 
> What I'm asking is, how does a legit IM2000 client deposit an email on
> a widely trusted mailstore with something a lot like AUTH?

Same. You deposit your message in your own mailstore account (e.g. one which
your ISP has given you on their mailstore), and the protocol requires you to
authenticate to it to achieve that.

> My concern is that IM2000 isn't *enough* permanently harder, and has
> an excessively high built-in failure rate for *legit* email due to its
> additional points of failure, to justify the expense of rollout.

You may well be right; I've not made my own mind up on that either, and
certainly there may be ways to improve it.

> >Furthermore, if there is better evidence of where the spam came from, then
> >anti-spamming laws might be more effective.
> 
> Yup, this is why item 2f is such a big win for me.  Instead of
> recipients of spam forwarding (possibly forged) spam to a government
> agency, imagine how much cleaner the system would be if (IM2000)
> recipients simply forwarded *notifications*, allowing the agency to
> retrieve the *contents*, which, of course, couldn't be claimed to be
> forged without implication the agency *itself*.
> 
> (Of course, said government agency would employ its own army of 0wned
> machines, or their equivalent, so a spammer's mail store would have a
> much harder time responding differently to a knock on its door by the
> email equivalent of a 'narc.  ;-)

Absolutely. And that's also a good reason for sticking an MD5 hash of the
message in the notification - so they can't just give out a different one
instead :-)

SMTP could be improved to allow something similar, but it would probably
require signatures on messages. Problems there include message body
mangling, and the privacy concerns that people might not *want* a long-term
proof of sending of a message. With IM2000, the notification proves the mail
was sent by X if you can use it to retrieve the message from X's message
store, but once the message has been unpinned, the proof is gone.

> >This is unlike SMTP, where there is no permanent record of mails passing
> >through the system, and all outbound mails are lumped into a single queue,
> >not a separate queue for each user.
> 
> I'm not sure how this differs *fundamentally* from IM2000, except the
> ISP has to expect to hold lots more outgoing messages in its queue
> than otherwise.
> 
> After all, an outgoing SMTP queue *can* be structured so it is
> per-user.
> 
> (I'm not sure an IM2000 mail store has to be so structured, offhand,
> by the way.)

Well, for example you don't want one user to be able to query the status of
messages sent by another user, or to delete messages sent by another user.
That means you authenticate to the mailstore, and have control only over
your own messages.

SMTP *could* have per-user queues, but what I really meant was per-user
*state* giving the history of messages sent. That's what I meant by a shared
database, in the case of a cluster.

With a cluster of SMTP relays, you *could* use a shared NFS server to
maintain history for each user, but it's a clunky form of database, and you
might as well use SQL or LDAP or something like that. That's because there's
no other reason for an SMTP relay to have access to shared storage; it can
keep its own queue of outbound SMTP messages on its own local disk.

> >> >And you'll still need a new blacklist infrastructure for looking up
> >> >[IP,auth-sender] instead of just IP address.
> >> 
> >> Sounds like item 2a on your web page!  Again, how is it different?
> >
> >That's the point. Moving to a parallel SMTP world would be pretty much as
> >difficult as switching to IM2000.
> 
> Um...except we already *have* SMTP AUTH client and server software
> deployed, etc....?

We have *bits* of what is required for the SMTP NWO. As outlined above, we
need some way to force people into the NWO, which I believe means some way
of telling whether a particular server complies with the NWO or not, which
is very difficult (to the point of impossible).

A new blacklist infrastructure (based on sender+IP, not just IP) is just an
example of another extra thing which has to be done to implement the SMTP
NWO.

> >Like SPF, you may end up breaking more than you fix.
> 
> Indeed.  IM2000 (or any similarly "new" system) has the advantage of
> unceremoniously dumping all *previously* broken software.

Maybe. The problem is that it's hard to detect broken versus non-broken
implementations of existing software.

This means you can't piggyback on existing mechanisms (like returning 4xx
to a DATA section); instead you have to advertise a new SMTP capability in
EHLO such as "250 PROPER4XX" which means that the recipient agrees to handle
it properly. In which case, you might as well define a new extension which
does what you actually want.

> >Ultimately, I believe a new messages store with an SMTP gateway would
> >probably end up being easier to build *and* easier to interwork with the
> >rest of the Internet... and it could deliver tangible benefits immediately.
> 
> I tend to think so as well, especially if we are indeed heading
> towards a world with fewer, more-trusted, relays/stores.
> 
> On the other side of the equation, I personally hold out hope for a
> more distributed, jungle-like world of clients injecting email
> directly into recipient's systems (via their servers), since I believe
> that, in the long run, that's a more robust model for a mode of
> communication that *has* to be robust to be useful.

I see where you're coming from - more peer-to-peer and less centralised. The
instant-messaging model fits well here.

The problem comes with spam and identities. Whatever identity I choose for
myself (whether it be a domain name or a cryptographic key), if I can
generate new ones at will, blacklisting becomes completely ineffective. You
then have to consider options like limiting each person to one domain or
having crytographic keys signed by the government, neither of which I like.

Having a third-party involved - a message store on a fixed IP address - does
give *some* control over the rate at which new identities are created,
and the rate at which new identities can send mail. Other ideas for
achieving the same goal are of course very interesting!

Any sort of closed-user-group model doesn't have this problem. But if you
wish to be able to accept mail from strangers on the Internet, then it is a
major problem. Some FUSSP suggestions are basically ways of making it
difficult for strangers to introduce themselves to you - such as "hashcash".
Challenge-response systems are in that class too (but they won't work once
spammers build systems to response automatically to the challenges)

> Or, the mess of SMTP bodges might justify designing and rolling out a
> system that is completely new, like IM2000, but is push-based, like
> SMTP, yet bounce-free, like IM2000, because it employs tracking, like
> FedEx, yet leaves responsibility delivery with the sender a la the
> end-to-end principle, like TCP.

The familiar push model has advantages too. It might be worth trying to
design a new protocol under this model, and see if you can end up with the
same advantages as IM2000 (or other ones).

Regards,

Brian.