Re: Comments on IM2000

>On Thu, May 05, 2005 at 05:07:15PM -0000, James Craig Burley wrote:
>> >But that does make me think of another option. Suppose all notifications
>> >were forced to come from the originating message store. If you want to
>> >forward a notification to someone else, you do it by asking the originating
>> >message store to do it for you. Hmm... I'd have to think if there would be
>> >any benefits in that.
>> 
>> I think JdBP's IM2000 proposal, which is much more detailed than djb's
>> and which commits to certain design decisions, commits to this one as
>> well: *all* notifications come from the originating message store.
>> 
>> Whether that permits a recipient to simply forward a notification to
>> another party without notifying the store depends mainly whether the
>> request for the contents of a message include the recipient's IP
>> address (or similarly unique identification) in the key or
>> authorization for such a request.  If not, forwarding of notifications
>> is trivial; if so, then, yes, the *store* would have to handle
>> forwarding at the request of a recipient.
>
>JdBP appears to allow the forwarding of notifications without the original
>message store being involved: [...]

Okay.  That seems reasonable.

>The blacklisting is done using the domain name of the message store
>(illegal.com), so I imagine it must be against notifications which come in
>where the originating message store is illegal.com, not that the source IP
>address of the notification is *.illegal.com

I believe that's the case as well.

>> On the SMTP side, it's not all *that* hard to prioritize incoming
>> messages based on a recipient's perception of the trustability of
>> upstream relays, including whether SMTP AUTH was used.  (But in cases
>> where use of AUTH isn't reliably reported in "Received:" headers,
>> either the relay in question always requires AUTH and thus acquires,
>> over time, lots of trust, or will have to find other ways to be sure
>> it avoids the problem of being a mixed source of UBE and desireable
>> email and, thus, untrusted.)
>
>The trouble is you can't rely on the sender *telling* you that the message
>was submitted via SMTP AUTH, and that therefore you should trust it. If so,
>spammers would just make their mail systems make the same declaration.
>
>With SMTP, you need to verify independently whether the mail server in
>question is actually trustworthy - for example by having trusted agents on
>that ISP's network performing the testing for you. Ick.

I'm saying, why care whether a third-party relay uses AUTH or any
other *particular* mechanism?

All you *really* care about is whether that relay is well-run enough
to avoid accepting more UBE, likely destined for your system, than you
feel comfortable with.

Leave it up to the operator of that relay to decide how best to gain
that trust.  It can advertise that it uses AUTH generally, or, via
"Received:" headers, *when* it uses AUTH.  Whether that impresses you
is up to you, since one relay might be AUTHless but UBE-free and
another might always AUTH but be a huge source of UBE.

AUTH is just another of those things that is an extra hoop to jump
through but, otherwise, not really needed most of the time.

Accordingly, until it's *required* to be used most or all of the time,
people won't bother with it.

And if they *do* require it, there'll be considerable pressure to
provide it in a "seamless" way that would ultimately make it nearly
useless in terms of determining whether a *particular* human being
tends, or tends not, to deliberately send UBE to the rest of us.

(Even if you do test a network for AUTH, what does that really tell
you?  That it won't let just *any* random spammer access it -- just
the paying ones?  Etc.)

Make AUTH sufficiently easy for everyone to use, and everyone --
including those with 0wned machines -- will use it.

>> >With dynamic IP pools, source IP doesn't tell you *which* paying customer,
>> >only that they are *a* paying customer.
>> 
>> Really?  Why not?
>
>I dial into a modem, and I get an IP address out of a pool. The SMTP server
>knows that I'm connecting from IP address x.x.x.x, but not my customer ID.
>It could only find that out by some out-of-band database which associates in
>real time IP addresses with customers - for example, you could build some
>infrastructure which takes RADIUS accounting packets and builds a real-time
>mapping, and query that using some suitable protocol (LDAP perhaps). Even
>then, you'd have to live with the fact that RADIUS accounting is UDP-based
>and unreliable, and so occasionally you will associate x.x.x.x with the
>wrong customer.

Amazing.  I just assumed ISPs had better internal coherency in their
infrastructure -- that any (internal) server could do the equivalent
of an rDNS lookup on x.x.x.x and get a unique ID for a customer, even
though the outside world might see something much more opaque (like
the same IP address encoded and decorated).

(I'm still confused as to how IM2000 significantly changes any of
this, other than to perhaps *require* ISPs to do what you're saying
they would find annoyingly necessary to do for SMTP anyway.)

>> >Any sort of closed-user-group model doesn't have this problem. But if you
>> >wish to be able to accept mail from strangers on the Internet, then it is a
>> >major problem. Some FUSSP suggestions are basically ways of making it
>> >difficult for strangers to introduce themselves to you - such as "hashcash".
>> >Challenge-response systems are in that class too (but they won't work once
>> >spammers build systems to response automatically to the challenges)
>> 
>> Right.  My ideal is that incoming email from total strangers would
>> *tend* to be immediately accepted, and my *proposals* revolve around
>> the notion that, prior to the human recipient actually reading a given
>> message, the sending stranger's MUA repeatedly requesting status (or,
>> in ecrulisting, resending the message) would, in the absence of other
>> information on the tendency of the sender to send Bulk email, tend to
>> *increase* the priority of that message as seen in the recipient's
>> list of pending messages.
>
>OK, but a spammer can emulate that behaviour too.

Of course.  A spammer can do so for any *particular* message she
sends.

Can a spammer do it reliably, consistently, for all the 100M or so
messages she sends out each day, without triggering auto-detect
mechanisms?

I'm not sure that'd be possible.  And I think this approach even
handles the problem of massive numbers of 0wned machines on the net,
until their 0wnership is turned into making the machines *accept*
incoming UBE and prioritize it highly (at which point their users will
take their machines' 0wnership more seriously ;-).

>> Since it would be mostly a matter of automation to infer whether it is
>> Bulk Email as well, that means that a sender of UBE becomes recognized
>> as such on a per-site basis.
>
>In the limiting case, the spammer generates a new on-line identity for every
>individual message sent out. It looks like lots of new individuals popping
>up on the Internet, sending their first E-mail. How do you deal with that?

They have only so many IP addresses from which to send their *Bulk*
email, and they send them to so many spamtraps, that the recipient's
MTAs and MUAs can fairly easily detect such activity and flag such
messages as "likely UBE", even before any content analysis is
performed (though it can be performed anyway).

>> So, by sites individually choosing to deploy some spamtraps, and
>> perhaps using RBL and other technologies not to necessarily *block*
>> incoming messages but, rather, to simply *ignore* them and let the
>> senders repeatedly inquire as to their status...
>> 
>> ...*receivers* of UBE turn their systems into "black holes" for mail
>> that actually *is* Unsolicited Bulk Email.
>...
>> If it's not Bulk, a receiving system can easily recognize that it
>> hasn't seen 1M messages coming in from that particular sender's IP
>> address in the past 48 hours, most or all of which have gone unread or
>> marked as Unsolicited by their readers.
>
>Ah, but that's it then. You *do* need to rely on a sender's IP address as
>part of their identity in order to detect "bulk", not just their domain name
>or public key.

More precisely, one relies *primarily* on a sender's IP address as
part of their identity, and can often not bother looking up their
domain name, public key, reputation, etc., in order to determine that
the IP address in question is a source of UBE.

A receiving agent can accomplish this without relying on any third
party, *including* DNS!

I think that's pretty cool, and darn near bullet-proof, assuming the
concept as a whole makes sense (which it might not).

>Which brings us back to shared SMTP relays, and the fact that legitimate and
>bulk E-mail may spew forth from the same machine, which is really what makes
>life difficult right now. So then it's just down to whether spammers 'track'
>their outbound mail in a pattern which is similar to how a legitimate sender
>would do so.

My approach makes it much more expensive, over time, to run SMTP
relays that accept UBE, since they find it much harder to foist
responsibility for that UBE on to other hosts, and in the meantime
even *legit* email tends to sit around on the server much longer while
downstream entities (including recipients) use finer-grained
techniques to assess the likelihood that any *given* email is UBE,
given that they can't easily make that decision based on the IP
address of the relay (as it is a "mixed blessing").

Indeed, I assume (some) spammers *will* simply use the same sort of
tracking algorithm that legit MUAs employ.  That's part of the beauty
of this approach: they'll be investing much more, in the way of
resources, into each email they send.  It's what IM2000 promises,
without the warts (delays reading messages) or expense (designing and
implementing a whole new email infrastructure) -- namely, that
*senders* bear more of the cost of sending UBE.

>> So it "punishes" senders of UBE in that they either become (well,
>> remain) careless senders -- which recipients can more easily
>> distinguish from caring ones -- or they expend a great deal more
>> resources to send and care for each outgoing piece of their Bulk
>> email, while taking the very significant risk that Bulk email sent to
>> spamtraps and/or sites that otherwise autodetect Bulk sending will
>> result in such care accomplishing literally *nothing* in terms of
>> getting a message into a given end user's eyeballs.
>
>The 'care' taken is, presumably, comparable to that needed to deal today
>with getting a 4xx response and to resend later. I don't think it's hard;
>it's what standard MTAs do all the time.

That's correct, modulo a few well-known cases (old versions of Lotus
Notes; Yahoo Groups; etc.), I gather.

>If I were writing a bulk SMTP sender, I wouldn't use a standard MTA, which
>is not optimised for keeping track of 10M recipients for one message.
>Rather, I'd write my own which keeps one copy of the spam and tracks the
>recipients in some sort of database (10M recipients can be kept in a hash
>table in RAM very easily, given say 512MB of RAM). I'm almost inclined to do
>so as a proof-of-concept.

Sure.  As with IM2000, it isn't the huge number of message *contents*
that a UBE sender has to deal with, because, as you noted, a typical
sender of UBE has just a small number of *unique* messages to send to
a huge audience.  (Accordingly, IM2000 actually penalizes senders of
large numbers of *unique* messages moreso than it does most senders of
UBE.)

So, let the sender track *all* those messages, as you suggest.  Now
they're expending much more, in the way of their resources, per
*important* outgoing message than the typical sender of legitimate
email, *both* of whom get to shovel the *content* across the wire the
first time they try.

(Note that this implies the legit sender will tend to spend much more
of her content-shoveling bandwidth on *desired* email than will the
sender of UBE.  With IM2000, the sender of UBE isn't wasting their
outgoing bandwidth on email that's never read, while the sender of
legit bulk email has not only a huge outgoing bandwidth budget but
also the need to run a mail store.  *Always* try to tilt the economic
wins in favor of the white hats, and the losses towards the black
hats.)

As counterintuitive as that sounds, it give legitimate senders and
recipients of email a "leg up" on spammers, because they *already*
care about *their* messages, don't mind having to demonstrate that
care, and in fact, on today's Internet, would likely welcome a system
that would tend to *reduce* the costs of exchanging their email
(mainly because it basically eliminates bounces) while making it
easier to know just how a given message is progressing.

(In particular, a sufficiently smart MTA that handles all incoming and
outgoing messages for a site can decide to reject or simply ignore all
incoming *bounces* as long as it has no outstanding deliveries that it
is unable to track in the new way.  Bounces don't vanish overnight;
they, along with joe jobs, become *gently* less expensive as they're
needed less often.  Of course, "outstanding deliveries" would include
messages sent to traditional SMTP servers anytime in, say, the
preceding two weeks, and for which no conclusive bounces/DSNs had been
received.)

>I don't think spammers are stupid; E-mail is their business, and they know
>how it works very well. Maybe the few days' programming needed isn't worth
>their while yet, which is why the greylisters are seeing some benefit, but
>it only needs the next version of SpamSenderPro or whatever to have this
>feature, and greylisting will die overnight.

Yes, that's long been predicted, and will probably come to pass.  But
greylisting provides no benefits to those exchanging legit emails,
because it can only *delay* such exchanges.  So the UBE senders are
still in a better position, since they don't really care whether their
UBE arrives in a mailbox immediately, or five, ten, or twenty minutes
later, as long as it is accepted before they are forced to move on,
whereas people exchanging legit emails *often* want instant delivery.

(Ecrulisting, as well as my other proposal, doesn't make things any
worse for ordinary users exchanging emails.  It's the underlying
infrastructure that might have to work harder, or not -- maybe a lot
harder for the sender, a little harder and a bit more cleverly for the
receiver, etc.)

By "move on" I mean senders no longer (re)send email from that same IP
address, or at all.  Equivalently, they no longer request tracking of
messages previously sent from that same IP address; they no longer
send message notifications from that address; they no longer provide a
message store from that address; etc.  And maybe "from that address"
is impertinent; it's not clear there's any need to design a
requirement that tracking or resending must come from the same IP
address into the system.

Continuing here, I'm assuming that many users don't read their inbox
frequently enough to be sure the sender is still online when they
finally see the message.  So the general question here is what happens
when the sender has moved on before the reader finally gets around to
invoking their MUA and seeing the in-box?  I don't mean in specific
cases, I mean what happens in terms of the ability of senders of large
amounts of legit email versus senders of UBE?

Looking at these issues, with vanilla SMTP, once a sender moves on,
the fact that her outgoing email has been fully accepted makes it that
much more complicated for recipients to handle the fact that, despite
senders having "moved on" and/or been determined to send UBE, all the
*accepted* email must still be delivered, dropped, or bounced.  After
all, responsibility has already been accepted.  And bouncing email
after accepting responsibility for it during an SMTP conversation is a
big problem, as we all know.

With IM2000, if that email has not actually read by a real person, it
cannot be read at all, as the sender has moved on.  Once a few MUAs
detect that, they can notify the "collective" that *all* pending
notifications from that source are suspect.  Here, responsibility has
not been accepted, but it's not clear whether false positives are
involved (the "moving on" might have been a legit laptop sending legit
email being disconnected or given a new IP address), so there's a
problem with the fact that the messages aren't actually available to
the reader (or a content-analysis engine).

With my system, until that email is *fully* delivered, the sender
retains responsibility for it.  "Moving on" can therefore be
legitimately interpreted as no longer demonstrating interest in it.

But the message itself is (usually) available to the reader.  If the
reader wants to accept responsibility for the message, or signal some
other error, that sort of "bounce" is communicated upstream to the MTA
(SMTP server) -- tagged to the unique ID for the message, whatever
*that* is, kept by the MTA --- so that it can respond accordingly to
the next tracking request (or, with ecrulisting, next delivery
attempt).

I believe this makes it much more feasible for a host to keep track of
"origins of interest" for incoming email, and thus notice when a given
origin, representing a sufficient number of emails whose contents,
have been accepted but not for which responsibility has been accepted,
suddenly stops requesting tracking information (for ecrulisting, this
means it stops retrying delivery).

Such an event doesn't result in *loss* of email, since, in my system,
exchange errs on the side of duplicate delivery and duplication of
effort, but it can result in determining that the source was not
sufficiently interested in delivery -- or that it was, but lost
connectivity.  And it can do this based on a sampling of the pending
messages.

In the extreme (but likely frequent, in today's world) case, if an MTA
simply decides to drop *all* pending email from a given source, it can
do so in a fashion that either lets the sender discover the fact by
issuing new tracking requests (which either fail or yield "discarded
by server") or resending (which the MTA might /dev/null but not
disclose accordingly).  Such mail is never "lost", however, since it
was never truly *sent*, in that responsibility was never
*transferred*, in the first place.

My proposal also can perform content analysis of the sort that is
performed today -- without, as IM2000 tends to do, disclosing the
existence of "listening" recipients -- because the "default" is to
accept the full message without accepting responsibility for it.

This is definitely *not* the sort of "hard system" that most everyone
else proposes to deal with UBE.  It's more of a "soft" response that
allows for failure, in that it has fewer points of failure.

But "soft" responses, or defenses, can be much more effective against
an opponent that throws a flurry of misguided punches and then gives
up, only to move on to attack someone else, or return later.  Each
such punch goes further, because it isn't repelled, and sucks the
attacker into spending more time and energy throwing punches that, in
the end, accomplish little more than never having engaged in the
attack in the first place.

There are difficult aspects to my proposal.  With a new, clean design,
the challenges are mostly coming up with a design that won't have
aggravating aspects down the road; with SMTP, it'd be annoying to have
to take into account problematic aspects of that infrastructure,
including a biggie, the difficulty of uniquely identifying any message
*transmission* regardless of the *path* that message takes to reach a
given MTA.  (E.g. "Received:" headers largely don't make a difference
in message uniqueness, but the "older", or more-upstream, ones, if
they disagree about the origination point, might be enough to denote
two delivery attempts as pertaining to two distinct messages even if
the contents are otherwise identical.)

And, as with IM2000, my proposal requires lots more close
(fine-grained) interaction between MTAs and MUAs.  Probably something
new in place of mbox's and maildirs, for example.  I don't know for
sure.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>