Re: Of Messages and Testing...

>  Why not build a mail-server that REQUIRES a "Delivery Confirmation"
>  and a "Clean-Logoff" before the message is delivered to the
>  recipient? This way a message can be checked, and delivered ONLY if
>  1) the Sending machine waits for the confirmation and 2) it passes
>  the checks made on the message by whatever spam/virus checkers the
>  server admin chooses? Doing this will help prevent "Drive By
>  Spamming" where the spam sender hangs out long enough to send the
>  message and then skedaddles. Also, this will allow the
>  administrator to perform other tests on the message (including
>  virus checks, DNS checks, RBL checks, banned user tests, etc) via
>  whatever "Plug-Ins" they add to their server, and then the server
>  will either pass or fail the message accordingly.

You're not describing IM2000, or pull-style delivery generally, which
make "drive-by spamming" very easy from a protocol point of view,
leaving it up to recipient agents to decide whether to continue
transactions based on criteria such as reputation of the outgoing
message store to which notifications regarding spam would have to
point.

In a "naive" IM2000 deployment, a drive-by spammer will be quite
successful injecting a bazillion message notifications into a great
many recipient agents...but then his outgoing message store will
presumably be overwhelmed with requests for those outgoing messages,
on a schedule he'll have a hard time predicting, until recipients
decide to no longer trust that particular store.

So, for pull-style delivery, there's not really much to be gained by
requiring an injecting agent to sit around until delivery has been
confirmed and then cleanly log out, or anything like that.

What you describe is, generally, a reasonable tactic to be able to
deploy for SMTP, or push-style delivery, insofar as it allows
recipient agents to "raise the bar" for resource expenditure by email
injectors (SMTP clients) on a per-client, or per-sender, basis,
without necessarily (if designed right) doing so for all exchanges of
email.

But SMTP isn't really solid enough, at present, to withstand much more
of this sort of injection-time analysis and such, because of a certain
critical window in its transaction design.

It so happens that, late last night (really, early this morning) I
suggested to Russ Nelson that, to support certain forms of
content-sensitive authority checking (DomainKeys, as it happens,
though variants of SPF that rely on information found in the messages
apply here as well), the SMTP protocol be extended to separate message
*content* transmission from message *submission*, e.g.:

> MAIL FROM:<blah>
< 2xx whatever
> RCPT TO:<blech>
< 2xx whatever
> MESG                     <-- Like DATA, but termination doesn't submit
< 3xx go ahead
> [message content, delimited somehow]
< 2xx got it
> SUBMIT                   <-- This is how client finally submits message
< 2xx ok 548729542 qp 45315

Without this distinct step, a problem with any content-based scanning
or artificial delaying tactics that takes more than a short, finite
amount of time is that the window of opportunity for duplicate
delivery increases: after "\r\n.\r\n", the client is awaiting a
success indicator that indicates both acceptability and successful
internal submission.  If acceptability takes too long to determine,
the connection might be lost, but the server might have already
internally submitted the message and sent the success indicator (or,
permanently rejected it).  The client, seeing the dropped connection,
retries delivery a short time later.

By separating message content transmission from message submission, an
arbitrary amount of time can pass between transmission and the
server's report of the acceptability of the message as a whole
(contents plus envelope plus client info like IP address), without
risking a lost connection leading to both a submission and a
subsequent attempt at delivery.  Then, once the message is approved,
the client either SUBMITs it (and *that* is where the critical point
involving potential duplicate delivery can occur), which should take
the server very little time to accomplish and return a status result,
or takes any of several other actions that cancel the pending
submission (closes the connection, or issues RSET, QUIT, or MAIL FROM,
mainly).

(A side benefit of this modification is that transparent proxies could
void a suspicious message being transmitted through them on behalf of
a suspect client without dropping the connection entirely.  That's
useful when the suspect client is a legitimate MTA innocently relaying
suspicious email.  Whereas, right now, a proxy can end the
message-content stream by either dropping the connection or implicitly
requesting submission of the message.)

So, delaying tactics could be employed up to the response to the end
of the message content (following MESG), but should never be used for
the response to SUBMIT unless the server already knows it will return
a temporary failure (since that's how a client normally interprets a
lost connection).

An SMTP protocol extended along these lines might as well allow
non-committal "continuation lines" in responses, forcing clients to
read through to the final line of each response, if that's what the
server wants.  E.g.:

> MAIL FROM:<[email protected]>
  [2-minute delay]
< 000-You know, I don't really think you are legitimately
  [2-minute delay]
< 000-sending email from that domain name.  But maybe you
  [2-minute delay]
< 000-are.  So, I'll just see how important it is to you
  [2-minute delay]
< 2xx before I accept this part of your envelope.
> RCPT TO:<...>

On a small scale, a delaying tactic like this doesn't accomplish much.

But if enough SMTP servers deploy it, in even a barely cooperative
fashion (e.g. if they exchange info on suspect or noxious sources of
spam and vermin, keyed by IP addresses), which is the sort of
cooperation and sharing that IM2000 proponents apparently envision
being necessary to successfully ostracize problematic outgoing message
stores anyway, the result could be that delivering spam and vermin
becomes expensive to the degree they're delivered, in quantity and
size of destination audience utilizing these tactics, from a given IP
address.

Sadly, however, that's not the whole story.  The problem with SMTP
even with this change is that, as long as it depends on a single
connection to make a delivery as a transactional unit, *anything* that
causes the connection to be open longer than strictly necessary for
delivery (e.g. doing all sorts of server-side checking for UBM, such
as RBL lookups, SPF lookups, DomainKeys lookup, and delaying tactics
such as the ones I describe above) can lead to loss of connection and
therefore loss of the delivery in progress.

That's because, generally, the odds of a TCP connection being lost go
up as the length of time it is open go up.  This might not reach a
point of concern, but I'd rather not assume it won't by hoping SMTP in
its present form will suffice for the next few decades and writing off
something potentially great like IM2000.

Anyway, there are two general solutions I can think of, offhand, to
solve this problem in any email system:

  1.  Allow a transaction aborted by loss of a connection to be
      continued by subsequent connections.

  2.  Explicitly separate message notification, message transmission,
      and delivery status, to at least some degree, such that the
      transaction consists of two or more distinct connections.

IM2000 essentially chooses #2, in that each of the three actions is
accomplished via a distinct connection, or transmission.  That allows
the receiver of a message notification to cleanly take arbitrary
amounts of time to decide whether to retrieve the message
(transmission), since no connection needs to remain open between
notification and transmission; and, further, to cleanly take arbitrary
amounts of time to decide whether to continue delivery (e.g. based on
scanning the contents of a message) and report success ("unpinning",
among other things), since no connection needs to remain open between
transmission and delivery-status notification.

SMTP already allows separating out of delivery-status notification to
some degree (via DSNs, or bounces), but since there's no authenticated
path for delivering those notifications, and because they're
heavyweight and not standardized, it's best to implement any
predictable criteria governing deliverability during the SMTP session,
meaning notifying the injecting SMTP client of success or failure of
the delivery.

That, in turn, leads to the requirement that a comparatively
heavyweight transaction consist of no more than a single successful
connection, which poses increased reliability risks when compared to
similar transactions that take less time (because less on-the-fly
analysis is done and/or fewer delaying tactics are employed to foil
senders of UBM).

So, to "fix" SMTP to have comparable robustness as IM2000 in these
areas, a means to resume a transaction across a lost connection would
be needed.  Done well enough, that could lead to *more* robustness
than with IM2000, since there'd often be no need to make three
successful connections (one of which goes in the opposite direction to
the outgoing message store -- representing an extra point of failure)
in order to successfully deliver a message, as IM2000 requires in its
proposed form.

But it wouldn't be a trivial modification to SMTP, as a protocol or to
SMTP clients and servers, to accommodate suspension of a transaction
upon loss of connection and subsequent resumption of it.  (E.g. load
balancers would complicate the picture, right off the bat.)

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>