Re: Replacing SMTP Et Al

>On Fri, Jan 30, 2004 at 05:06:26AM -0000, James Craig Burley wrote:
>
>> I can't remember who said this, but it has been said many times: it
>> may be pointless to create and deploy an NG email system, as long as
>> clients have the option of using SMTP.
>
>And before RFC821 got widely accepted, UUCP bangpaths were seen as the way 
>to do things. The great thing about SMTP is that the address format became 
>easier to handle - it addressed one of the biggest problems with UUCP mail.

That's basically before my (Internet) time; I went from being fairly
comfortable with the ARPANET circa the mid-'70s to being stuck on
PRIMENET (Pr1me's networking software, not an actual worldwide net or
anything like that) for many years, then slowly returning to the
Internet circa the late '80s.

>A NG mail system should have several objectives:
>
>- A realisation that it is not Next Generation. It is an evolution we hope 
>to present as the current generation. Putting "Next Generation" into the 
>name makes it look rubbish a decade after it's been deployed. Think Star 
>Trek here. :-)

I don't care about that right now.  I think it's fairly "easy" to
extend and/or evolve the current design; this mailing list is called
"mail-ng" for a reason.

What I'm personally interested in is a) throwing my ideas, such as
they are, out to this audience and, even more important, b) seeing
what people think is most important about a truly NG email system, one
that is designed from the ground up with the *experience* gained from
the existing installed base in mind, but not necessarily full
backwards and forwards *compatibility* with that base.

It's not that I think the new system would offer little or no
compatibility with the old.

But the problem is hard enough for a lot of very smart people to
grapple with to start off worrying about compatibility when trying to
come up with a list of requirements.

And designing compatibility into an NG system from the beginning has a
long history of severely warping the design of that NG system,
sometimes making it even worse than a more-modest extension of the
previous incarnation of that system.  (Think of how long it took
MS-Windows to evolve away from MS-DOS before it finally got good
enough for most MS-DOS advocates to switch to it; think of IA64 versus
AMD's 64-bit version of the ix86 architecture, whatever that's called;
and I'm personally familiar with many other examples from my own
history that won't be familiar to most people here.)

When an "ideal system" based substantially on achievable technology
(mainly today's IPv4/IPv6 with DNS, URLs, etc.) crystallizes in terms
of requirements, *then* let's look at whether those requirements can
be met while accommodating various degrees of backwards and forwards
compatibility.

>- It should address the biggest problems perceived with current protocols 
>without losing the things we like about the current protocols.
>
>- To reduce complexity, it may be advisable to look at extending the 
>existing protocol unless the bad things about the current protocol are so 
>terribly awful, there is no choice but to remove it.
>
>These should, in my opinion, become our mantras.

I disagree.  The *first* should, but the second should be deal with in
some other forum, as a sort of alternate approach that might be
deployable in the short run, "conscious" of the mail-ng effort to come
up with a better system with less historical baggage (so less "new"
baggage is added that'll just have to be mothballed).

I've been toying with an idea I call SMTP2004, which is the basis for
my earlier long post.  There's not really much about SMTP2004 that
makes it difficult for me and "a few good men" to nail down, specify,
document, implement, and play with.

But it's not mail-ng, so I don't really want to talk about SMTP2004
much on this list.

>> As sites deploy NG email to try it out, they will, at some point, come
>> to appreciate the ability to exercise more-fine-grained control over
>> delivery of envelopes vs. messages (header+body), enjoy better overall
>> performance, and so on.
>
>Is performance an issue?

It becomes more of an issue as SMTP servers resort to various means to
slow down incoming connections from untrusted clients, a la
Greylisting, Defer Hostility, using multi-line greetings, and so on.

Spamware/verminware will respond to such measures by improving their
own resiliency, approaching legit SMTP clients in terms of their
operational profiles.

Server sysadmins will probably respond by slowing things down even
more, as they observe the spamware/verminware connections giving up,
which rarely happens for legit email.

That, and many anti-spam and anti-vermin measures are, at present,
highly dependent on working DNS and rDNS systems, and beat up on those
system fairly severely during outbreaks, slowing things down for
everyone.

>Indeed. In fact, despite bayesian filtering, the constant addressing of open 
>relaying, the real-time blackholing we use today, most people would agree 
>spam is on the increase.

Seems to me that's the case as well.

>Well, the trick is, I think, to make sure that the spammers are going to
>find it difficult to switch - we already have laws preventing spam in most
>juridstictions, but the problem at the moment is that it is difficult to
>actually bring a prosecution because of the nature of how the spam is sent.
>Once you ensure that spammers find it hard to evade either law enforcement,
>or their mail not getting where they want it to, the motivation for valid
>users to switch becomes huge.

I'm concerned about valid users not wanting to switch to any new
system that inherently includes increased potential for false or even
malicious litigation or prosecution.

That's why I'd prefer any new email system to be about as agnostic as
the current system is, in terms of identity, transport, copyright, and
related issues.  I'm advocating introducing clear, clean, fine-grained
levels for these concepts to be dealt with, though.

>If I actually AM able to
>send you my credit card number via e-mail securely, and the tools do not
>require the skills of an undergrad in CompSci to use... well, I think we can
>see the benefits that kind of technology could bring.

Yes, though I still think that's orthagonal to email.  It's already
quite easy to do this via HTTPS, right?  But I don't know, myself, how
to set up my web server to do that; surely that's not so much a
problem of web serving being complicated (I'm running one web server,
serving three domains, one of which got real busy last week due to the
"Guninski bug" patch for qmail I posted ;-) as it is a problem of
dealing with the pertinent security-software and certificate issues.

And *that* problem would either remain for NG email, making it hard to
deploy at all if NG email required such security in its protocols, or
it'd have been made easy for HTTPS and the like.

>> ...because SMTP can and will be made *artificially* more expensive via
>> tactics such as Greylisting, Defer Hostility, Tarpitting, and so on.
>
>These are not bad things in themselves. The problem is, they are being added 
>onto a protocol that fundamentally does not care about authenticating the 
>senders of messages and being able to give control to recipients over who 
>they receive e-mail from. The tools can be added to SMTP now, but the 
>complexity of doing so is high.

I believe most of the complexity is not due to the nature of SMTP
itself, so I don't think we can significantly simplify the situation
by offering authentication built-in to NG email.

>> With NG email (as I conceive it anyway ;-), the server can indicate
>> that it can and will deliver the envelope, but that it won't take
>> responsibility for the message, yet it "will call" for it later on if
>> desired.
>
>That is one approach, but not one I would agree would actually defeat
>spammers or viruses or malware in itself. I think it would just push the
>marketing into the envelope of whatever system you design around that 
>approach - MAIL FROM: Hot Babes at xxx.com <[email protected]>

I've been assuming a proper envelope should have *no* arbitrary
information in it, meaning, no "names" in addition to actual email
addresses.

Instead, give the "email:" protocol I proposed earlier the capability
to provide full names in whatever languages and/or character sets
apply to the addresses being queried via that protocol.

So one sends email to today's [email protected] via tomorrow's system by
"injecting" the email via the protocol "email://example.com/joe".

That protocol says, in today's terms, "MAIL FROM:<[email protected]>"
and "RCPT TO:<[email protected]>".

When Joe's end of the software chain wants to look up the formal name
and other info on the envelope sender, despite it presumably being
included in some fashion in the actual message, all that software need
to do is issue an appropriate query to "email://sample.org/fred".

That provides a fairly general mechanism to handle all sorts of issues
regarding authentication, naming, "face" icons, encryption, security,
and the like.

If the envelope contains anything arbitrary, it should be in the form
of a single sequence of octets.  An email with such a sequence but no
message could be thought of as a "postcard".

And, per my earlier proposal, there'd be no guarantee the recipient
would ever see that sequence of octets: the recipient's server or mail
agent could choose to accept only the Unique Transmission ID (UTID)
portion of the envelope and request redelivery and/or the ability to
call back for it later.

A scheme this flexible would give spammers much less assurance that
their multi-million spam run would result in enough eyeballs seeing
"Hot Babes at ..." to make that run worthwhile.  It'd keep spammers
and verminware authors guessing.  Right now, they count on widespread
deployment of inflexible, slow-to-adapt systems, being used by
relatively naive, gullible people.  We can't do much about the people,
but we can make the new system more flexible and able to swiftly adapt
to changing circumstances.

Such flexibility and adaptability wouldn't necessarily hurt regular
email delivery, since, during low-spam-volume periods, the "walls"
could be lowered, allowing acceptance of entire envelopes and either
messages in single TCP sessions.

So, NG email as I'm outlining would be fairly attractive for in-house
(corporate-LAN) deployment; it wouldn't carry much of a burden that's
excessively devoted to coping with deployment on the hostile Internet.

-- 
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>
--Fix qmail's qmail-smtpd so it doesn't crash on a big header line:--
		   <http://www.qmail.org/netqmail/>