Can we back up a bit and ask some basic questions? An alternate model

( Very long -- contains both a high-level critique and the basis 
for an alternate proposal that preserves local-part opacity and 
skips ACE forms and goes directly to UTF-8 or another common 
Unicode encoding. )

Hi.

I've been trying to follow the traffic about this proposal, and 
am frankly overwhelmed.  I hope I'm not addressing anything that 
has already been discussed..., but I haven't seen such traffic.

(1) A context for email internationalization and a critique of 
IMAA's starting assumptions...

IMAA starts from the assumption that the right way to handle 
domain names is, with slight modifications, the right way to 
handle email local-parts.  The debate, with few exceptions, has 
been about details and options given that choice.  I'd like to 
move up a half-level and suggest that choice may be the wrong 
one, and that there are other options that will serve us, and 
Internet email generally, better.

A different way to put my concern is that I wonder whether, with 
the IDNA-hammer in hand, email is just the nearest handy thing 
that can be construed as a nail.

Several of the things that imposed constraints on the DNS 
solution are different for the local-part (LHS) of email.  That 
may point to a different solution.   The two important examples 
that occur to me include:

(i) Single-turnaround UDP transactions versus multistep, 
TCP-based, SMTP.

Because of the use of UDP, and some other things, there is no 
possibility for the DNS client and the server to interact about 
capabilities, form of names, etc.  While multi-hop (relay) 
transmissions of mail complicates things, we do precisely those 
types of negotiations in SMTP all the time.  In some respects, 
almost every mail command is such a negotiation.  But, more 
important, we've got well-established (and, by now, widely 
deployed) precedents that permit clients to send extended 
materials only to servers that are prepared to deal with them. 
That is a feature, not a bug.  It is not something we can do 
with the DNS, which was one of the major reasons why IDNA was 
needed.

(ii) An established DNS syntax versus an opaque local-part

For the DNS, the syntax of names used by client and server is 
fairly well specified.  While, in theory, a client could somehow 
use, e.g., slashes or equal signs to separate labels in 
presentation, but parse things appropriately for DNS queries and 
responses, in practice it just doesn't happen.  And someone 
might be able to read the specifications to prohibit such 
variations in a lot of cases.

With email, the idea of a completely opaque local-part -- one 
that can be internally parsed, decomposed, and interpreted only 
by the delivery MTA -- is required by existing standards.  More 
important, we have taken huge advantage of it over the years to 
do all sorts of interesting things.  Many of the ones that were 
very important some years ago have fallen into disuse, but it 
would be, IMO, very unfortunate, and very dangerous, to discard 
the capability.  Of course, if doing so were the only way to get 
internationalization, that would be worth considering anyway. 
But it isn't, as a sketch of a counterproposal below will, I 
hope, demonstrate.

There has been a good deal of discussion of subaddresses on this 
list, but they are certainly not the only case, or even the most 
important one.  We have used trickily-written local-parts to 
embed routing information (with the %-hack and bang-strings), to 
express X.400 addresses (MIXER and the UFN work), to implement 
Dan Bernstein's idea that binds return-paths to target addresses 
(the idea is interesting and useful, regardless of how the 
spammers use it), to implement portions of a variety of services 
whose commands are transported through email, and so on, for a 
long list of brilliant ideas, cheap hacks, and most points in 
between.

The variety of those techniques is such that, unlike the DNS, we 
can't reasonably say "no sane person would construct a name such 
as XN--abcDeF" and then look at enough zone files to increase 
confidence that no one has.  With email, we've had people 
working for years to find strange combinations and sequences of 
characters that could safely be used to delimit important (to 
them) information in different contexts.   It would be 
impossible to claim that every combination has been used, but a 
plan that requires uniform interpretation of the local-part is 
in a lot more danger of wrecking something that some group or 
enterprise cares deeply about than with IDNA (or, more 
generally, prefixed-ACE) strings in the DNS.

Even were "no human would construct such a name" true, it 
wouldn't help.  Some of these systems use a hash of the origin 
address, or a variant on the message ID, or part or all of the 
message itself.  Incorporating hashed or encoded identifying 
information in addresses goes back at least to some of the 
proto-groupware projects of the late 70s and early 80s.  And the 
most aggressive of the "user name doesn't go in the email 
address because it discloses information" crew have been using 
random or encrypted strings instead.  For years.

Remember, too, that case-insensitivity for local names isn't 
something the protocol calls for.  It is a suggestion for those 
mail servers who support email address that contain people's 
names or other obvious "name" or "word"-type strings.  But the 
MTAs that support non-case-sensitive names have, historically, 
done it in a number of ways.  I've seen systems in which 
local-parts such as

    John.C.Klensin
    John.Klensin
    john.klensin
    klensin
    J.Klensin
    J.C.Klensin

and so forth, all match --and do so by algorithm, not alias 
files-- but where

    johN.klenSin

is going to bounce.  Maybe that is reasonable, maybe it is 
perverse, but it is certainly conforming.

There are at least a couple of systems and configurations in 
which a leading underscore on the name implies something like 
"use it, but don't pass it through aliasing or translation". 
And a leading tilde means something else -- Ned would remember, 
but I've blocked it out at the moment.

There are many more of these examples.  There are enough of them 
that one just can't give up opacity -- or shouldn't want to-- 
unless there is no alternative that still gives us 
internationalized names.  The question is not "how character X 
should be handled", but "can we get away with interpreting the 
local part in the sending MUA or MTA".  And the answer, I 
suspect, is "no".

To put this differently, a prime condition for IDNA was that it 
not require changes in the protocols or assumptions of the DNS. 
A stated condition for IMAA is that it not require changes to 
the mail infrastructure.  But weakening or crippling the "opaque 
local part" rule does as much, or more, violence to the email 
infrastructure than, e.g., just putting UTF-8 in the addresses. 
I'll come back to that.

Again, if this were the only way to do internationalization, it 
would be worth considering the risk of breaking things.  But it 
isn't.  Let me outline an idea, just to prove that one exists 
(if it gets any traction, I'd be happy to try to write, or 
collaborate on, an I-D).


(2) A different set of assumptions

What we have usually said about email, especially at the mail 
transport (SMTP) level, is that, if the sender wants to do 
something that is (historically) unusual, it must get the 
permission of the receiver first.   That permission originally 
came out of band -- as a private agreement among consenting 
adults-- and then we introduced ESMTP to provide an in-band 
permission mechanism that could actually scale.  If, as the 
administrator of an MTA, I want to accept "XN--AbCdEf" as a 
local-part for one of my mailboxes, I can do that today -- all I 
need to do is to put such an address in my alias files.  And, if 
my MUA is configured to see it and turn it into a displayable 
Unicode string, that is my problem or feature and not anyone 
else's.  The problem arises if the sender wants to type in a 
Unicode (or local CCS) string in the hope of getting the right 
one to me.  And that is a mess, with a large selection of loose 
ends and tradeoffs, as 250 or so messages in less than five days 
attests.

But we don't need to do it.  With one change --and a different 
way of looking at the problem-- International local parts do not 
need to be handled differently from ASCII ones.  That is how, 
IMO, it should be.  It is is not, we either break existing mail 
conventions, or put non-ASCII users at a permanent disadvantage 
relative to ASCII ones.  Those two options, to use a technical 
term, stink.

So, instead, let's assume

* We want to simply move to Unicode strings in local-parts.

* There isn't really a lot of point in sending off 
strangely-encoded addresses to systems that can't handle them. 
Gibberish is gibberish, and users don't much like gibberish.  If 
you are going to send me a message in English, and assume I can 
read English and that you might want a reply, you are better off 
using ASCII addressing.  If you are going to send me a message 
in Klingon... well, I can't render it or read it, and whether 
you expect me to have an address in Klingon, or want to put 
Klingon characters in your return address (reverse-path or From: 
field), is going to be the least of our problems.  Indeed, I 
would _prefer_ that it get dumped by some MTA because, if it 
doesn't, I want my little antispambot to toss it in the same bin 
with

  From: "=?x-unknown?Q?=C1=F6=BE=D0=BB=F7=B4=DE?="
  Subject: =?x-unknown?Q?[=B1=A4=B0=ED]=C7=D1=B9=E6 [...]

(chosen at random from the most recent bin-addition)

* We don't want to destroy the "opaque local part" principle -- 
even with Unicode local-parts, the sender needs to either know 
the exact form of the mailbox name or the alias and conversion 
rules adopted by the recipient MTA.  Guessing what those rules 
are will work sometimes, just as they do in today's ASCII 
environment. And sometimes it won't... ditto.

* Unlike the MIME situation, where we were trying to be sure 
that users could get multilingual and multimedia messages even 
if their system mail adminstrators were slow about upgrading, no 
one is going to get a mailbox that can be reached with a 
non-ASCII string unless some system administrative process 
happens.  So the upgrade question is rather different, unless 
people are really worried about ASCII mailbox names (forward 
paths) but i18n addresses in [2]822-level "From:" and "To:" 
fields.    I don't think that is a major issue, but, if it is, 
we should probably be looking at RFC2047-like updates.



(3) Strawman semi-proposal

This is not a proposal, because there are still loose ends (some 
identified below).  But it should be sufficient to act as an 
existence proof that there is a plausible alternative to IMAA, 
one that meets the assumptions/ conditions above.

(a) We define a new SMTP extension.  For purposes of discussion, 
let's call it UTF8ADDRESSES.

	Loose end a.1: This probably won't work unless 8BITMIME
	also works.  That may need to be specified.  Conversely,
	perhaps we could extend 8BITMIME so that this became
	just a parameter somewhere.  I haven't thought that
	through and it doesn't make a lot of difference right
	now.
	
	Loose end a.2: I'm assuming that UTF-8 is the right
	choice here.  It may not be, although we should really
	pick one, and only one, encoding.  But there is no
	reason that I can see to force an ACE; 8-bit characters
	should be fine, modulo the limitation in (b.1).  If
	UTF-8 isn't the right answer, make appropriate
	substitutions elsewhere in this note.
	
	Loose end a.3: Because of the opacity requirement, I
	believe that even IMAA would ultimately require an ESMTP
	extension and negotiation to work properly.  So, in some
	ways this proposal and IMAA are compatible and
	complementary if an IMAA ACE string is used rather than
	UTF-8.   I think that would be overkill unless we really
	like that hammer.

(b) If a server advertises UTF8ADDRESSES, the local-part 
definition is changed so that the characters in local parts are 
construed as being Unicode in UTF-8 (or whatever is chosen), but 
are otherwise left essentially unchanged.  E.g., the special 
rules about @ (ASCII 0x40), " (ASCII 0x22), \ (ASCII 0x5C), 
etc., simply get promoted to U+0040, U+0022, and so on.

	Loose end b.1: I think UTF-8 helps here, because any
	occurrence of ASCII characters gets represented as the
	relevant single octets.  So the string is fairly easy to
	parse.  Other codings would need to ensure that there
	was no confusion with anything 2821 thinks is a
	delimiter.

	Loose end b.2: some tidying will have to be done to the
	2822[bis] text to make this work, but I don't think it
	is rocket science.  The 2821[bis] tuning is trivial,
	since the work would be done in the extension document.

(c) The parsing rules don't change.  Everything to the left of 
[unquoted] "@" is the local-part, everything to its right is a 
domain name.  Local-parts are opaque and interpreted only by the 
delivery MTA, modulo the quoting rules.  And they normally don't 
get interpreted by anything else either.

	Loose end c.1: Life would be a good deal easier if any
	sender taking advantage of this feature were flatly
	prohibited from using source routes.  It wouldn't harm
	anything that I can imagine, and it would make it a lot
	easier to safely decompose the string.
	
	Loose end c.2: I can't see any particular reason why the
	domain name in this arrangement would be forced to be
	transmitted in punycode (as an ACE).  UTF-8 would
	probably work as well.  Or one might be able to write
	the spec to permit either punycode or UTF-8 (or
	whatever) on the RHS.  Of course, this doesn't change the 
	requirement for nameprep.

	Loose end c.3: The opacity principle probably prevents
	rules about folding, special provisions for full width
	characters, etc.  If those are used, and the delivery
	MTA doesn't match them up as intended (whatever that
	means), the mail would be undeliverable (just like today
	with ASCII).  Some mappings/foldings would be wise for
	receiving MTAs to support, just as case-insensitive
	ASCII handling is wise, and others would be stupid.  We
	should give advice, but opacity prevents requirements.
	Nor, IMO, are requirements needed.

(d) An originating or relay MTA that received a forward-path 
address containing non-ASCII characters, but that discovered the 
next MTA in sequence didn't advertise UTF8ADDRESSES would more 
or less follow the rules for 8BITMIME relaying.  I.e., it would 
either have to find a valid address (presumably ASCII) it can 
forward for delivery, or find a routing path that would permit 
sending the UTF-8 addresses, or bounce the mail because it has 
no clue how to process it.

	Loose end d.1: The way such an MTA gets the information
	needed for those first two options is outside the scope
	of the standard.  That isn't much different from the
	situation today when a relay MTA gets an address it
	can't figure out how to parse because some rule or other
	is violated.  And it is better for something that can
	explain what is happening to bounce the mail than to
	deliver it to something that might blow up on the 8-bit
	characters and not return any non-delivery information.
	Accepting the message and then blowing up of course
	violates the standard, but that fact and a dollar will
	get you...

(e) The issue of what the delivery MTA actually puts in the mail 
store, and how the receiving MUA(s) handle that, has never been 
the subject of Internet protocols and that should probably not 
change.  But, again, we can give advice.    It seems to me that 
a great deal of this week's discussion could usefully be turned 
into that advice.  E.g., "Dear sysadmin, if you configure your 
MTA so that you have mailboxes whose names contain characters 
that look like quotes or the at-sign, you are inviting big 
trouble".  It also seems to me that IMAA might well turn out to 
be (or be easily transformed into) a good delivery MTA -> 
mailstore or delivery MTA-> final MUA protocol for MUAs that 
have not been updated.  But, again, these are basically 
local-machine and user interface issues, which are traditionally 
outside IETF scope.


(f) There are some other issues here, but that is the general 
picture.  For example, we would have to _very_ carefully work 
out what went into reply messages, given that those might hit a 
host that wasn't prepared to have i18n characters in them.  But 
that is a problem to be worked out, not a showstopper.

Comments?

     john