Re: if you really want utf-8 headers...

Keith (and many others),

I've been sitting out this discussion because I've been trying 
to work through the long-term and transition cases and sort them 
out in a coherent way.  I've also been struck by the degree to 
which much of these discussions reflects an apparent lack of 
operational experience with the way email works in practice.

So let's take a few steps back:

Anything we do will cause some interoperability problems 
--whether impact on transport or software systems or as 
perceived by users -- somewhere.  "8:" headers will mess 
something up, somewhere, because of the issues you and others 
have identified (including parsing issues, header consolidation 
algorithms, special headers coming through but getting trashed 
without warnings to the recipient, and so on), even if the 
assumptions that cause the problems aren't justified in the 
standards.   And users will be furious if they see IMAA/ACE 
local-parts, even if the mail goes through, and will also be 
furious if mail that they consider to be well-formed bounces. 
While RFC 1342 and its successors were a brilliant solution 
given the constraints of the network at that time, you've 
certainly got enough operational experience, and are a keen 
enough observer, to know how much users hate actually seeing 
them (and how much abuse Quoted-printable took).   Any of these 
changes will cause problems, and will make people unhappy -- 
probably, in the short term, more people will be unhappy than 
happy about them.

That is, superficially, a really strong argument for "keep it 
all in readable ASCII" and "don't try to see what else can be 
squeezed into header formats" positions.   With MIME, there was 
clearly value-added because we had no specification for how to 
handle multimedia or non-ASCII mail bodies.  And, in that 
context, 1342 was a really nice hack because it solved the 
problems for a specific set of fields and subfields that we 
could identify as specifically intended for humans, with few 
protocol implications.

But the argument won't work for this case, not because it is 
irrational or logically wrong, but because there is _immense_ 
user pressure out there for a fully multilingual Internet, one 
in which, as I have said a few times, English is just another 
language and ASCII is just another script.  That pressure won't 
accept, at least in local, homogeneous, environments, an 
English/ASCII network with various kludges --which work if all 
of the environments are perfectly aligned, but are unworkable or 
at least ugly otherwise-- attached for other languages and 
scripts.

That leaves us with a different problem, one I think we need to 
get very serious about rather than distracting ourselves with 
discussions about how we would behave in a better, more ideal, 
world.  We are going to have UTF-8 headers, and probably EUC and 
BIG-8 and KOI and 8859-1 and 8859-5, headers, and many more. 
We are also going to have those characters in local-part 
addresses.   And, in some parts of the world, we already do, 
typically justified by "we need them", "they work in our 
environment", and other variations of the old "just send 8" 
story.  They are not going away, if only because (a) the modal 
interpersonal email message goes between people who share 
languages and scripts and (b) because the obvious proprietary 
alternatives to SMTP/822/MIME are Unicode-clean today or shortly 
will be.

The question is not whether or not people will have and use 
non-ASCII local parts and non-ASCII header fields, nor about 
whether or not that is really necessary.  The question is how we 
will deal with that fact in a way that:

	(i) Maximizes global interoperability of the mail
	infrastructure, especially when it is actually important
	in practice (not just in theory).
	
	(ii) Minimizes damage when things leak out of Unicode or
	local CCS environments.
	
	(iii) Avoids driving users and mail systems toward
	proprietary environments because they provide a better
	experience.

I think those are our goals, or should be.  If they are not, 
then we should, IMO, be discussing that issue, not solutions.

Now, it seems to me that there are two main possible models for 
getting there.  And, much as I hate (and have been resisting) 
putting alternatives out there, they are probably compatible 
(although at some cost):

(1) We accept the conclusion that that the proprietary, local 
CCS (which might be Unicode in UTF-8 or some other form), 
local-header-definition, systems are out there and are going to 
be with us forever.  We then view this strictly as a gateway 
problem.  Given all of our other constraints, that gateway 
problem is probably best dealt with by encapsulation, e.g.,
	
	1.1 We insist that gateway systems work in terms of
	Unicode and UTF-8, keeping local character sets out of
	the public Internet.  Note that this condition is _not_
	necessary (see below), but it would avoid lots of problems.
	
	1.2 We invent message/rfcNNNN, where "rfcNNNN" basically
	says "just like RFC2822, but all header fields are
	defined as being in UTF-8, not ASCII".
	
	1.3 The gateway converts all envelope addresses to IMAA
	form and encapsulates the original message using
	message/rfcNNNN, so we have a MIME body of...

	   From: "1342/2047 PersonalName"
	       <IMAA-local-part@IDNA-domain>
		To: "1342/2047 PersonalName2"
	       <IMAA-local-part2@IDNA-domain2>
	   Date: RFC2822-date
	   MIME-Version: 1.0
	   content-type: message/rfcNNNN
	   content-type-encoding: <as needed>
	
	   <original message, with original headers, in original
	form>

	I hope we can avoid it, but a charset parameter for
	message/rfcNNNN would certainly not be rocket science to
	define.
	
	1.4 Clever receiving systems notice "message/rfcNNN" and
	unwind the situation in some appropriate way, with no
	information loss.   And note that the model above is
	pure, unextended, MIME and hence causes no Received or
	Return-path issues at all.  Non-clever receiving systems
	are going to make users unhappy.
	
(2) We really work on a Unicode-clean environment, supported by 
transport option negotiation.  In that environment, the sender 
accepts the notion that i18n communication is going to occur 
only with fully internationalized environments (target systems, 
intermediate relays, etc., and for addresses, mailbox names, 
headers, and so on).   If an environment is encountered that is 
not fully internationalized, the transport at the boundary is 
going to either take on a gateway role and adopt the conversion 
above or will bounce the mail (as with 8BITMIME, etc.).

But the second option will permit all of the edge cases for 
subaddresses with non-ASCII delimiters, bidi according to strict 
Unicode rules, etc., to work in predictable and obvious ways. 
And, within environments that would otherwise shift to 
proprietary solutions, it would deliver full i18n functionality 
while permitting staying with "real" Internet mail, albeit with 
upgraded (but backward compatible with existing messages and 
addressing) MTAs and MUAs, while the IMAA conversions may still 
lose or distort some information (although the original 
information would presumably be preserved in the encapsulated 
message).

Are either of those two options wonderfully attractive from an 
architectural standpoint?  Nope.  Will we end up with as much 
interoperability as we would if everyone in the world could be 
persuaded to stay with pure ASCII (or encoded-to-ASCII) 
SMTP/MIME?  Nope.  But that second question isn't relevant in 
practice --"they" can't be persuaded-- and we either face 
reality or give up the game.

       john

p.s. For those who may be surprised or confused, option (1) 
above represents a significant change of position for me.  I 
think it identifies a role for IMAA encoding in transport that 
is not long-term unacceptably harmful... and it specifies, as 
draft-hoffman-imaa-03.txt does not, what really happens to the 
message headers and body to preserve real operational 
compatibility, not just squeeze in an addressing variation.   I 
hope it helps demonstrate that some of us are really trying to 
learn from these list messages, not just dig in and repeat the 
same arguments over and over again.  I know I'm not the only 
one, but the combination of repeated arguments and epicycle-like 
models from a few people is getting tedious.



--On Thursday, 01 January, 2004 01:34 -0500 Keith Moore 
<[email protected]> wrote:

>
> Okay, I still see zero justification for utf-8 headers.  The
> improvement in transmission and storage efficiency is
> miniscule. They make both user agents and mail transports more
> complex and less reliable, because MTAs need to have
> conversion code (which will break messages and cause delivery
> failures) and UAs need to be able to handle old messages that
> use RFC 2047 (resulting in multiple code paths and additional
> failure modes).
>...