Re: UUCP, etc., and SMTP/822/MIME mail (was: Re: I-D ACTION:draft-hoffman-utf8headers-00.txt)

--On Thursday, 01 January, 2004 23:58 -0500 Keith Moore 
<[email protected]> wrote:

>> We are, I think, in complete agreement: we couldn't have
>> rationally  done anything else than what we did, and the text
>> you cite explains  exactly why we made that decision.  I was
>> only suggesting that, faced  with similar decisions, but a
>> new context, today, we are not, and  should not be, obligated
>> to replicate the decision of what is now  thirteen or
>> fourteen years ago.
>
> Of course not, but nobody has really suggested that we do so.
> At the same time there are some things about email that were
> true then that remain true now - one of which is that
> different pieces of the message header are generated at
> different places by different agents, which won't all use the
> same native character encoding and won't all get upgraded to
> utf-8 at the same time.

Of course not.

Keith, might I respectively suggest that you stop firing off 
responses and, instead, try reading the notes to which you are 
responding carefully enough to understand what they really say 
before reacting to them.

I'm certainly not naive enough to believe in "all get upgraded 
to UTF-8 at the same time" and didn't suggest that.  It is 
obvious to me that we will be living with (and _should_ be 
living with) 2047 formats for some time, probably indefinitely. 
I am suggesting only that:

	(1) Anything that puts in an 8-bit header field must do
	so using UTF-8, so that it is not necessary to tag those
	header fields as to which "charset" they represent.  In
	other words, the question isn't whether everyone
	upgrades to UTF-8 at once, only whether people are
	encouraged to upgrade to other things on the way to
	UTF-8.
	
	(2) MTAs wanting to send UTF-8 headers be required to
	negotiate that capability, using ESMTP options, with
	recepients.  This eliminates what would otherwise be a
	requirement for message-body-scanning and heuristics
	about whether "binary" headers are present.  If properly
	defined, it should also eliminate having to wonder
	whether some header containing 8-bit information is
	really UTF-8 and not some random, non-conforming,
	nonsense.
	
	(3) If the "UTF-8 header" capability cannot be
	negotiated, the MTA wishing to send that information be
	required to either bounce or encapsulate the message.

What I am arguing against is getting involved with any more 
tagging at this stage, e.g., the sort of "put any charset you 
like there, just identify it" logic that 2047 uses (and for 
which I argued very strongly at the time, as did almost everyone 
else).  If binary stuff goes in, let's restrict it to UTF-8. 
As Ned pointed out, that decision was controversial when we made 
it.  I was on the negative side, for several reasons, and still 
think I was right.  But, at this point, incompatibility and 
optionality, IMO, would be a worse choice for the Internet than 
permitting UTF-8 and some other binary option, even if the 
latter were objectively better.

Note that (3) is approximately what we have today for 8BITMIME; 
it is not a new strategy or one that has been generally 
rejected.  Indeed, it seems to have worked fairly well as a 
transition strategy.  And, personally, I prefer (for the reasons 
described in my previous note) an encapsulate strategy for 
downgrades to a fancy-coding one.  One can argue for fancy 
encoding instead, but more fancy encodings feels like more 
options and complexity to me, and I think we are going to need 
an encapsulation option anyway.

Also note that we shouldn't go overboard about "different pieces 
of the message header are generated at different places by 
different agents".  The reality, you will recall, is that 
"headers" are generated or altered only under the following 
circumstances:

	(1) All base and optional headers by the generating MUA
	
	(2) A Received header and some fix-up and substitution
	headers by the initial submission MTA.
	
	(3) Received headers (_only_) by relay MTAs
	
	(4) Received headers and a Return-path, plus
	non-conforming and extension headers to accommodate
	communication with the message store or receiving MUA,
	by the delivery MTA.
	
	(5) Any header rewritten by an actual gateway to bring
	it into conformance with the spec.

Anything else is a protocol violation and, while we can (and 
should) sensibly talk about damage control in the context of 
existing practices, we shouldn't do crazy and permanent things 
in order to make (hopefully-temporary) non-conforming behavior 
work better.

    john