UUCP, etc., and SMTP/822/MIME mail (was: Re: I-D ACTION:draft-hoffman-utf8headers-00.txt)

--On Thursday, 01 January, 2004 09:04 -0500 Keith Moore 
<[email protected]> wrote:

> On Jan 1, 2004, at 6:17 AM, Charles Lindsey wrote:
>
>> I don't think you would need to downgrade for UUCP, because
>> it is  already 8-bit clean. But my point was that a message
>> might happily  wander around within one protocol (UUCP or
>> NNTP) without anybody  needing to care about the encoding or
>> to check for "UTF-8-HEADERS".  Then suddenly it arrives at a
>> gateway into something else (e.g. SMTP  or an IMAP store)
>> where the distinction really matters.
>
> the possibility exists that the gateway isn't aware of the
> utf-8 extension, so it injects messages with utf-8 headers and
> addresses into the legacy mail system without doing a
> conversion.

Or, worse, that it "downgrades" the UTF-8 by zeroing out all of 
the high bits.   For anyone who doesn't know (I know Keith 
does), we have seen both behaviors many times.   And what this 
really says is that, if UUCP-based mail is now defined as "8-bit 
clean", it is a requirement of a gateway that conforms to RFC 
2821 that it detect the presence of 8bit characters and do 
something intelligent.  Now that requirement exists today, and 
existed long before this particular discussion and mailing list 
got started.  If a UUCP-based mail message that contains 8bit 
information in the body gets to a gateway into an [E]SMTP 
environment, it must (sorry, MUST) tag that information 
appropriately with MIME headers and must either generate an 
8BITMIME negotiation or convert the relevant body parts with 
some appropriate content-transfer-encoding.

Similarly, if there are eight bit fields in the headers, they 
had better be fields that can be tagged and converted according 
to RFC 2047, and the gateway must perform those actions.  If 
elements of the headers, such as address fields or elements that 
are not known to the gateway and defined as "text" or "word", 
cannot be converted, then the gateway must decide between some 
form of encapsulation and dropping the field -- as RFC 2047 puts 
it (end of section 1):

	It specifically DOES NOT define any translation between
	"8-bit headers" and pure ASCII headers, nor is any such
	translation assumed to be possible.

>> Yes, Email carrries more weight within IETF, and if that
>> means this  can be brought straight to standards track, then
>> I would be delighted.
>
> I think it's exactly the opposite.  email is viewed as an
> essential service; usenet isn't.  also, many people feel that
> usenet is already a hopeless mess and they haven't quite
> gotten to feeling that way about email (though there is a
> trend in this direction).  so there is considerable reluctance
> to making disruptive changes to email, whereas with usenet,
> the attitude is more likely to be "who cares?"  or "why are
> you bothering to upgrade usenet anyway?"

I would have said, instead,... The Internet's email standards 
have been designed to be as accommodating  to the requirements 
of other environments that might generate mail that will be 
injected into the Internet as reasonably possible.  However, the 
responsibility for ensuring that mail that is injected into the 
SMTP environment really conforms to that environment's rules 
rests, of necessity, with the gateways that do the conversion 
and injection.  And that is true whether the "other" system is a 
"hopeless mess" or the most wonderfully-designed environment 
around.

>> Yes, our standard will say that the code used in headers MUST
>> be  UTF-8, and other codes MUST NOT be used. That, sadly, is
>> not  sufficient to prevent it from happening. Which is why I
>> suggest that  our Foobar header should contain a possible
>> handle to indicate other  usages, though clearly that handle
>> "MUST NOT be used".
>
> at the time we were working on what became RFC 1342 we
> realized that a single header field to tag the charset used
> throughout the header would not be sufficient, because
> different parts of the header are generated by different
> agents on different machines.   one of the reasons for 1342
> was to be able to encode such things in ASCII, but another
> reason was to be able to tag each bit of human-readable text
> with a separate charset.  what we might be finding out is that
> it's not reasonable to expect everyone to use utf-8, and that
> we're going to continue to need to deal with multiple charsets
> (though perhaps fewer than are in use now) perhaps including
> different charsets in different parts of the message header.

Keith, while I agreed with (and strongly supported) that 
reasoning at the time, in the ensuing eleven or so years Unicode 
(and maybe UTF-8) have achieved sufficient adoption that it 
might now be reasonable to say "systems injecting non-ASCII 
characters into header fields or equivalent contexts are 
required to take responsibility for converting to UTF-8"... 
rather than tagging everything with what it started out being 
and then hoping that the receiving system can sort things out. 
As you are aware, MIME has gotten a bad reputation in some 
quarters as a mechanism for well-documented incompatibility 
rather than assuring interoperability.  Or, as I and others keep 
saying in other contexts: fewer options, and fewer profiles, 
lead to better interoperability.  More options tend in the other 
direction.   If we are going to something new here, it may be an 
appropriate time to draw the line, at least to the extent of 
"all headers in the same character set for a given message" and, 
ideally, to "all headers in _one_ character set".

      john