Re: UUCP, etc., and SMTP/822/MIME mail (was: Re: I-D ACTION:draft-hoffman-utf8headers-00.txt)

--On Thursday, 01 January, 2004 14:11 -0500 Keith Moore 
<[email protected]> wrote:

>> > at the time we were working on what became RFC 1342 we
>> > realized that a single header field to tag the charset used
>> > throughout the header would not be sufficient, because
>> > different parts of the header are generated by different
>> > agents on different machines.   one of the reasons for 1342
>> > was to be able to encode such things in ASCII, but another
>> > reason was to be able to tag each bit of human-readable text
>> > with a separate charset.  what we might be finding out is
>> > that it's not reasonable to expect everyone to use utf-8,
>> > and that we're going to continue to need to deal with
>> > multiple charsets (though perhaps fewer than are in use
>> > now) perhaps including different charsets in different
>> > parts of the message header.
>>
>> Keith, while I agreed with (and strongly supported) that
>> reasoning at the time, in the ensuing eleven or so years
>> Unicode  (and maybe UTF-8) have achieved sufficient adoption
>> that it  might now be reasonable to say "systems injecting
>> non-ASCII  characters into header fields or equivalent
>> contexts are  required to take responsibility for converting
>> to UTF-8"...
>
> that's onerous but more-or-less doable.  what doesn't seem
> doable is to prevent subsequent systems that handle the
> message (or reply to it) from adding their own, non-utf-8,
> header contents.

I'm not sure I see the issue.  At one level, nothing can prevent 
anything or anyone from adding trash, anywhere they like.  In 
particular, nothing prevents someone from putting things into 
2047 (or content-type: text/plain, charset=foo) form and lying 
about the charset in use today.  If they do, they are violating 
the standard and screwing their users, but, obviously, some 
folks won't care.   At another level, if there is a 
specification that says "if you add 8bit header content, it must 
be UTF-8; anything else must either be converted into RFC 2047 
form or must be converted to UTF-8", then we are probably ok. 
If receiving systems are going to interpret any 8bit content 
they get as UTF-8 (or invalid, if it doesn't meet UTF-8 coding 
rules) then people (or sending MUAs or MTAs) who violate that 
rule are just going to screw their users.  This is a "today, any 
8bit info is invalid; when we make it valid, we are going to 
define the one valid case" situation not one in which codings 
that are valid today are suddenly given a new meaning or 
interpretation.

We _could_ have adopted a rigid, Unicode-only, rule a decade ago 
rather than doing charset-specific tagging in text content types 
and 2047 encodings.  It would have made many things more simple. 
But it would not have been practical, since, despite the claims 
and optimism of their advocates, Unicode (in UTF-8 form or 
otherwise) wasn't nearly widely enough deployed.   But, in the 
ensuing years, it has gotten more widely deployed and, at least 
as important, we have made a number of other decisions and 
standards, including IDNA, that assume it.   So a "UTF-8 only" 
decision just takes a step further down a path to which we are 
clearly already committed.

   regards,
      john