Re: UUCP, etc., and SMTP/822/MIME mail (was: Re: I-D ACTION:draft-hoffman-utf8headers-00.txt)

--On Sunday, 04 January, 2004 10:59 +0100 Dan Oscarsson 
<[email protected]> wrote:

>> - Is a message tagged with Charles' header allowed to contain
>> RFC 2047 stuff? (I would propose we say: MAY contain RFC
>>   2047-encoded stuff)
>
> I prefer not to have a header, only negotiation before
> transfer of data. If a header is used (there is a problem with
> a header due to it have to be before all other headers to
> simplify message handling), the rules should be the same as
> fpr SMTP negotiated UTF-8.

Yes.  And some of the comments below are ultimately the 
strongest motivation for this.   A sending MTA has to be able to 
say "I'm about to do something strange, i.e., send non-ASCII in 
the headers. Is that ok and do you know enough to avoid screwing 
things up?" and get back an affirmative response. 
Interestingly, if affirmative responses rarely occur, that would 
be an in-the-field demonstration of what I take to be Keith's 
key argument, i.e., that, despite all of the interest in the 
engineering community and discussion on this list, implementers 
and users really don't care and sticking with ASCII, or other 
things encoded into ASCII, is quite adequate.  Like you, I don't 
believe that: I think that non-English-speaking communities, and 
especially non-Roman-script communities, will deploy this stuff 
relatively quickly in both MUAs and MTAs because they are 
convinced that they need it (just as (at least mimimal) 
implementations of 8BITMIME deployed fairly quickly in 
communities that thought it was important).  But, ultimately, 
only the marketplace can figure that one out: we should try to 
focus on sensible ways to let people do things -- which they are 
convinced are important enough that they will do them in 
non-standard ways if we don't supply a standards -- but do them 
in a safe and rational way.  And that brings me to...

> When MTAs have negotiated for UTF-8, only UTF-8 should be used
> - not RFC 2047. (the only exception to that rule could be to
> send characters not in UCS). The reason for this is to
> simplify handling of headers (parsing, decoding etc). When
> UTF-8 is negotiated no RFC2047 handling should be needed.

Dan, I don't think this is realistic, for the reasons Keith has 
cited repeatedly (even if I don't agree with his other 
conclusions).  In practice, headers are supplied in different 
software at different points in the system.  Our traditional 
rule for MTAs has been "don't tamper with what you get, but pass 
it on", a rule that recognizes the experience that, when MTAs 
try to correct or reformat message text (including headers), a 
lot of them get it wrong and create a mess.  That is also why I 
was trying to work through an "encapsulate, rather than convert" 
strategy a few days ago.

In this particular case, requiring an MTA to convert is a 
requirement that it support conversion between an arbitrary 
character encoding (in 2047 Q or B form) to Unicode and UTF-8. 
That implies that it must have tables to convert from every 
2047-valid encoding to UTF-8, which is a near-impossibility, 
especially since some of those charsets may not have unambiguous 
conversions.

The situation is obviously somewhat better when the 2047 
encoding is of UTF-8 or some other Unicode flavor.  But, still, 
I think that asking MTAs to start sorting through headers they 
receive, looking for translations to perform and then applying 
them, is just looking for trouble.   We have just had far too 
much trouble in the past with systems declaring themselves 
gateways on the slightest pretense and then thoroughly messing 
up the header environment on the assumption that they know 
better about what was really intended than the original 
submission process did.

     john