Re: if you really want utf-8 headers...

John C Klensin wrote


>	(i) Maximizes global interoperability of the mail
>	infrastructure, especially when it is actually important
>	in practice (not just in theory).
>	
>	(ii) Minimizes damage when things leak out of Unicode or
>	local CCS environments.
>	
>	(iii) Avoids driving users and mail systems toward
>	proprietary environments because they provide a better
>	experience.
>
>I think those are our goals, or should be. 

They are goals I have worked for and is the reason I only
want ONE way to encode characters. I will come back to it later on.

>Now, it seems to me that there are two main possible models for 
>getting there. 
>
>(1) We accept the conclusion that that the proprietary, local 
>CCS (which might be Unicode in UTF-8 or some other form), 
>local-header-definition, systems are out there and are going to 
>be with us forever.  We then view this strictly as a gateway 
>problem.  Given all of our other constraints, that gateway 
>problem is probably best dealt with by encapsulation, e.g.

This is like MIME/IDNS/IMAA - encapsulate non-ASCII inside ASCII
preserving much of all the problems with interoperability.

>(2) We really work on a Unicode-clean environment, supported by 
>transport option negotiation. 
Which I prefer.

Back to the goals above.
I could upgrade the goals to be applied to all Internet communication.
Global interoperability is needed for all protocols.

Keith have wondered why we need UTF-8 in headers.
The major problem with how things work today is to make
interoperability work without misunderstanding/failed character
identification. Today we have many ways to encode characters.
You have many different character encodings in use (for example
ASCII, ISO 8859-1, ISO 10646, ISO 8859-2), and many ways
to transfer these encodings between systems (for example,
URL %-encoding, MIME header encoding, IDNA), and theses ways
can be intermixed inside same character string.
This results in complex parsers and encoders. Lots of possibilities
to make mistakes in parsing, decoding or encoding.
Also lots of code and data to handle all the formats of encoding/decoding
and character sets.
When I write my software I have to spend a lot of time trying to
get everything right.

If we instead agreed on using a single character set (like UCS) and
a single transfer encoding on the wire (like UTF-8) things would be
a lot easier! For example:
  - You only need one decoder to translate from transport encoding
    into internal encoding (the one you use in your system).
  - You only need one encoder to translate from internal encoding
    to transport encoding.
  - A parser for e-mail headers need only parse e-mail related data.
    No need to parse for embedded encoded character sets.
    Removing a lot of failures that we get today.
  - A lot easier to write applications handling e-mail as all problems
    of translating of character sets are much simpler due to the fack that
    only one encoding is used for all character data.
  - Software gets smaller size due to simpler code and translation data,
    and quicker due to easier character handling.
  - In summery: simpler.
    
Moving to a clean usage using single character set for interoperability
will take time and will require gateways between legacy world and
single character world. From my view gateways are best because
then you either are in legacy world or in single character set world.
Using types of embedding results in never getting a clean simple
environment.

A simple clean gateways point between old and new could be to
define that protocols over IPv6 only use UTF-8. This would result
in applications switching between IPv6 and IPv4 handling the gateway
function and applications only in IPv6 do only have to handle the single
character set. Unfortunately I suspect it is to late to define this
simple solution so we have to do negotiation in each protocol instead.

That is my reasons for wanting a single character set to be used for
interoperability.

   Dan