Re: Fwd: I-D ACTION:draft-hoffman-utf8headers-00.txt

On Sat, 3 Jan 2004 10:11:59 +0000, Adam M. Costello=20
<[email protected]> wrote:

> Martin Duerst <[email protected]> wrote:
>

> Charles Lindsey <[email protected]> wrote:
>
>> ...I don't like the "8:" header prefix.  In some environments (notably
>> Netnews) it would be much simpler to leave the headers in their
>> present form (otherwise, all agents will have to learn to recognise a
>> new set of headers which are really just synonyms for existing ones
>> - that could be true of mail user agents too).  The advantage of the
>> special header is that agents that don't need to be aware of the
>> distinction can just ignore it.
>
> There seems to be assumption here that existing "8-bit clean" software
> will automagically understand "UTF-8 header fields" that use the same
> field-names as existing ASCII header fields.

Eh? Of course they will, because those ASCII header fields are already=20
correct UTF-8. Nothing automagic needed there.

>  But "UTF-8 header fields"
> have not even been defined yet, and there are plenty of important
> details to work out.  All standard header fields (like To:) are defined
> by grammars that currenly allow only ASCII characters.  UTF-8 header
> fields would have different grammars.  Exactly which Unicode characters
> would be allowed, and where?  The Unicode standard recommends that
> equivalent strings be treated the same.  Will that be true for UTF-8
> header fields?  If so, it means normalization needs to be done at some
> point.  At what point?  When the field is created, or when it is parsed=
?
> Which normalization, canonical or compatible?  Or some profile of
> Stringprep?  What profile?

Right, these are issues not discussed in Pete's present draft, but they=20
need to be. So here is a stab at it:

For RFC 2822 headers, you allow UTF-8 in all phrases, comments and=20
unstructureds, and maybe in quoted-strings too.

You allow them in domains, subject possibly to some limitations regarding=
=20
allowed characters and normalization/nameprep. Precise details would need=
=20
to be looked at rather carefully, but it does not seem inherently=20
difficult. Maybe 'atom' gets redefined in the process.

You allow them in local-parts, subject to some similar limitations.

And for RFC 2822, that is ALL. Everything else (header-names, date-times,=
=20
msgids) remains in ASCII.

Then you look at the MIME headers, and maybe you find some places where=20
UTF-8 would be useful, though I think allowing them in quoted-strings=20
would probably suffice. But you would have to say something about body=20
part header fields (e.g. that they can use UTF-8 aubject to the whatever=20
downgrading rules you have set for top-level headers). Note that transpor=
t=20
of such headers is no problem in transports that already support 8BITMIME.

And then you look at headers defined in other assorted documents (e.g.=20
headers that currently allow URIs might now allow IRIs, and Usefor would=20
presumably provide a suitable rule for Newsgroups).


Next problem is implementation of MUAs.

1. You ensure that all internal data paths are 8bit clean. This is=20
probably true already in most MUAs.

2. You arrange to display (and print) UTF-8 wherever it occurs in headers=
.=20
Modern MUAs tend to use Unicode internally, so accepting UTF-8 on the=20
front should be rather easy (indeed, some present agents would likely do=20
it out of the box). If the agent does not currently use Unicode=20
internally, then you do the best you can (what do you currently do with=20
=3D?utf-8=3D...?=3D ?). If that means downgrading to hex of QP or rowns o=
f=20
"???????"s, then that will do for a start.

3. If you provide value-added services, like displaying lists of Subjects=
=20
in alphabetic order, then you do what you can. Note that sorting UTF-8=20
based on octet order actually produces tolerable results.

4. You deal with domains and local-parts. This AFAICS is the only bit tha=
t=20
might be "hard". But you are going to have to do that hard bit for IDNA=20
and for whatever other solution whis WG invents for local-parts, and doin=
g=20
it this way does not seem inherently harder than that.

--=20
Charles=A0H.=A0Lindsey=A0---------At=A0Home,=A0doing=A0my=A0own=A0thing--=
----------------------
Tel:=A0+44=A0161=A0436=A06131=A0Fax:=A0+44=A0161=A0436=A06133=A0=A0=A0Web=
:=A0http://www.cs.man.ac.uk/~chl
Email:[email protected]=A0=A0=A0=A0=A0=A0Snail:=A05=A0Clerewood=A0A=
ve,=A0CHEADLE,=A0SK8=A03JU,=A0U.K.
PGP:=A02C15F1A9=A0=A0=A0=A0=A0=A0Fingerprint:=A073=A06D=A0C2=A051=A093=A0=
A0=A001=A0E7=A065=A0E8=A064=A07E=A014=A0A4=A0AB=A0A5