Re: Fwd: I-D ACTION:draft-hoffman-utf8headers-00.txt

At 23:54 03/12/22 +0000, Charles Lindsey wrote:

>Indeed, the next big problem is how servers and other agents are to
>recognize whether any of the headers of a message contain any Non_ASCII.
>Yes, you could scan the headers of every message looking for an octet
> > 127, but that is a great expenditure of effort considering that 99.9% of
>the world's emails will have pure ASCII headers for several years to come.
>Far better to have some indication in the message that it is contains 8bit
>stuff (most likely an extra header to say so). Indeed, Mark Crispin is on
>record as saying that, if he is to have his arm twisted into having UTF-8
>headers in IMAP, he would insist on such a header).

I think such a header is not a bad idea. I don't think it's particularly
important, but if it helps, why not. As for actually scanning the headers,
I'm not sure about the 'great expediture'. If you have to scan all
headers to find the header that says it's UTF-8, doing the > 127 check
on the side is almost free.


>In addition to that, SMTP is not the only mechanism for transporting email
>(or netnews). There is UUCP. There is NNTP. There is X.400 (complete
>with complex gatewaying rules in and out). There are satellites and
>carrier pigeons and goodness knows what. Not all of these protocols will
>want to implement a UTF-8-HEADERS extension. Indeed, for UUCP and NNTP it
>is quite unnecessary, because they are 8bit clean already, and the
>upcoming NNTP draft already assumes UTF-8 (in the few places where it
>would notice).

For X.400 and UUCP, my assumption would be that things would be
downgraded anyway, which would mean to remove the header. Satellites
are not a protocol, and carrier pigeons carry paper, where we don't
even need UTF-8 :-). But in connection with NNTP, and for certain kinds
of local processing (procmail,...), it would probably make sense.
It may also ease implementation because it gives guidance for
internal (mail spool) formats.

I definitely like a header much more than the 8: header prefix
proposal, because it looks to me that it is much more straight-
forward to implement. There are no issues such as "what happens
if there is a To: and an 8:To: header?", and 8-bit-clean software
can just work on headers without having to care about 7-bit/8-bit
issues except at very specific points (downgrading/upgrading).


>But far more than that is the political advantage in having such a header.
>Today, the great bulk of the internet message system uses ASCII headers
>and nothing else. A few brave souls are determined to use UTF-8 (or,
>shudder!  GBxxxx) in their headers. OK. They should bear the cost of
>bringing it in.  That includes the trouble of having to mark their
>messages as "unclean". Of causing suitable user agents to be implemented.
>Of persuading their server admins to provide enabled POP3 and IMAP
>servers. But, most of all, to persuade SMTP servers around the world to
>carry their stuff at least without destroying/munging it. Their own user
>agents and local servers are more or less under their control. Not so the
>uncaring SMTP relays through which their messages may have to pass (we may
>assume that the bulk of the people they want to communicate with will be
>speaking their own languages, and will thus also have enabled software
>available). But to get random SMTP servers worldwide to upgrade will be a
>hard slog, and it will only be the dedicated people who want to use the
>facility who will have reason to apply the pressure to make it happen.

I can see the 'political advantage' of such a header. But I don't see
the relationship to server upgrade patterns.



>Which is why I think it better for this to be an Experimental Protocol in
>the first instance. It is less "threatening" to the IETF establishment; it
>silences those people who will not allow anything incompatible with what
>is already deployed without workarounds and kludges and yet more encodings
>already in place. By all means, if you can get it through on the standards
>track, then good luck to you, but not at the price of holding it up for 5
>years. Time is not on our side. People are already using UTF-8 (and,
>shudder!, GBxxxx) in headers because "it works for them". They are not
>going to wait.
>
>Usefor has already been through this. Internationalized newsgroup-names
>were to have been the major advance of the project. But we have been
>persuaded to remove them from the draft and to bring them forth later as
>an experimental protocol. Even though they had been shown to work without
>problem within the existing Usenet without any server upgrades.

I think there are various ways to see this. You seem to be saying
"we didn't get further than experimental for usefor, so better not
try to get it for email". But I think it is better to see this as
"usefor alone didn't make it, but email and usefor together should
make it". Email carries a lot more weight within the IETF. The main
issue with the UTF-8 extension for usefor only going to experimental,
as far as I understand, was the interaction with email. This of course
is gone once email is also moving towards UTF-8.


>So let me suggest a header so that UTF-8 users can mark their messages as
>"unclean".

I don't see anything 'unclean' in UTF-8.


>Header-Transfer-Encoding : "Header-Transfer-Encoding:" ( "8bit" / "7bit" )
>                                *( ";" parameter )
>
>OK, it needs CFWS and all that jazz in the proper places. We can argue
>later whether the operative keyword is "8bit" or "utf-8".

Allow me to start now: I think the name "Header-Transfer-Encoding"
is problematic, because it will further increase confusion about
the various encoding layers. Second, I very much think the
distinction should be between US-ASCII and UTF-8, not 8bit and 7bit.



>Note the
>optional parameters (syntax as RFC 2045) which allow extensibility. The
>only parameter I would propose initially is "language = <language-code>".
>I explicitly OMIT a charset parameter, because the REQUIRED charset for
>Non-ASCII headers is UTF-8. And I make that omission very EXPLICIT because
>it indicates to the Chinese how they could workaround using GBxxxx within
>their own borders supposing that they refuse to use UTF-8, as they most
>assuredly will.

>Some people have doubts about including a language header. I put it there
>to forestall Bruce Lilly who will otherise come before us pointing out
>that the word "boot" has different meanings in German and English, and
>more importantly by pointing out that there is an IETF requirment to
>include language specifications in all protocols. And even with that
>parameter in place, he will still complain that it does not allow
>different languages to be specified in different headers :-( .

I'm definitely very doubtful about this. There is already a
Content-Language: header, and except for the odd case where all
the headers are in one language, and the body in another, this
parameter would not add anything.


Regards,    Martin.