Re: First strawman for UTF-8 headers proposal

Paul Hoffman / IMC <[email protected]> wrote:

> - The dual motivations are to allow UTF-8 everywhere in the headers
> and to not bounce any messages just because they originated with UTF-8
> headers.

That's a reasonable goal, but later you say:

> - If the initiator knows the mapping for any recipient (through
> caching or an address book), they SHOULD put it in the map header.  If
> they don't include a mapping and the message hits a non-UTF-8-HEADERS
> SMTP server, the message will bounce.

I don't like that bouncing.

I don't understand why there should be such different policies for the
domain part and the local part.  For the domain part, your proposal is
willing to downgrade to an ACE rather than bounce the message.  We might
as well define an ACE for the local part too, so that there would never
be a need to bounce messages.

On the other hand, your proposal is willing to map local parts, so
that recipients who can't handle non-ASCII local parts can see a
human-friendly (non-ACE) ASCII local part.  But they still have to see
an ugly ACE domain part.  If the mapping feature is there anyway, we
might as well allow it to be used for the whole address.  The proposed
syntax could support that.

> - Updated sending MUAs will create all headers in UTF-8.

What exactly does that mean?  There are details to be worked out.
Given an old Foo: header with its old ASCII grammar, what exactly is
the new grammar?  Will canonically (or compatibly) equivalent strings
necessarily be parsed the same way?

> - Transmission is protected by a new ESMTP command, UTF-8-HEADERS.

Every protocol that carries messages will need an analogous tagging
mechanism.

> - The terminal SMTP server is responsible for knowing whether or not
> the message store can handle UTF-8 headers.

Maybe the message store can handle them (or doesn't care), but what
about the things that retrieve messages from the message store?  Or
manipulate messages in the message store?  If the message store is a
plain text file, what chaos might ensue?  Perhaps the UTF-8 headers
should be segregated somehow, so they don't accidentally fool old
software into thinking it knows what to do with them.  For example, they
could use different field names, or they could be inside a shim header.

> - Free text fields are downgraded using quoted-printable encoding;
> SHOULD be into UTF-8 charset. Downgrading MUST only be done if
> necessary.

I assume you mean encoded-words, which can use either Q or B encoding.

There is something that encoded-words can do that your UTF-8 header
proposal cannot do: encoded-words can indicate the language of the
text.  A single field can contain multiple encoded-words, each tagged
with a different language.  If one goal of the UTF-8 proposal is
to make encoded-words unnecessary, does the UTF-8 proposal need a
language-tagging capability?  Or is this not useful enough to warrant
the added complexity?

Here's one idea I had:  Rather than introduce yet another escaping
mechanism (to escape the language tags), extend an existing escaping
mechanism: the folding mechanism.  Everywhere a field is folded it could
change the language, something like this:

Subject;en: hello,
 jp: konnichi wa,
 fr: bonjour

(I'm not bothering to use the proper non-ASCII characters, because
that's irrelevant to this example.)

If the language tag is absent, there is no change.  The colon is still
required.  You can tell which fields use the extended folding mechanism
because they have a semicolon in the field name (which has always been
allowed, but has never appeared in practice).

Because of the explicit terminator (the colon), the extended folding
mechanism can allow folding anywhere, not just before white space:

Subject;en: supercalifra
 :gilisticexpialidocious

But maybe there would still be a recommendation that folding should not
happen within words/atoms when it can be avoided.

Simon Josefsson <[email protected]> wrote:

> It might be nice to use the RFC 2047 encoding instead, so that the
> header is rendered properly in MIME aware clients.

Would it be?  I think a MIME-compliant MUA cannot decode things that
look like encoded-words in unrecognized header fields, because the
client has no way of knowing whether encoded-words are allowed in that
field.

AMC