Re: Normalisation and matching

Dan Oscarsson <[email protected]> wrote:

> > But for IDN-aware and IMA-aware slots, there is no need to use any
> > particular form, because the software reading the slots knows how to
> > do all the conversions.
>
> There is no software today that knows how to read those slots as there
> is no defined protocol for those.

Very true.  All IDN/IMA slots today are ASCII-only.  Until new protocols
are introduced, the outcome of this debate has no consequence.

> But there is a lot of software handling UTF-8 and expecting it to be
> normalised form NFC.  I want to use IDN and IMA without breaking that
> software.  I can see no reason to extend my software with the complex
> and unnecessary code to do normalisation when it is so easy to send it
> normalised between systems.

That argument applies equally to all Unicode text, not just IDNs and
IMAs.  For example, it applies equally well to message bodies and web
pages.  The software you speak of sees IDNs and IMAs as generic text,
not as identifiers.  [Why?  Because what distinguishes identifiers from
generic text is that identifiers come with precise matching rules.  But
matching IDNs and IMAs requires the use of Nameprep, which includes case
folding followed by normalization.  You have described the software in
question as lacking the ability to do normalization, so it must not be
doing anything with the IDNs and IMAs beyond what it could do with any
generic text.]

Your paragraph quoted above argues that UTF-8 text should always use
NFC.  If that's true, it would imply that IDNA and IMAA should recommend
NFC *if* they recommend UTF-8.  But they don't recommend UTF-8.  Why
should they?  The whole point of using textual identifiers is so that
they can go wherever text can go, in whatever encoding is used for
text in that place.  The choice of UTF-8 (versus some other Unicode
transformation format) is a low-level text-encoding issue independent of
IDNA/IMAA.  If NFC goes hand-in-hand with UTF-8, then I would conclude
that the choice of NFC (versus NFD, or neither) is likewise a low-level
text-encoding issue independent of IDNA/IMAA.

Perhaps what you really want is a new charset, utf-8-c, which would
be just like utf-8 except that only strings in normalization form C
are valid.  Everywhere that utf-8 is used today, you would like to see
utf-8-c used instead.  Yes?

That might not be a bad idea.

But I think it's really a charset issue concerning Unicode text in
general, not an issue for any particular kinds of identifiers (like IDNs
or IMAs).

Even before utf-8-c is registered, a protocol could easily specify that
a slot contains "Unicode text in normalization form C encoded as UTF-8".
That could be done for any slot containing Unicode text, including slots
that happen to contain IDNs or IMAs.  It's not a restriction on IDNs or
IMAs per se, it's a restriction on how Unicode text is represented in
that protocol; therefore it doesn't belong in the definitions of IDN and
IMA, it belongs in that protocol spec, or in a spec for utf-8-c.  Maybe
what you really want to fight for is to change the BCP that currently
recommends utf-8 to recommend utf-8-c instead.

AMC