Re: Normalisation and matching

Dan Oscarsson <[email protected]> wrote:

> Well, why not allow UTF-8 encoding of ASCII and over long sequences?
> That is no problem at all, you just normalise when you need it.
>
> From what I have heard the reasons for me wanting normalised UCS is
> the same as people wanting normalised UTF-8.

It took me a few minutes to recognize the issue you are referring to.
In case anyone else is still wondering:  The basic mechanism of UTF-8
provides several encodings of each code point, where each encoding has
a different length, but only the shortest encoding is allowed, and all
others are forbidden.

The question is why forbid the longer encodings?  Why not let receivers
of UTF-8 strings shorten the encodings if they need to?  The answer lies
in the original motivation for UTF-8, as indicated in RFC 2279:

    US-ASCII values do not appear otherwise in a UTF-8 encoded character
    stream.  This provides compatibility with file systems or other
    software (e.g. the printf() function in C libraries) that parse
    based on US-ASCII values but are transparent to other values.

    UTF-8 was originally a project of the X/Open Joint
    Internationalization Group XOJIG with the objective to specify
    a File System Safe UCS Transformation Format [FSS-UTF] that is
    compatible with UNIX systems, supporting multilingual text in a
    single encoding.

There was a lot of existing software based on the "extended ASCII"
model, in which byte values 0-127 represent ASCII characters, and values
128-255 respresent locale-dependent characters that are treated as
opaque and compared exactly.  The primary goal of UTF-8 was to encode
Unicode in a way that could be fed to this existing software.  That
means UTF-8 was designed for use in UTF-8-unaware slots.  Obviously the
existing software reading such slots would not know how to shorten the
encodings, so the UTF-8 strings needed to be in shortest form before
being put into the slots.

IDNA and IMAA take the same approach for IDN-unaware and IMA-unaware
slots.  The existing software reading such slots doesn't know how to
convert between the ASCII and non-ASCII forms, so the IDNs and IMAs need
to be in ASCII form before being put into such slots.

But for IDN-aware and IMA-aware slots, there is no need to use any
particular form, because the software reading the slots knows how to do
all the conversions.

One might ask why the UTF-8 shortest-encoding rule is not relaxed for
UTF-8-aware slots.  It certainly could be, but then there would be two
forms of UTF-8, strict UTF-8 (for UTF-8-unaware slots) and loose UTF-8
(for UTF-8-aware slots), which is conceptually more complex, and offers
no advantage.

For IDNs and IMAs, on the other hand, there is a real advantage to
having two forms: ASCII (for IDN/IMA-unaware slots) and non-ASCII (for
IDN/IMA-aware slots).  The ASCII form is compatible with old software,
while the non-ASCII form is much friendlier to humans.

AMC