Re: Normalisation and matching

Adam M. Costello wrote:

>There was a lot of existing software based on the "extended ASCII"
>model, in which byte values 0-127 represent ASCII characters, and values
>128-255 respresent locale-dependent characters that are treated as
>opaque and compared exactly.  The primary goal of UTF-8 was to encode
>Unicode in a way that could be fed to this existing software.  That
>means UTF-8 was designed for use in UTF-8-unaware slots.  Obviously the
>existing software reading such slots would not know how to shorten the
>encodings, so the UTF-8 strings needed to be in shortest form before
>being put into the slots.
>
>But for IDN-aware and IMA-aware slots, there is no need to use any
>particular form, because the software reading the slots knows how to do
>all the conversions.

There is no software today that knows how to read those slots as there
is no defined protocol for those.
But there is a lot of software handling UTF-8 and expecting it to
be normalised form NFC. That is the form Unix/Linux selected for
use in UTF-8.
I want to use IDN and IMA without breaking that software. I can see
no reason to extend my software with the complex and unnecessary code
to do normalisation when it is so easy to send it normalised between
systems.

>
>One might ask why the UTF-8 shortest-encoding rule is not relaxed for
>UTF-8-aware slots.  It certainly could be, but then there would be two
>forms of UTF-8, strict UTF-8 (for UTF-8-unaware slots) and loose UTF-8
>(for UTF-8-aware slots), which is conceptually more complex, and offers
>no advantage.

That is also the reason that I whant only ONE form of UCS data.
More that one form/encoding is conceptually more complex and for software
more complex.

   Dan