Re: Unicode implementation concept

Markus Scherer,

Thank you for the feedback.  That page about UTF-16 is helpful
and UTF-16 is clearly the way to go.

> carrying offset tables or tables of character lengths is
> doable, but inefficient.

You misunderstand the implementation concept.  It does not
involve offset tables or character length tables.  The whole
beauty is that it eliminates the need for them AND for
string scanning.

The idea allows the language to manipulate Unicode strings just
like old-fashioned C arrays with fixed-width chars.  Each array
slot contains the first code point for a full Unicode character.
The array is an array of first code points.  Hence the offset
to character N is always N, without exceptions.  The scheme permits
true random access to any Unicode character without scanning the
preceding characters.

The auxiliary lookup table is sparse, and contains only those
string characters which require more than one code point.
It might be a hash table that uses array index as the hash key.

Now this hybrid data structure is not a true Unicode string,
but for language purposes serves just as well.  That is to
say, the internal workings of Unicode are opaque to end users.
A language statement like myString[10] gives the tenth
Unicode character, extracting it either from the array or from
the lookup table, neither of which is visible to the user.

All the user needs to do is trust the language to track Unicode
properly and convert it to serialized format where required.
If and when a Unicode API call is made to some external library,
the string would be serialized in memory by the language.

Most of the time, strings will be manipulated directly in the
language such that serialization is not an issue.  All Unicode
data can be processed in this customized format.
The big trick here is to allow the language implementor to
slice and dice Unicode strings just like arrays, without having
to worry about variable length characters.

In some ways it surprises me that Unicode was not designed like
this in the first place!

Mark