Re: Unicode implementation concept

OK, the web pages are nice - but what about the implementation
concept?  So far the only critique arose from a misunderstanding of
the proposal.  We are talking about a "Unicode string" data type
implementation composed of two things:

- an array of first code units, exactly one per Unicode character
  in the string

- a lookup table containing only those string characters that
  require multiple code units

If a character requires multiple code units, then (a) its first
code unit appears in the array, and (b) all of its code units
appear in the lookup table, contiguously.  (Yes this scheme entails
a small amount of memory duplication.)

If a character requires only one code unit, then (a) its first and
only code unit appears in the array, and (b) none of it appears in
the table.

The lookup table is generally going to be sparse.  Some common
cases will have empty lookup tables, in fact.  Such cases are
UTF-32, ASCII expressed as UTF-8, and UTF-16 "almost always"
insofar as UTF-16 has a very low frequency of extended characters.

The advantage is that string processing can operate on the array
with fixed indexing, just like C.

The disadvantages include

- small amounts of memory duplication

- requirements to serialize/de-serialize to/from external Unicode
  APIs at runtime

However it is my opinion that

- these disadvantages are significantly mitigated by the sparseness
  cases just enumerated - people working with UTF-16 will hardly
  notice the difference, for example

- they are acceptable in any case for "small" strings which in
  practice constitute the majority of all strings

Mark