Unicode implementation concept

Hello, I am a newcomer to this library and not a Unicode expert.
What I have is a rather low-level development concept/question
and I believe this is the right place for it.  I wish to
float a Unicode implementation concept and ask (a) whether ICU
does anything like it and (b) whether you think it could fly.
First a little background.

Some months ago I exchanged ideas with a language designer so
obsessed with performance that he would not consider my proposal
to build Unicode strings directly into his language.  I suggested
that the language should use UTF-16 internally for all strings,
much as the ICU library does.  There could also be intrinsic
UTF-8 and UTF-32 string types, too.

I will say that I believe his performance concerns are
largely misguided, but there they are anyway.  For example, he
forgets that Asian-language strings actually have a smaller
memory footprint in UTF-16 than in UTF-8.  And he tends to
neglect the fact that intrinsic Unicode support provides him
with direct control over the performance of his language.

One can argue whether Unicode support should live in libraries
or in the language.  With respect to this particular language,
its author is already supporting a "wchar_t" type (wide char)
along with C chars, in a half-baked attempt at intrinsic Unicode
support.  He claims existing UTF-8 and UTF-16 support
on that very basis (wrongly).  So in a sense he is persuaded
that Unicode should be intrinsic, but lacks the willingness
to break with fixed character sizes.  His claim for existing
UTF-8 and UTF-16 support is a bit like claiming that language
bit manipulation features imply language support for JPEG, since
JPEG images are composed of bits.

While he cares very much about internationalization, he has a
mental block on the problem of indexing into strings.  He wants
it to work just like C at the language implementation level.  So
for example, his internal array slicing and dicing routines will
also work with strings, modulo minor tweaks - and just as fast.
My proposal, below, was intended to address that requirement.
What do you think of it?

Please forgive in advance any mistakes in terminology on my part,
or any lack of Unicode understanding - I do not claim expertise.
Feel free to enlighten me about ICU which looks like a fabulous
library.

- Mark

==============================================================

There's no hard requirement for serial bytewise storage of the
proposed intrinsic Unicode strings.  Other ways to build Unicode
strings exist.  The one offered here would do little or no damage to
the current compiler.  Really it's just a set of small additions.

Consider a Unicode string made of two data structures:  a C-style
array, and a lookup table.  The C-style array holds the first code
point for each character. The table holds only characters needing more
than one code point.  (A 'code point' meaning 8/16/32 bits for UTF
8/16/32 respectively.)  The keys to the table are the indices of the
string.  So if character #100 has extra code points, they are accessed
via some function like table_access(100).  If a character is in the
table, its first code point is still in the array, but only the first
code point.  So there is some redundancy in storage, but it is small,
and easily pays for itself with runtime efficiency and elegance.

This setup unifies C array indices with Unicode character indices.  So
the language can employ straight pointer arithmetic to find any
character in the string. Character index = array index.  String length
(in chars) = implementation array size (in elements).  These features
may address your hesitation over implementation issues that are
complex in the serial case.

Having found the character, the language need only check the high
bit(s) which flag additional code points.  Unicode requires such a
test in any case; it's unavoidable.  If flagged, the language performs
a table lookup.  This table lookup is the only serious runtime cost.
The table could take whatever form is most efficient.

* UTF-32 has no extended codes, so UTF-32 strings don't need tables.
* UTF-16 characters involve only a few percent with extended codes.
  Ergo - the table is small, and the runtime cost is, say, 2-3%.
* UTF-8 needs the biggest and most table entries, but manageably so.

A downside might be file and network serialization - but we might
skate by.  The language could supply streams on demand, without an
intermediate serialized format.  If I tell the language "write(myFile,
myString)" no intermediate format is required.  The language can just
empty the internal array and table to disk in proper byte sequence.
The disk or network won't care how the language gets the bytes from
memory.

The only hard serialization requirement would be actual user
conversion to byte arrays or Unicode APIs expecting serial data.

Serialization at choke points has a cost of (a) zero, because the
string has no extended codes (say typ. 95%+ of UTF-16 and by
definition 100% of UTF-32), or (b) an alloc plus copy equivalent,
which is acceptable for small to medium strings (another statistically
large class in software programs).

You run into problems only with large UTF-8 strings that are
frequently passed to/from Unicode APIs.  Windows uses UTF-16 so it's
no problem.  Where you find UTF-8 happening is on the web, but that
has inherent delays of its own, so the cost might go unnoticed.
Consider for example that plenty of web sites are driven with UTF-8 by
slower languages.

This scheme supports 7-bit ASCII.  An optimization could yield raw C
speed.  Put an extra boolean flag inside each string structure.  This
flag is the logical OR of all contained Unicode bit flags.  If the
string has no extended chars, the flag is FALSE, and the language can
use alternate string code on that basis.  (No bit tests, no table
lookups.)  That works for UTF-32, 7-bit ASCII, and the majority of
UTF-16 strings.

The idea can be nitpicked to death, but it's a concept.  Unicode
strings and characters will never enjoy the simplicity or speed of
7-bit ASCII.  That's a fact of life, meaning that implementation
concepts cannot be faulted on such a basis.

What would be nice is to make Unicode maximally simple and maximally
efficient for the language users.