Re: Unicode implementation concept

Yes of course, everyone knows that software
design is about tradeoffs.

For encodings with very low frequency of wide chars,
the scheme imposes almost no overhead while
guaranteeing compliance.  The low-frequency
argument is one I've already made myself.  It tends
to mitigate the serialization problem, the only
real drawback.

For UTF-8 in various Asian and Indian tongues
the frequency of wide characters is heavier.  In
these languages the performance gain from fixed
code point indexing is obvious, for slicing and
dicing purposes.

Overall it seems to me win-win.  Encodings close to
"fixed width" suffer no loss, while encodings
with heavy "variable width" enjoy great gains in
the context of slicing and dicing.

Already tossed out was an idea for the problem
of updating each table entry individually.  The
table struct can hold a master pre-offset for the
entire string.  Adusting this single number in
effect adjusts all table entries at once.  It
won't work for every conceivable operation, but
some.

If one is dealing with lots of wide chars,
say UTF-8 Japanese, even individual table entry
adjustments involve no more overhead than is
already considered acceptable for such strings
in terms of block moves and string scanning.
However as I've said UTF-16 should be the canonical
common format (and is what Win32 APIs want, too).

Thank you
Mark