Re: Unicode implementation concept

Mark Evans wrote:
> OK, the web pages are nice - but what about the implementation
> concept?  So far the only critique arose from a misunderstanding of
> the proposal. ...

Your idea is interesting and (I think) novel.

It is not clear to me what problem you are trying to solve. The only positive feature of this 
approach seems to be that code point indexing is efficient. However, code point indexing is rare, 
and there are several disadvantages to this scheme:

- I would not underestimate the fact that you need to convert
   a string each time you go into and out of
   standard-string-type functions.

- Discontiguous storage makes it
   + less efficient with CPU caches (access localization decreases)
   + hard to deal with in C
   + inefficient to read a multi-unit sequence

- This looks like a lot of bookkeeping for
   + small strings
   + short-lived strings
   + strings that are being modified heavily
     (insert/replace/delete operations seem expensive)

You appear to propose a general mechanism for many character encoding forms, while I tried to argue 
that most Unicode-savvy software uses UTF-16, which would not appear to benefit from such a scheme.

In particular, for UTFs:

- UTF-8: Text is either ASCII-heavy (so UTF-8 is quite efficient),
   or it is not, and then your approach bogs down because
   the lookup table is used all the time.

- UTF-16: Most texts are BMP-heavy or -exclusive, so UTF-16
   is quite efficient.
   A much simpler optimization is to store the index of
   the first supplementary code point, so that code point
   indexing is trivial below that threshold;
   for most texts, the threshold will be the same as the string length.
   (I have done this in the ICU Punycode implementation.)

- UTF-32: Fixed-width already.

This is my personal opinion.

Best regards,
markus

-- 
Opinions expressed here may not reflect my company's positions unless otherwise noted.