RE: Unicode implementation concept

Edward,

The bug disadvantages of UTF-16 are the big vs. little endian issues as well
as the fact that it does not sort in Unicode sort order.  Now with
surrogates UTF-16 is an MBCS characters set with all the multi-byte
processing issues.  As such is does not offer as much over UTF-8.  However,
it is far easier to debug even in little endian systems than UTF-8.

> Back-fitting legacy code to use UTF-16 is
> often a big issue - you don't buy enough end-user
> functionality for the trouble in many cases so you
> can't justify it. Backfitting to utf-8 is bad enough
> in many cases - sometimes completely backfitting
> everything to use utf-8 can't be justified either. So
> you end up with a patchwork.

We have to provide people a clean migration method.  Client often have large
systems that are undergoing development and have to interface to legacy
system some of which they have no control over how they work.

This is why I developed xIUA.  It gives users the ability to access the
richness of ICU with minimal changes to their existing applications.  They
can later chose to convert portions of there code in pieces to use UTF-16 to
improve performance.  It give them a framework so that they can migrate
without tears.  They can write a single piece of code that may be used
throughout the application that will dynamically adapt to using codepage,
UTF-8, UTF-16 or UTF-32 data.  It also allows them to add ICU support
anywhere.  For example I can have a utility function that may nest many
layers deep in the code and is use throughout the application.  How do I
pass information such as locale without changing the world?  We can not ask
developers to have to change all APIs just to add globalization.  This was
the first major problem that my clients had.  To solve the problem I stored
this information in thread locale storage.  Once I had that mechanism
working for cross platform environments I found that I could also pass
converters.  No I could convert on the fly so that I did not have to convert
all data to UTF-16 but I could do it as needed. Then I added time zone info
because that may also vary by thread.

Converting on the fly and calling ICU as needed created another problem.
You need work areas.  Having each API calculate the needed storage too a big
burden of the developers.  It also provide a performance enhancement.  Using
malloc/free is high overhead so you need a routine the reuses the storage
but does not hold on to large blocks of storage.

To really support operating in a mix environment you also need function like
strtok that have to have separate implementations for codepage, UTF-8,
UTF-16 and UTF-32.

Back to the real world there are applications that need to process different
contexts.  For example I have a PHP script in Shift_JIS, the user browser is
running in EUC and the database uses UTF-8 and ICU needs UTF-16.  You have
to be able to switch contexts, convert between contexts and have callable
functions that operate on the data based on the current context.

By using a middle layer you can improve performance by putting the smarts
into a single place.  For example if the browser is using a char set of
UTF-8 and your database has a context of UTF-8 it recognizes that it does
not need a converter but can just move the data.

This approach also allows you better platform independence.  You can have an
OS context that reflects UTF-16, UTF-32 or UTF-8 Unicode support.  It also
lest you transform UTF-16, UTF-32 or UTF-8 without needing an ICU converter
if that is all that is needed.

Another issue is that I believe that to do i18n right you have to understand
the issues.  Most programmers for example don't understand collation.  To
give them a collator with all the whistles and bells is a disaster.
Different programmers with produce different sequences and things will not
match.  The differences can be subtle so that to code passes the QA tests
but mysteriously does not work in the field.  I believe that you need to
provide customer tailored APIs that meet the requirement of the facility but
that programmers can use consistently.  The also isolates people from
changes.  For example, ICU 1.6, 1.7 and 1.8 could produce different results.
But if you changed the calling parameters you could get the code to work the
same.

> Someday storage concerns will disappear - sure - but
> that horizon doesn't seem any closer today than 20
> years ago - and maybe it is in fact receeding.

I disagree.  With today's prices storage is not the issue.  The first
operating system that I wrote only have 600 bytes of resident code.  No one
today looks at machine language instruction timings to see how to write
their code.  We are wasteful.

People screamed that Windows resource tables were in Unicode.  Now no one
thinks twice about it.

> > Having found the character, the language need only
> > check the high
> > bit(s) which flag additional code points.  Unicode
> > requires such a
> > test in any case; it's unavoidable.
>
> Here's a minor variation on your idea - instead of
> putting the real first code point in the main array
> put a user-defined character there where the value of
> the user-defined character helps you to do the look-up
> in the 'side' array. So, for example, the
> lowest-numbered user-defined character points to the
> lowest side array character (either directly or via a
> pointer array) which, in this idea, now contains the
> full code-point sequence. There are obvious advantages
> and disadvantages to this idea (the scope of the
> meaning of the substitute is one issue) - just some
> food for thought.

This sounds like Unicode compression or (Yeeech!!) ISO-2022.

Carl