Re: Unicode implementation concept

Edward J. Batutis wrote:

> Hey, I know that guy :-).

Keep things anonymous for objectivity's sake.
There is more than one language designer
in view here.


> They are not
> going to rewrite their world in Unicode.

We are talking about a brand-new language
with a design goal of serving international
needs.  There is no rewriting.

Any new language in this day and age should
offer built-in Unicode support.


> various reasons for this - good and bad. I don't think
> Markus's TN will sway such people much at all. They've
> heard these arguments before (or figured them out on
> their own).

It will - there's confusion out there.  The TN will help.
Some points are completely lost on ASCII-minded folks such
as UTF-8 taking more space than UTF-16 for many languages.
Expand that section and offer statistics.


> There *are* good reasons not to like
> UTF-16. Back-fitting legacy code to use UTF-16 is

Back-fitting is not the issue at hand.  Even if it
were, the concept would help it along.  The language
would store strings in UTF-16 format, internally, but
would accept other encodings as input.  So the back-
fitting is automatic.

> Dealing with
> huge amounts of text is one case.

"Hard cases make bad law."

That amount of data it will mandate special-purpose
code in any case, no matter the encoding.

The point about Asian languages still applies,
even for huge data sets (even more so in fact).


> Someday storage concerns will disappear - sure

No one is arguing that storage is unimportant.  The
concept presented involves almost no more memory
beyond what Unicode itself requires.

Unfortunately the language designer has deceived
himself about the nature of internationalization
storage requirements.  He thinks UTF-8 is the way
to go worldwide, forgetting that it takes more space
than UTF-16 for many tongues.


> Here's a minor variation on your idea - instead of
> putting the real first code point in the main array
> put a user-defined character there where the value of
> the user-defined character helps you to do the look-up
> in the 'side' array.

Not sure that would help.  The lookup is already simple.

I would construct the table as a real lookup table,
not a 'side array.'  Array implies contiguous storage,
but nothing says the table should have such structure.

The lookup would be based on character index in the main
array.  The table would contain the full code-unit
sequence for each wide character stored there.  So for
example, a pointer to one table entry points to a complete
Unicode character.

The lookup mechanism is a question.  I think basing it
on main array index is the way to go, because the whole
purpose of this concept is to facilitate fast indexing.

For slicing and dicing purposes the table can contain
adjustment modifiers.  If say the first 10 chars of the
string are dropped off, then we put +10 in the modifier
field.  When lookup time comes, we add it to the nominal
index in the (now smaller) string array.  So each slice
and dice operation entails adjustment of only one field,
not all table entries.


> Overall I like your idea a lot. The complexity of it
> is a bit worrysome, but I like it.

Good, thank you.  Personally I think it would make
Unicode simpler.

> Debugging will be a
> bit more 'interesting' - ... now you have new ways
> to crash

If the language has intrinsic support for these things,
then only the language designer need worry.  That is
one of the main reasons to advocate intrinsic support.

If end users want/need to shuffle bytes around, as
for example in your huge data set scenario, the language
allows them to do that.  Those are however rare use
cases.  Most programmers have very basic string needs.


> One interesting thing is that you can back-fit
> old code by just passing the main array in and
> forgetting about all the extra-long characters. Your
> legacy code just has to ignore the indicator code
> points (probably trivial).

That's one reason I would not go for the idea of
indicator code unit.  Unicode is already
designed to 'indicate' in this way via the first
code unit's high bits.  (Correct me if I'm wrong.)
The flag is already there.  Using array index as
the lookup parameter gives the other piece of
information.



> If I had to guess at the weakest point,
> I would say that it is in converting to and from
> 'normal' Unicode strings.

See my earlier discussion about serialization
choke points.  They are limited.  Many strings
in many usage scenarios will not require conversion.

Windows uses UTF-16 internally for its APIs.
So it should be a no-brainer for the language
designer to employ UTF-16 for internal string types.

In most UTF-16 cases, the main array can be passed
to Unicode APIs without serialization - it is
already serialized if no extra wide chars exist.
The odds of hitting a serialization choke
point at runtime would be low, around 5% say.


> If cost of that in the
> running system is minimal

QED.


Thank you Ed.  I appreciate your remarks.

Best regards,

Mark