Re: Unicode implementation concept

--- Mark Evans <[email protected]> wrote:
> Some months ago I exchanged ideas with a language
> designer so
> obsessed with performance that he would not consider
> my proposal
> to build Unicode strings directly into his language.

Hey, I know that guy :-). Actually I've met quite a
number of brilliant engineers who are highly resistant
to Unicode. They concede that it is important; they'll
support it up to a point, but that's it. They are not
going to rewrite their world in Unicode. There are
various reasons for this - good and bad. I don't think
Markus's TN will sway such people much at all. They've
heard these arguments before (or figured them out on
their own). There *are* good reasons not to like
UTF-16. Back-fitting legacy code to use UTF-16 is
often a big issue - you don't buy enough end-user
functionality for the trouble in many cases so you
can't justify it. Backfitting to utf-8 is bad enough
in many cases - sometimes completely backfitting
everything to use utf-8 can't be justified either. So
you end up with a patchwork. Also, there are cases
where UTF-16 just doesn't fly regardless. Dealing with
huge amounts of text is one case. I've worked on
several such applications, but one in particular is
designed to read in 100's of GB of text - while
keeping a considerable percent of it in memory - and
then generating multi-TB-sized databases (via bulkload
files). Telling an architect of such a product to use
UTF-16 because it is good from an Unicode-centric
ideal perspective will earn you frownie points :-).
Someday storage concerns will disappear - sure - but
that horizon doesn't seem any closer today than 20
years ago - and maybe it is in fact receeding.
Markus's document might earn a higher grade from such
folks if it contained more counter-examples, but they
aren't going to be swayed anyway.

> Having found the character, the language need only
> check the high
> bit(s) which flag additional code points.  Unicode
> requires such a
> test in any case; it's unavoidable.

Here's a minor variation on your idea - instead of
putting the real first code point in the main array
put a user-defined character there where the value of
the user-defined character helps you to do the look-up
in the 'side' array. So, for example, the
lowest-numbered user-defined character points to the
lowest side array character (either directly or via a
pointer array) which, in this idea, now contains the
full code-point sequence. There are obvious advantages
and disadvantages to this idea (the scope of the
meaning of the substitute is one issue) - just some
food for thought.

Overall I like your idea a lot. The complexity of it
is a bit worrysome, but I like it. Debugging will be a
bit more 'interesting' - one gets used to string
buffer overflows and such causing crashes in a
recognizable way - but now you have new ways to crash
:-). One interesting thing is that you can back-fit
old code by just passing the main array in and
forgetting about all the extra-long characters. Your
legacy code just has to ignore the indicator code
points (probably trivial). It will be tempting to do
that in many cases - maybe too many!

The true test of your idea will be in the details. It
will succeed or fail based on the performance of the
operations and the data that it is called upon to
support - which we don't know and maybe you don't
entirely know. If I had to guess at the weakest point,
I would say that it is in converting to and from
'normal' Unicode strings. If cost of that in the
running system is minimal I think your algorithm will
serve you well. The implementation cost would be a
concern for me - it sounds like you think it is low,
but I'd be skeptical. Also, testing will be
complicated by the fact that the plan is to support
multiple encodings (I think you said).

Good luck, and keep us posted!

Regards,

=Ed Batutis
[email protected]



__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free web site building tool. Try it!
http://webhosting.yahoo.com/ps/sb/