Re: Unicode implementation concept

I have my doubts as to whether the proposed structure would be generally useful.
There are very few times, in my experience, when one cares to index according to
code point instead of simply indexing by code unit.

Now, it might be in a specialized case that it is important. I can think of an
analogy; if random access to word-starts were important to me in a particular
application, it would be worth having a special data structure that indexed
strings by word starts, so that I could have random access. I would use, say,
the ICU boundary class to make a pass through the string, and set up the index.
Similarly, I could see having an index to code points, or grapheme cluster
boundaries, or syllable boundaries all being important for specific purposes.

Remember also that supplementary code points are going to be exceedingly rare as
a percentage of text. You would probably get the same performance just with a
flag on the string that you maintain that says "contains supplementary code
points". If it is off, indexing by code point would simply use the code unit
offsets; otherwise it would take a slow path.

Can you supply a more detailed example scenario that illustrates the kind of
thing you want to do with this?

Mark
__________________________________
http://www.macchiato.com
► शिष्यादिच्छेत्पराजयम् ◄

----- Original Message ----- 
From: "Mark Evans" <[email protected]>
To: <[email protected]>
Sent: Sat, 2004 Jan 24 11:21
Subject: Re: Unicode implementation concept


>
> Edward J. Batutis wrote:
>
> > Hey, I know that guy :-).
>
> Keep things anonymous for objectivity's sake.
> There is more than one language designer
> in view here.
>
>
> > They are not
> > going to rewrite their world in Unicode.
>
> We are talking about a brand-new language
> with a design goal of serving international
> needs.  There is no rewriting.
>
> Any new language in this day and age should
> offer built-in Unicode support.
>
>
> > various reasons for this - good and bad. I don't think
> > Markus's TN will sway such people much at all. They've
> > heard these arguments before (or figured them out on
> > their own).
>
> It will - there's confusion out there.  The TN will help.
> Some points are completely lost on ASCII-minded folks such
> as UTF-8 taking more space than UTF-16 for many languages.
> Expand that section and offer statistics.
>
>
> > There *are* good reasons not to like
> > UTF-16. Back-fitting legacy code to use UTF-16 is
>
> Back-fitting is not the issue at hand.  Even if it
> were, the concept would help it along.  The language
> would store strings in UTF-16 format, internally, but
> would accept other encodings as input.  So the back-
> fitting is automatic.
>
> > Dealing with
> > huge amounts of text is one case.
>
> "Hard cases make bad law."
>
> That amount of data it will mandate special-purpose
> code in any case, no matter the encoding.
>
> The point about Asian languages still applies,
> even for huge data sets (even more so in fact).
>
>
> > Someday storage concerns will disappear - sure
>
> No one is arguing that storage is unimportant.  The
> concept presented involves almost no more memory
> beyond what Unicode itself requires.
>
> Unfortunately the language designer has deceived
> himself about the nature of internationalization
> storage requirements.  He thinks UTF-8 is the way
> to go worldwide, forgetting that it takes more space
> than UTF-16 for many tongues.
>
>
> > Here's a minor variation on your idea - instead of
> > putting the real first code point in the main array
> > put a user-defined character there where the value of
> > the user-defined character helps you to do the look-up
> > in the 'side' array.
>
> Not sure that would help.  The lookup is already simple.
>
> I would construct the table as a real lookup table,
> not a 'side array.'  Array implies contiguous storage,
> but nothing says the table should have such structure.
>
> The lookup would be based on character index in the main
> array.  The table would contain the full code-unit
> sequence for each wide character stored there.  So for
> example, a pointer to one table entry points to a complete
> Unicode character.
>
> The lookup mechanism is a question.  I think basing it
> on main array index is the way to go, because the whole
> purpose of this concept is to facilitate fast indexing.
>
> For slicing and dicing purposes the table can contain
> adjustment modifiers.  If say the first 10 chars of the
> string are dropped off, then we put +10 in the modifier
> field.  When lookup time comes, we add it to the nominal
> index in the (now smaller) string array.  So each slice
> and dice operation entails adjustment of only one field,
> not all table entries.
>
>
> > Overall I like your idea a lot. The complexity of it
> > is a bit worrysome, but I like it.
>
> Good, thank you.  Personally I think it would make
> Unicode simpler.
>
> > Debugging will be a
> > bit more 'interesting' - ... now you have new ways
> > to crash
>
> If the language has intrinsic support for these things,
> then only the language designer need worry.  That is
> one of the main reasons to advocate intrinsic support.
>
> If end users want/need to shuffle bytes around, as
> for example in your huge data set scenario, the language
> allows them to do that.  Those are however rare use
> cases.  Most programmers have very basic string needs.
>
>
> > One interesting thing is that you can back-fit
> > old code by just passing the main array in and
> > forgetting about all the extra-long characters. Your
> > legacy code just has to ignore the indicator code
> > points (probably trivial).
>
> That's one reason I would not go for the idea of
> indicator code unit.  Unicode is already
> designed to 'indicate' in this way via the first
> code unit's high bits.  (Correct me if I'm wrong.)
> The flag is already there.  Using array index as
> the lookup parameter gives the other piece of
> information.
>
>
>
> > If I had to guess at the weakest point,
> > I would say that it is in converting to and from
> > 'normal' Unicode strings.
>
> See my earlier discussion about serialization
> choke points.  They are limited.  Many strings
> in many usage scenarios will not require conversion.
>
> Windows uses UTF-16 internally for its APIs.
> So it should be a no-brainer for the language
> designer to employ UTF-16 for internal string types.
>
> In most UTF-16 cases, the main array can be passed
> to Unicode APIs without serialization - it is
> already serialized if no extra wide chars exist.
> The odds of hitting a serialization choke
> point at runtime would be low, around 5% say.
>
>
> > If cost of that in the
> > running system is minimal
>
> QED.
>
>
> Thank you Ed.  I appreciate your remarks.
>
> Best regards,
>
> Mark
> _______________________________________________
> icu mailing list
> [email protected]
> http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
>