Re: Unicode implementation

Obvious to you perhaps. But your concrete problem is no example at all.

Suppose I say: I need a function on strings that gives me every letter replaced
by the square root of its codepoint.
You say: I don't think that is so common an operation that it need be generally
supported. Please provide me a concrete example of where someone needs that.
I say:

strC = replaceLettersBySquareRoot(strA)

But that is simply a restatement of the desire, no real example at all. Lots  of
software (including ICU) works fine with Unicode strings using code unit
indexing, so clearly it is possible to do, and really not that hard. I agree
that code point indexing is easier on programmers, but whether it is worth the
cost or not really depends.

>Strings are collections of letters.

Not really. Strings are collections of encoded characters. Some of them are
letters, some symbols, some punctuation, and some combining marks (that really
require interpretation in context). And what the user thinks of as a letter may
be *very* different than a code point. See http://www.unicode.org/reports/tr29/
for more information.

The main issue is that so much of text processing requires that characters be
treated in context, so for higher level processing I don't know that it is that
much of an issue whether strings are indexed by code point or by code unit.

But in the end, I agree with you: it is not worth continuing this discussion.

Mark
__________________________________
http://www.macchiato.com
► शिष्यादिच्छेत्पराजयम् ◄

----- Original Message ----- 
From: "Mark Evans" <[email protected]>
To: <[email protected]>
Sent: Sat, 2004 Jan 24 17:42
Subject: Re: Unicode implementation


> > I just don't quite see why you are coming to the conclusion you have
> > reached; that's why I asked for a concrete scenario.
>
> The obvious nature of the problem should not require me to elaborate.
> All right, a concrete problem.  Take strA and strB, perform
>
> strC = strA[1000:1200] + strB[555:-204]
>
> where the indices are Unicode characters (comprising 1-N code units
> each) and negative indices count from the rear.
>
> > True, but that doesn't mean that random access to code points is an
> > important operation.
>
> It is if:
>
> > I do have to be cognizant of code point boundaries for many operations.
>
> Which really means all of them. Strings are collections of letters.
> Neither end users nor the programmers care about code units. A code
> unit, qua code unit, is conceptually useless.
>
> Consequently any string operation needs to be cognizant of letters,
> not (merely) code units.
>
> True, libraries can be (and have been) written to manipulate Unicode
> strings in serialized format. There is nothing wrong with that. Nor is
> there anything impressive in the statement that character boundaries
> can be determined via string scanning. They can. It is just that this
> particular case involves resistance to the idea of variable-width
> characters, and string scanning time sinks, and the concept addresses
> those issues.
>
> The main question I had was whether anyone sees any particular reasons
> coming out of Unicode that would pose problems for such an
> implementation. So far I can see none, and indeed think it would lead
> to significant performance gains for UTF-8.
>
> Thanks, that's about all I have. It is getting to be more work talking
> about this idea than it will be to write it...still I appreciate your interest
everyone.
>
> Mark
> _______________________________________________
> icu mailing list
> [email protected]
> http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
>