Re: [icu-support] Re: [HELP] Strange sort order. Why?

> FUNDAMENTAL PROBLEM is that this breaks  a < b < c

This is not true. Let's step back a bit here. What you have hit is not an
issue with Unicode or ICU, it is simply a reflection of the way that
different languages sort. Now, the thing to realize about linguistic
ordering is that strings, as whole entities, still do absolutely obey the
normal mathematical requirements of a total ordering: the ≤ comparison is
transitive, reflexive, antisymmetric and total. However, what you *can't* do
is assume anything about concatenations of strings. It is just a fact about
the Slovak language, for example, that "c" < "d" < ... < "h" < "ch", so
appending an 'h' onto a string may change the order significantly. (See
http://unicode.org/cldr/data/common/collation/sk.xml).

The sorting of accents and case is also dependent on the context; these are
not (normally) taken into account unless there are no primary differences.
Of course, there are languages where some accents do count as primary
differences, such as Danish, where z < å, and there are tailorings for them
(see http://unicode.org/cldr/data/common/collation/da.xml).

Where you want to have a sort that "honors" concatenation -- such that x < y
implies x & z < y & z -- then you just can't use a comparison function that
is correct for languages. So:

1. If you need a fixed internal sort, such as for a file system, where the
internal order does not need to reflect user's expectations about sorting in
their language, and wants to honor concatenation, then just use a binary
sort according to either code point or UTF-16 order.

2. If you need to have a sort that does reflect user's expectations about
sorting in their language, then use ICU sorting according to the given
locale (which is a tailoring of UCA).

For more on these issues, see http://icu.sourceforge.net/docs/, under
Collation, and http://www.unicode.org/reports/tr10/#Introduction.

Note that a related issue to concatenation is interleaving levels (see
http://www.unicode.org/reports/tr10/#Interleaved_Levels). ICU does provide
methods for dealing with that case.

‎Mark

----- Original Message ----- 
From: "Ruslan Zasukhin" <[email protected]>
To: "Vladimir Weinstein" <[email protected]>
Cc: "jda" <[email protected]>; <[email protected]>;
<[email protected]>
Sent: Monday, February 21, 2005 14:51
Subject: [icu-support] Re: [HELP] Strange sort order. Why?


On 2/22/05 12:11 AM, "Vladimir Weinstein" <[email protected]> wrote:

You mean that I should build own collation rule ???
But I afraid problem not only in  "a á"
So this looks to be not easy task....

As far as I see now, fundamental problem is ICU compare using
     Unicode Collation Algorithm, which is a multi-level  sort
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is unusual. This break all my search algorithms which I have use for
years. Let me explain. We develop database and we build index of words.

All my life we have build index which looks as

        allocate
        anything

        big
        bravo
        brother

        close
        create
        cross

So if now I do search:

    WHERE fld >= 'bi' and fld <= 'cr'

Then it was simple as find start of range "big",
And end of range "create".
After this EACH string inside of range was considered as result.

--------------------------------------
What we see now?!

            Sadler Sánchez Saunder

Find WHERE START WITH "Sa".     => Sadler Sánchez Saunder
User say: wrong (!!!)  a <> á


Again, fundamental problem IMHO, that all my life we have write algorithms
which did compare strings LETER BY LETER. In this way did work old good
strcmp().

Now unicode offer us to compare the whole strings for primary base of chars.
And only if they are equal consider accents.... Wow.
I am in the deep shock :-(

FUNDAMENTAL PROBLEM is that this breaks  a < b < c




> Yes, if you make a tailoring that establishes a-accent as a primary
difference
> from a. There are languages where this is the case
(Croatian/Serbian(Latin),
> Slovak etc.). But for English, accented a is just an a with an accent.
>
> so your tailoring might look something like:
> &a < á
>
> Hope this helps.
>
> Regards,
> v.
>
> Ruslan Zasukhin wrote:
>> On 2/21/05 11:29 PM, "Vladimir Weinstein" <[email protected]> wrote:
>>
>> Hi Vladimir,
>>
>> Then I ask in other way. Can I get THIS ORDER using ICU ?
>>
>>         Sadler  Saunder  Sánchez            (1)
>>
>> I need this order, because I use this sorted list for range search.
>> I need find all strings that START WITH 'Sa'.
>>
>> Having order
>>
>>         Sadler Sánchez Saunder              (2)
>>
>> I get incorrect results. Because I find left point of range "Sadler"
>> Then I find right point of range "Saunder"...
>> And into result I get all 3 words including "Sánchez"
>>
>> But users say:
>>     Hey, I have choose SECONDARY option.
>>     I do not want to see accents in the result!
>>
>> So how to get order (1) ?

-- 
Best regards,
Ruslan Zasukhin      [ I feel the need...the need for speed ]
-------------------------------------------------------------
e-mail: [email protected]
web: http://www.paradigmasoft.com

To subscribe to the Valentina mail list go to:
http://lists.macserve.net/mailman/listinfo/valentina
-------------------------------------------------------------




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id396&op=ick
_______________________________________________
icu-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/icu-support