Re: Comparing inputs with source strings

On 05/11/2016 02:04 AM, Daniel Dehennin wrote:
> Karl Williamson <[email protected]> writes:
>
>> On 05/09/2016 08:53 AM, Daniel Dehennin wrote:
>>> Hello,
>>>
>>> I tried to make my Perl5 code unicode compliant after reading a post =
on
>>> stackoverflow[1].
>>>
>>> As suggested in the post:
>>>
>>>       =E2=80=9Calways run incoming stuff through NFD and outbound stu=
ff from NFC.=E2=80=9D
>>>
>>> I got a hard time finding why my Test::More was failing but displayin=
g
>>> exactly the same strings for =E2=80=9Cgot=E2=80=9D and =E2=80=9Cexpec=
ted=E2=80=9D.
>>>
>>> I finally check how UTF-8 sources are handled and found that they are=
 in
>>> NFC form, I run the following script:
>
> [...]
>
>> I'm afraid that when it comes to normalization in Perl5, you have to
>> do it yourself.  I hear that Perl6 is much friendlier in this regard,
>> but I have no personal experience with it.  Your $unistring is in
>> whatever normalization you made it when you typed it into your editor,
>> or whatever your editor did with it as you were typing.  You could
>> have typed it in NFD, but probably the most natural way to enter
>> things on your keyboard will underlying it all be NFC.
>
> That's what I finally find out in another post, normally all my inputs
> are NFD but my tests used static string to match, I declared them with
> NFD to make it explicit.
>
> I added a note in my POD to signal that the sub returns NFD strings.

I forgot to mention that if you're just dealing with collation, it may=20
be that comparisons actually work properly regardless of normalization,=20
if you are doing the comparisons within the scope of 'use locale' and=20
the locale is recognized by Perl5 to be a UTF-8 locale.  It depends on=20
the libc implementation for your platform.  There are bugs in Perl5's=20
handling of these, however, which I have fixes for, and expect to put=20
into the latest development version, called blead, within the next week=20
or two.
>
>> Normalization is tricky, and the Unicode Consortium has had to modify
>> things years after they were first specified, because no one could
>> reasonably implement what was expected.  I may tackle getting
>> normalization to be more developer friendly in future Perl5 versions,
>> but not in the next couple of years.
>
> Thanks, as soon as my little work project is working well I'll try to
> redo it in Perl6.
>
> Regards.
>