Re: Gobo Regexp and Unicode support.

--- In [email protected], Colin Paul Adams <colin@...> wrote:
>
> >>>>> "Ted" == Ted  <kkkkg999@...> writes:
> 
>     Ted> The patch basically does three things: 1. Makes that piece of
>     Ted> code treats a Unicode point code as a character, rather than
>     Ted> a code less than 255 (CHARACTER_8), which could be a byte of
>     Ted> UTF8, not representing a Unicode character.
> 
> I don't understand you.
> Do you mean you are translating both the regular expression and the
> data to the UTF-8 byte representations?

In UC_STRING, `item_code' correctly returns Unicode points which in
the patch now are correctly appended to the string to replace. It
wasn't correct because of `item' (CHARACTER_8) returned a single byte
of UTF8 stream (known in UC_STRING object) rather than the whole
Unicode point.

> This is a trick Franck Arnaud suggested, but it doesn't always
> work. For instance, . in the regular expression will now match one
> byte, not one character.
> 
> For example, try matching the regular expression "a.a" against
> "aHa". It will match - correctly. Now substitute any single Chinese
> ideogram for H. It will not match - but it should.

I tried "a.a" to match either `aHa' or `a&#19981;a' (a Chinese char in
between), they both perfectly matched.

> Conversely, try the regular expression "a..+a" against the same two
> data strings. In thge first case it will not match (correctly). In the
> second case it will match (incorrectly).

I also tried "a..+a" to match either `aHa' or `a&#19981;a', neither of them
matched.

> There are many more problems. For instance, case insensitive matching.

Now I understand the case insensitive matching is a tough task. It is
language specific, for example, 0069(i) <-> 0049(I), but in Turkish
0069 <-> 0130.
With the patch, case insensitive matching only remains what it is.

Ted