Re: Gobo Regexp and Unicode support.

> > But the library is using an 8-bit state automaton - it only knows
> > about 8-bit codes.
> > Are you taking the code modulo 256?

I removed that mod 256 for byte code as you see in the patch, which
mean byte code compiled from a pattern accepts Unicode points.

> > Have you tried to match ".*a.*" against a string of 256 successive
> > codepoints (not including ASCII")?

Yes, I tried the following as subjects:
1) -- 一个例子
2) -- 另一个例子子a

using pattern:
a) ".*a.*"
b) ".*另.*" 
c) ".*另{1}.*"
d) ".*另+.*"
e) "(.*)(另)(.*)"
f) "(.*)(\另)(.*)"
g) "(.*)([另a])(.*)"
h) "(.*)(\w)+(.*)"
i) ".*(子)\1.*"

For all patterns above, 1) didn't match; 2) matched
Only h) is wrong (both should match)

As you see, I tested quantities, escaping, capturing, character class
and back reference against Unicode chars 256 successive for both
subjects and patterns.
So I think the patch does a basic work turning the original 8-bit
machine into 32-bit machine.

The shorthand "\w" is definitely a problem which needs to be extended
to full Unicode too.

Another problem I have found is Gobo Regexp doesn't yet support \0, \x
or \U for full Unicode points.


> What would be interesting is to test and try to use UC_STRING with
> non-ascii characters not only for the subject, but also for the
> regexp pattern (the string passed to the `compile' routine).
> That's where we could have problems.

See above. I used all patterns and subjects as UC_UTF8_STRINGs.

(Noticed Yahoo does not display Unicode chars correctly, a form of
Unicode code point are displayed instead.)

Ted