Re: Flex & Unicode

Hans Aberg <[email protected]> Thu, 22 Sep 2005 19:01:08 +0200

Newsgroups	gmane.comp.lex.flex.general
Message-ID	<[email protected]>

On 22 Sep 2005, at 16:42, Frans Englich wrote:

> I need a unicode-compatible Flex, and from visiting Google &  
> reading the
> manual I understand that Flex doesn't have that out of the box, but  
> one have
> to apply The unicode patch.

What you call the "Unicode patch" I think is merely a patch that  
redefines tables from being 8-bit to 16-bit. Unicode now exceeds that  
number. So it is dead.

You can use Flex with Unicode by merely feeding it a UTF'8 input  
file. Then the pattern "<string>" will match the same string in the  
generated lexer. For Unicode character classes, one has to recompute  
them in terms of UTF-8 regular expressions. I posted some such  
functions written in Haskell here earlier.

   Hans Aberg