Re: Workaround to a unicode bug needed

You need to have a 'use utf8;' statement at the beginning of your 
program to tell Perl that it is encoded in utf8.

I tested it with that, and it works.

Pierre Nugues wrote:
> Dear All,
> 
> I wrote a simple tokenizer for texts containing Latin9 characters. It does not behave as expected with the Swedish text below and I would like to find a workaround.
> 
> More precisely, perl does not remove properly the Swedish quotes: » (RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK, U+00BB) from the text. See the first character of the first line of this text.
> 
> When I run the program on a Mac Snow Leopard, with version 5.8.8 on the text encoded in UTF-8, Perl outputs a defective UTF-8 code for this character: <BB>
> I could solve the problem by removing the û character from the tr// list (LATIN SMALL LETTER U WITH CIRCUMFLEX, U+00FB.)
> Do you know of a better, cleaner way to work around this bug?
> 
> Thank you for your help
> Pierre
> --
> 
> ### The Perl Program
> ### An elementary tokenizer. Save it in UTF-8
> __BEGIN
> 
> while ($line = <>) { 
>    $text .= $line;
> }
> $text =~ tr/a-zåàâäæçéèêëîïôöœßùûüÿA-ZÅÀÂÄÆÇÉÈÊËÎÃÔÖŒÙÛÜŸ'()\-,.?!:;/\n/cs;
>   # The dash character must be quoted
> $text =~ s/([,.?!:;()'\-])/\n$1\n/g;
> $text =~ s/\n+/\n/g;
> print $text;
> 
> ___END
> 
> ### The text to reproduce the bug. Save it in UTF-8
> 
> ___BEGIN
> »Tjuvgömmare!» säga skatorna och se ut som samvetet självt. »Vi äro 
> polisbetjänter, vi. Hit med tjuvgodset!» 
> »Å, tyst, era rackare! Jag är gårdsfogden.» 
> »Just den rätta!» håna de. 
> ___END
> --
> Pierre Nugues, Lunds Tekniska Högskola, Institutionen för datavetenskap, Box 118, S-221 00 Lund, Suède.
> Tél. (0046) 46 222 96 40, http://www.cs.lth.se/~pierre
> Visiteurs: Lunds Tekniska Högskola, E-huset, rum 4134A, Ole Römers väg 3, S-223 63 Lund.
> Mon livre/My book: http://www.cs.lth.se/home/Pierre_Nugues/ilppp/
> 
> 
>