Re: Workaround to a unicode bug needed

Pierre Nugues wrote:
> Dear Michael,
> 
>> Pierre Nugues schrieb am 06.09.2010 um 11:09 (+0200):
>>> I wrote a simple tokenizer for texts containing Latin9 characters.
>> You probably mean "non-ASCII characters". Latin9 alias ISO-8859-15 is
>> the encoding. It's worth while making a distinction here.
> I meant what I wrote. The text contains a subset of the Latin9 characters. The list I consider in tr// is:
> a-zåàâäæçéèêëîïôöœßùüûÿA-ZÅÀÂÄÆÇÉÈÊËÎÃÔ֌ÙÛܟ
> 
> It is not a subset of Latin1, because of œ and Œ. 
> The encoding of the text is UTF-8
> 
>>> When I run the program on a Mac Snow Leopard, with version 5.8.8 on
>>> the text encoded in UTF-8, Perl outputs a defective UTF-8 code for
>>> this character: <BB>
>> As someone said, given correct input acquisition, the output depends on
>> your binmode setting and your terminal. Use ":utf8" or "encoding(UTF-8)"
>> for a UTF-8 terminal.
> I tried different combinations of binmode input and output and none of them works. Here is my terminal configuration:
> Pierres-MBP-3:ch04 pierre$ locale
> LANG="fr_FR.UTF-8"
> LC_COLLATE="fr_FR.UTF-8"
> LC_CTYPE="fr_FR.UTF-8"
> LC_MESSAGES="fr_FR.UTF-8"
> LC_MONETARY="fr_FR.UTF-8"
> LC_NUMERIC="fr_FR.UTF-8"
> LC_TIME="fr_FR.UTF-8"
> LC_ALL=
> 
> The setting LC_ALL='C' does not work either.
> 
>>> ### An elementary tokenizer. Save it in UTF-8
>> Then it should have "use utf8" indeed.
> I tried this too.
> 
>> If your data is text in UTF-8, you'll want to set your filehandle
>> (STDIN in this case) to UTF-8:
>>
>>  binmode STDIN, 'encoding(UTF-8)'; # 'utf8' also works but less correct
> This does not work either.
> 
>> I don't know your goal, but consider that \w in a regular expression
>> works fine to catch words:
> I know there are work around, but I would like to find one for tr//
> Pierre
> 
> 

Attached are two files.  One is my revision of your program that seemed 
to work on my machine.  The other is the output, so you can verify that 
it worked.

I ran it on 5.8.12, which is the only 5.8 version I have installed, and 
also on the latest 5.13.4 blead.  I got the same results with both. 
Instead of worrying about the I/O, I had just put your data into the 
same utf8 program.   I did use the -CO flag on the perl command line to 
get utf8 output.

If this program works for you, the problem is either in the I/O or it is 
a bug in your perl that was fixed by 5.8.12.  If it doesn't work for 
you, then it seems that it is a bug in your perl; and I don't know how 
to work around it.  You could use a Latin9 locale with a "use locale" 
and not encode in utf8.

use utf8;

while ($line = <DATA>) { 
   $text .= $line;
}
$text =~ tr/a-zåàâäæçéèêëîïôöœßùûüÿA-ZÅÀÂÄÆÇÉÈÊËÎÏÔÖŒÙÛÜŸ'()\-,.?!:;/\n/cs;
  # The dash character must be quoted
$text =~ s/([,.?!:;()'\-])/\n$1\n/g;
$text =~ s/\n+/\n/g;
print $text;
__DATA__
»Tjuvgömmare!» säga skatorna och se ut som samvetet självt. »Vi äro 
polisbetjänter, vi. Hit med tjuvgodset!» 
»Å, tyst, era rackare! Jag är gårdsfogden.» 
»Just den rätta!» håna de.

Tjuvgömmare
!
säga
skatorna
och
se
ut
som
samvetet
självt
.
Vi
äro
polisbetjänter
,
vi
.
Hit
med
tjuvgodset
!
Å
,
tyst
,
era
rackare
!
Jag
är
gårdsfogden
.
Just
den
rätta
!
håna
de
.