Re: Workaround to a unicode bug needed

Dear Karl,

It finally worked and I would like to thank you for your patience and your help. I could apply it to any text read from the standard input and avoid the CO options with these three lines in the beginning of the program.

use utf8;
binmode(STDOUT, ":encoding(UTF-8)");
binmode(STDIN, ":encoding(UTF-8)");

The program must read the standard input explicitly, which means that 

perl token_perl.pl text.txt

will not work. It has to be 

perl token_perl.pl < text.txt

instead.

Also, it is also pretty curious that the program worked fine in its original form, for all the characters with the exception of the quote » and that the problem disappeared when I removed the û character from the tr// list. Now everything works fine without this hack.

Thank you!!!!

Pierre