Re: About HTML unicode

John Delacour wrote:
> At 12:31 am +0800 3/12/04, He Zhiqiang wrote:
>> Now i encountered another problem,  there are a few files contains not 
>> only one charset but also two or more, for example, file1 contains 
>> japanese and chinese, if i use open() to  load the data into memory, 
>> ord and length etc.. can't correctly work! Perhasp i miss something to 
>> encode or decode the data ?
>> code:
>> #!/usr/bin/perl -w
>> use utf8;
>> open(FD, "< file1");
>> while(<FD>) {
>> chomp;
>> print "length = ".length($_);
>> }
>> close FD;
>> ----------
>> length() can not count the correct non-ASCII characters. :(
> 
> If the file is in UTF-8, then it may be in any number of _languages_ but 
> it uses only one character set -- Unicode.  So far as I know "use utf8" 
> is now redundant and ineffectual in Perl.  You will get the correct 
> character count (6 characters rather than 18 bytes) by opening the file 
> handle as utf-8 as below.

If I could say additional comment to the JD's for Zhiqiang, "use utf8" 
is just telling Perl parser that the program source file is written in 
UTF-8.
cf. <http://www.perldoc.com/perl5.8.4/lib/utf8.html>

No other effect is expected by that pragma.

Zhiqiang had to tell Perl the string is encoded with UTF-8. You should 
give length() the string which is so-called 'UTF8-flagged' form.

JD have already suggested how to enable UTF8-flag via ":utf8" I/O layer.
cf. <http://www.perldoc.com/perl5.8.4/lib/PerlIO.html>

Another way to enable the flag is to use "utf8::decode()" function.
My sample code is like below:

#!/usr/local/bin/perl -w
use 5.008;
use strict;
use warnings;

open (TXT, '<sample2.txt');
chomp(my @text = <TXT>);
close TXT;

print "utf8 flag desabled:\n";
foreach my $text (@text) {
     print length($text), "\n";
}

print "utf8 flag enabled:\n";
foreach my $text (@text) {
     utf8::decode($text);
     print length($text), "\n";
}

-- 
Masanori HATA
<[email protected]>
He's always with us!