Re: HTML::Tree, Unicode, and utf-8
Paul Bijnens <[email protected]>
| Newsgroups | gmane.comp.lang.perl.modules.lwp |
|---|---|
| Message-ID | <[email protected]> |
Terrence Brannon wrote:
> I would like to know how to place Unicode character sequences in an HTML
> file whose charset is utf-8. The plain perl program below works fine for
> this purpose, but I don't know what to do to get the HTML::TreeBuilder
> version to work.
>
> Also: I am not sure if this will remain the official support channel for
> HTML::Tree now that it has changed hands, so I am cc'ing the new maintainer
> as well.
>
> # Working Program
>
> use strict;
> #use utf8;
>
> my $string = "m\x{c3}\x{b8}\x{c3}\x{b8}se";
>
> open O, '>moose.html' or die $!;
>
> print O <<"EOHTML";
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
> </head>
> <body>
> $string
> </body>
> </html>
> EOHTML
This works because perl handles all strings as byte sequences, and
there is no notion of utf8 in ths program. You just constructed a
file with the correct byte sequence in it. The interpretation of
that bytesequence as utf8 happens in the browser.
>
> # Fails to preserve unicode characters
>
> use strict;
> use HTML::TreeBuilder;
>
>
> my $string = "m\x{c3}\x{b8}\x{c3}\x{b8}se";
>
> open O, '>tbmoose.html' or die $!;
>
> my $tree = HTML::TreeBuilder->new_from_content(<<"EOHTML");
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
> </head>
> <body>
>
> </body>
> </html>
> EOHTML
>
> my $body = $tree->look_down('_tag' => 'body');
> $body->push_content($string);
>
> print O $tree->as_HTML;
>
However TreeBuilder does have knowledge of encodings, and knows that
when inserting strings that strings needs to be in charset indicated,
UTF-8 in your case. The string that you supplied however does NOT
have the utf8 flag set. So TreeBuilder needs to upgrade the simple
byte string to a unicode string (implicit upgrading assumes they were
in ISO-8859-1 or latin1).
Setting the utf8 flag on $string is enough to make the program work.
Replace "my $string =..." with:
use Encode;
my $string = decode_utf8("m\x{c3}\x{b8}\x{c3}\x{b8}se");
or just force the utf8 flag (deprecated):
use Encode;
my $string = "m\x{c3}\x{b8}\x{c3}\x{b8}se";
Encode::_utf8_on($string);
You'll need to tell perl what encoding a string is in. If you don't
perl assumes it is a byte string. Implicit upgrading from byte strings
to Unicode strings assumes that they were encoded in ISO 8859-1 (Latin-1).
In a world without character sets, (like perl before unicode), all
strings are just bytesequences. And perl handles those very good,
as it did from the early days. When you need to handle strings
as text, with accented letters etc, you need to tell perl what
encoding a bytestring is, and then perl internally upgrades the
string to a complex structure suitable for handling text, instead
of bytestrings. That text structure happens to represent text in utf8,
and in reality is not very complicated and still efficient. But it
is different from a simple byte string.
Don't assume the utf8-flag is "on" when a string is valid utf8.
That's why I used decode_utf8() to decode the bytesequence as utf8.
I've seen people make the same error when the write a utf8 string
to a file, and later read it again, and wonder why perl doesn't
recognize it as utf8. You still need to tell perl it is utf8,
e.g. by setting "binmode(FILE, ':utf8')".
I've needed to read the "perlunicode" man page several times to
understand this (or I'm just too stupid to understand it while reading
the documentation the first time).
--
Paul Bijnens, Xplanation Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: [email protected]
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************