HTML::TreeBuilder utf8 troubles

William McKee <[email protected]> Thu, 20 Jan 2005 15:25:24 -0500

Newsgroups	gmane.comp.lang.perl.modules.petal
Message-ID	<[email protected]>

Hi Sean,

I'm been having some troubles with the TreeBuilder module when the
string that contains the data to be processed is in a utf8 format and
contains entities that have already been decoded into their Unicode
counterpart. Under these conditions, an extra character, Acirc or
\x{c2}, gets inserted before the element in question.

More curiously, this behavior only seems to apply when loading a file
from disk using PerlIO with utf8 encoding and only for some entities
(e.g. copyright and nbsp).

I have attached a sample script which demonstrates this behavior. Test 1
will generate the Acirc whereas Test 2 does not. However, you'll notice
that if the imported data is encoded back to latin1 (set
$recode_to_latin1=1), then everything is fine again. 

I'm not sure this is a bug so much as something worth noting in the POD.
Hope it saves someone else the trouble (though the time spent learning
utf8 was worth the effort).


Thanks,
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

html_treebuilder.pl (text/x-perl, 2.7 KB)

#!/usr/bin/perl -w
#===============================================================================
#
#         FILE:  html_treebuilder.pl
#
#        USAGE:  ./html_treebuilder.pl 
#
#  DESCRIPTION:  Demonstrate Acirc bug when parsing text with encoded
#  entities.
#
#      OPTIONS:  ---
# REQUIREMENTS:  ---
#         BUGS:  ---
#        NOTES:  ---
#       AUTHOR:  William McKee (http://www.knowmad.com), <william at knowmad dot com>
#      COMPANY:  Knowmad Services Inc.
#      VERSION:  1.0
#      CREATED:  01/20/05 13:23:14 EST
#     REVISION:  ---
#===============================================================================

use strict;
use warnings;
use Carp;
use HTML::TreeBuilder;
use Data::Dumper;
use HTML::Entities;
use Encode;

# Options
# setting this value to 0 will avoid the error
my $open_as_utf8 = 1;
# set to true to avoid the error
my $recode_to_latin1 = 0;

foreach my $file_name (@ARGV) {
  my $tree = HTML::TreeBuilder->new; # empty tree

  #
  # read in file as utf8
  my $data;
  {
    undef local $/;
    if ($open_as_utf8) {
      open FP, "<:encoding(utf8)", $file_name or die "Unable to open $file_name for reading: $!";
    }
    else {
      open FP, "<", $file_name or die "Unable to open $file_name for reading: $!";
    }
    $data = <FP>;
    close FP;
  }

  # Encode the $data back to latin1?
  $data = Encode::encode('latin1', $data) if $recode_to_latin1;

  # decode the entities
  decode_entities($data);
  #print Dumper($data);
  $tree->parse($data);

  # This comes out clean
  print "\nHey, here's a dump of the parse tree of $file_name:\n";
  $tree->dump; # a method we inherit from HTML::Element

  # This contains Acirc entity before the nbsp
  print "\n\nAnd here it is, bizarrely rerendered as HTML:\n",
  $tree->as_HTML, "\n";
#    print "And here are all of the elements:\n",
#      Dumper($tree->elementify), "\n";

  # Now that we're done with it, we must destroy it.
  $tree = $tree->delete;
}





print "\n\n\nTAKE 2 - From string\n\n\n";

my $data = <<"";
<html>
<head><title>Test</title></head>
<body>
<p>Ampersand: &amp;</p>
<p>Copyright: &copy;</p>
<p>Non-break space: Sticky&nbsp;Space</p>
</body>
</html>

# This encoding doesn't seem to affect TreeBuilder the same way as using PerlIO
$data = Encode::encode('utf8', $data);

my $tree = HTML::TreeBuilder->new; # empty tree

decode_entities($data);
#print Dumper($data);
$tree->parse($data);

# This looks fine
print "\nHey, here's a dump of the parse tree:\n";
$tree->dump; # a method we inherit from HTML::Element

# This causes the Acirc element to get inserted into the output
print "\n\nAnd here it is, bizarrely rerendered as HTML:\n",
$tree->as_HTML, "\n";

# Now that we're done with it, we must destroy it.
$tree = $tree->delete;

html_treebuilder.txt (text/plain, 154 B)

<html>
<head><title>Test</title></head>
<body>
<p>Ampersand: &amp;</p>
<p>Copyright: &copy;</p>
<p>Non-break space: Sticky&nbsp;Space</p>
</body>
</html>