Re: libwww utf8 woes

[email protected] writes:

> LWP seems to have issues with fetching pages that are utf-8 encoded.
> 
> Using a simple script like
> 
>     use LWP::UserAgent;
>     use Encode;
> 
>     my $ua = LWP::UserAgent->new();
>     my $resp = $ua->get("http://bild.de");
> 
>     if(Encode::is_utf8($resp->content)) {
>         print "utf8\n";
>     } else {
>         print "no utf8\n";
>     }
> 
> shows
> 
>     "no utf8"
> 
> (meaning that although the page is utf-8 encoded, the resulting Perl string isn't)
> and it prints the warning
> 
>     Parsing of undecoded UTF-8 will give garbage when decoding entities
>     at .../LWP/Protocol.pm line 114.
> 
> which seems to be related to a message I posted last year:
> 
>     http://www.nntp.perl.org/group/perl.libwww/2006/08/msg6801.html
> 
> although there were no responses at the time.

This is indeed an outstanding bug that I don't have any good fix for
yet, but you can work around the warning by setting the 'parse_head'
attribute to FALSE:

   my $ua = LWP::UserAgent->new(parse_head => 0);

Extracting the content using the:

   $resp->decoded_content

method will decode the UTF-8 if there is a proper header to be found
in the HTTP reponse.

--Gisle