Re: Fetching the charset when set in meta.

On 1/5/07, Bill Moseley <[email protected]> wrote:
> $response->headers->as_string shows this:
>
>     [...]
>     Content-Length: 17604
>     Content-Type: text/html
>     Content-Type: text/html;charset=UTF-8
>     Last-Modified: Mon, 23 Oct 2006 14:32:14 GMT
>     [...]
>
> Is there a better way to grab the charset than this?
>
>     my $charset;
>     for ( $response->header('content-type') ){
>         $charset = $1 if /\bcharset=([^;]+)/;
>     }
>     my $content = Encode::decode( $charset, $response->content )
>         if $charset;

LWP already provide a method to decode the content for you:

   my $content = $response->decoded_content;

This method will also undo various Content-Encodings for you.  If you
look at the source for decoded_content you will notice that it parse
the Content-Type header using:

        if (my @ct =
HTTP::Headers::Util::split_header_words($self->header("Content-Type")))
{
            ($ct, undef, %ct_param) = @{$ct[-1]};
            $ct = lc($ct);
        }

> I'm assuming that the last Content-type header found is the charset to
> use.

The code I quoted from decoded_content picks the last one.

>
>     my ($ct, $other) = $response->content_type;
>
> is only returning the first Content-Type header.
>
>
> How do other's decode content fetched with LWP?  Do you parse out
> charset and then decode it like above?
>
>
> Doesn't that seem like something that should be part of LWP (or some
> sub-class)?
>
>     use LWP::Simple;
>     $content = get( $url );
>
> Shouldn't $content be decoded there?

That would probably be an improvement, but it might also break some
code that depends on just receving the raw binary content asis.

-- 
Gisle Aas