Re: LWP produces unreadable characters

tom arnall wrote:
> i am trying to use LWP to fetch material from the following site:
> 
> 	http://education.yahoo.com/reference
> 			
> using, to pick an example:
> 
> 	http://education.yahoo.com/reference/dict_en_es/spanish/
> 		curia;_ylt=A9FJq_FmyklEIWYAkAD2s8sF
> 		
> the result is unreadable characters, tho' lynx does fine with it (tho', of 
> course, it doesn't return a true copy of the html), and the stuff looks fine 
> when i use the 'view source' function on the page displayed in my browser 
> window.
> 
> my code:
> 
> 	$host = 'education.yahoo.com';
> 	$base = 'reference/dict_en_es/spanish_index';
> 	$file = 'B;_ylt=Ag1EmmqDVPhfK7pmll28XUX2s8sF';

After a few minor adjustments, it works for me.

use strict;
use LWP;
use URI;

my $host = 'http://education.yahoo.com';
my $path =
'/reference/dict_en_es/spanish_index/B;_ylt=Ag1EmmqDVPhfK7pmll28XUX2s8sF';
my $uri = URI->new_abs($path,$host);

# I like a uri object as I often run in a loop and change query strings
# host etc.

my %header = (
    'Keep-Alive' => '300',
    'Connection' => 'keep-alive',
    'User-Agent' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10)
        Gecko/20050925 Firefox/1.0.4 (Debian package 1.0.4-2sarge5)',
    'Pragma' => 'no-cache',
    'Cache-control' => 'no-cache',
    'Accept' => 'image/png,*/*;q=0.5',
    'Accept-Encoding' => 'gzip,deflate',
    'Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
    'Accept-Language' => 'en-us,en;q=0.5',
    'Host' => $host,
    );

my $ua = LWP::UserAgent->new(%header);

my $res1 = $ua->get($uri);
my $page = $res1->content;
print $page;


-Tim