LWP::Simple, get and getstore() cannot get same page as browser.

[email protected] (Gary Yang) Tue, 21 Jun 2011 16:40:28 -0700 (PDT)
Newsgroups perl.libwww
Message-ID <[email protected]>
I use get and getstore() to retrieve links. However, the page I got with “get” and “getstore()” is different than the page I got from the browser. To debug it, I copied and pasted the link below to the browser’s address bar. Then, I view the HTML source code from browser. I searched the keyword, offer-listing. I found nine of them.

Below is one of nine keyword offer-listing I found:

<div class="mbcOlpLink"><a class="buyAction" href="/gp/offer-listing/1567651151/ref=dp_olp_all_mbc?

Below is the URL:
http://www.amazon.com/Vocabulary-School-Student-Norman-Levine/dp/1567651151

I use the script below to retrieve page and save it at found_it.html. However, I cannot find any “offer-listing” in it. The page generated by get / getstore is different than the browser with the same URL. What did I miss? Can someone help?

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
my $link = "http://www.amazon.com/Vocabulary-School-Student-Norman-Levine/dp/1567651151";
my $found_it = get("$link");
print "$found_it\n";
my $status = getstore("$link", "found_it.html");
print "Success\n" unless is_error($status);


Thanks,


Gary