Re: LWP::Simple, get and getstore() cannot get same page as browser.

On 22/06/2011 00:40, Gary Yang wrote:
>
> I use get and getstore() to retrieve links. However, the page I got
> with “get” and “getstore()” is different than the page I got from
> the browser. To debug it, I copied and pasted the link below to the
> browser’s address bar. Then, I view the HTML source code from
> browser. I searched the keyword, offer-listing. I found nine of
> them.
>
> Below is one of nine keyword offer-listing I found:
>
> <div class="mbcOlpLink"><a class="buyAction" href="/gp/offer-listing/1567651151/ref=dp_olp_all_mbc?
>
> Below is the URL:
> http://www.amazon.com/Vocabulary-School-Student-Norman-Levine/dp/1567651151
>
> I use the script below to retrieve page and save it at
> found_it.html. However, I cannot find any “offer-listing” in it. The
> page generated by get / getstore is different than the browser with
> the same URL. What did I miss? Can someone help?
>
> #!/usr/bin/perl -w
> use strict;
> use LWP::Simple;
> my $link = "http://www.amazon.com/Vocabulary-School-Student-Norman-Levine/dp/1567651151";
> my $found_it = get("$link");
> print "$found_it\n";
> my $status = getstore("$link", "found_it.html");
> print "Success\n" unless is_error($status);

The amazon.com site makes extensive use of client-side JavaScript, which
changes the displayed page comprehensively once it is loaded.

It is also likely to be contrary to the Amazon site rules to access it
robotically, as you are trying to do.

Rob