Re: Stuck automating login to reuters.com and getting a page

On Sun, 2011-01-16 at 04:48 -0800, Carl Wells wrote:
> Hi,
>=20
> I hope you don't mind my newbie question.  I'm new to web-programming (an=
d indeed am somewhat rusty with programming in general).  I'm out of work a=
nd trying to teach myself C++, PERL, SQL and other skills and in order to d=
o this I've set myself a project.  As part of this project I need to access=
 data from this URL:
>=20
> http://www.reuters.com/finance/stocks/incomeStatement/detail?perType=3DAN=
N&symbol=3DBATS.L
>=20
> the problem I'm having is that this redirects to the reuters.com login pa=
ge.  I've tried to use both existing cookie files from internet explorer (I=
 had to rename these because the name of the cookie involved my user name w=
hich incorporates a space and an @ e.g. fred [email protected] and Per=
l didn't seem to like that/my syntax was wrong) and setting up perl to rece=
ive a new cookie from the site.  Neither has worked for me.  I've spent the=
 past 3 days trying to glue bits of code together from various googles and =
the cpan module descriptions for LWP and Mechanize.  An example of code tha=
ts not working for me is as below:
>=20
> #!/usr/local/bin/perl -w
> use strict;
> use Crypt::SSLeay;
> use LWP::UserAgent;
> use LWP::Simple;
> use HTTP::Request::Common qw(POST);
> use HTTP::Cookies;
>=20
> my $ua =3D LWP::UserAgent->new;
> my $cookie_jar =3D HTTP::Cookies->new(file =3D> "lwpcookies2.txt",
> autosave =3D> 1);
> $ua->cookie_jar( $cookie_jar);
> $ua->agent('Mozilla/5.0');
> my $url =3D 'https://commerce.us.reuters.com/login/pages/login/login.do';
> my $req =3D POST $url, ['login' =3D> 'Fredbumblebee', 'password' =3D> 'Bz=
zZZZ!'];
> my $res =3D $ua->request($req);
> $cookie_jar->extract_cookies($res);
>=20
> if ($res->is_success) {
> # print out result to look at headers
> print $res->as_string;
>=20
> # access page with cookie secured after logged in
> my $req =3D HTTP::Request->new(GET =3D> 'http://www.reuters.com/finance/s=
tocks/incomeStatement/detail?perType=3DANN&symbol=3DBATS.L');
> $cookie_jar->add_cookie_header($req);
> $res =3D $ua->request($req);
> #print $res->as_string;
> } else {
> print "Failed: ", $res->status_line, "\n";
> }
>=20
> The cookie file only contains #LWP-Cookies-1.0.  I'm currently trying to =
use the Live HTTP Headers addon in firefox to figure out what is being pass=
ed to and from the web server but I am a bit out of my depth :(.
>=20
> Once I've done this for BATS I'm planning to get a few more pages for oth=
er stocks so I'm guessing I'll want to create a session, not create a new c=
ookie/log in again for each page request!  I also don't want to hammer thei=
r site, I gather one can use a 'sleep' command, do you have any advice on t=
his?
>=20
> I've managed to use HTML::tableextract to get tables I want from other re=
uters.com pages which didn't require the free logon but no joy here!  I sta=
rted using C++/CURL/tidylib/tinyxml but moved to PERL as its so much easier=
 to use!  Once I have done this I'll want to call PERL from C++ so that I c=
an pass my data into C++ objects; I've already looked into this and am find=
ing it tricky (running a simple perl script from C++ is fine but calling PE=
RL with modules such as LWP has not worked for me yet; I've read the docs b=
ut not managed to get the XS thing to run, Perl was saying it couldn't run =
dynamic code in this way; does anyone know a good, easy to use Perl Wrapper=
 for C++?? there are several but they all seem to be from 2003!! and not su=
re they will work)
>=20
> If some kind soul would help me out or even suggest what I might need to =
read to find my solution that would be very much appreciated!!
>=20
> Thanks,
>=20
> Carl
>=20
>=20
>=20
>=20
>      =20
>=20
Hi Carl , if I read your post correctly , your trying to scrape a
website  of some data  using the  Perl LWP  methods ,  It is a common
task  for  Perl ,  May I suggest  that you   do some research on
Scrapping  and Perl , you will find that there are several approaches to
navigating  the target site  ,  your user agent should be able to
respond to login request  from the target site , and  proceed to the
next page the site presents  as well as make selections from drop down
box's and fill in text entry fields and  press the  submit buttons,
check with perl.com for some tutorials. WWW::Mechanize  may be the
module your looking for. or a combination of LWP::UserAgent and the
Perl Expect.pm may  be hacked together.

I have found  when I  last wrote  a scraping script it helped  to
manually walk through each and every step , look at the source of each
page and record  the  form  widgets  names and  what they  were  suppose
to  contain , then  reproduce the same  experience programing with the
script

hope this helps=20

Greg