Stuck automating login to reuters.com and getting a page

[email protected] (Carl Wells) Sun, 16 Jan 2011 04:48:13 -0800 (PST)
Newsgroups perl.beginners.cgi
Message-ID <[email protected]>
Hi,=0A=0AI hope you don't mind my newbie question.  I'm new to web-programm=
ing (and indeed am somewhat rusty with programming in general).  I'm out of=
 work and trying to teach myself C++, PERL, SQL and other skills and in ord=
er to do this I've set myself a project.  As part of this project I need to=
 access data from this URL:=0A=0Ahttp://www.reuters.com/finance/stocks/inco=
meStatement/detail?perType=3DANN&symbol=3DBATS.L=0A=0Athe problem I'm havin=
g is that this redirects to the reuters.com login page.  I've tried to use =
both existing cookie files from internet explorer (I had to rename these be=
cause the name of the cookie involved my user name which incorporates a spa=
ce and an @ e.g. fred [email protected] and Perl didn't seem to like t=
hat/my syntax was wrong) and setting up perl to receive a new cookie from t=
he site.  Neither has worked for me.  I've spent the past 3 days trying to =
glue bits of code together from various googles and the cpan module descrip=
tions for LWP and Mechanize.  An example of code thats not working for me i=
s as below:=0A=0A#!/usr/local/bin/perl -w=0Ause strict;=0Ause Crypt::SSLeay=
;=0Ause LWP::UserAgent;=0Ause LWP::Simple;=0Ause HTTP::Request::Common qw(P=
OST);=0Ause HTTP::Cookies;=0A=0Amy $ua =3D LWP::UserAgent->new;=0Amy $cooki=
e_jar =3D HTTP::Cookies->new(file =3D> "lwpcookies2.txt",=0Aautosave =3D> 1=
);=0A$ua->cookie_jar( $cookie_jar);=0A$ua->agent('Mozilla/5.0');=0Amy $url =
=3D 'https://commerce.us.reuters.com/login/pages/login/login.do';=0Amy $req=
 =3D POST $url, ['login' =3D> 'Fredbumblebee', 'password' =3D> 'BzzZZZ!'];=
=0Amy $res =3D $ua->request($req);=0A$cookie_jar->extract_cookies($res);=0A=
=0Aif ($res->is_success) {=0A# print out result to look at headers=0Aprint =
$res->as_string;=0A=0A# access page with cookie secured after logged in=0Am=
y $req =3D HTTP::Request->new(GET =3D> 'http://www.reuters.com/finance/stoc=
ks/incomeStatement/detail?perType=3DANN&symbol=3DBATS.L');=0A$cookie_jar->a=
dd_cookie_header($req);=0A$res =3D $ua->request($req);=0A#print $res->as_st=
ring;=0A} else {=0Aprint "Failed: ", $res->status_line, "\n";=0A}=0A=0AThe =
cookie file only contains #LWP-Cookies-1.0.  I'm currently trying to use th=
e Live HTTP Headers addon in firefox to figure out what is being passed to =
and from the web server but I am a bit out of my depth :(.=0A=0AOnce I've d=
one this for BATS I'm planning to get a few more pages for other stocks so =
I'm guessing I'll want to create a session, not create a new cookie/log in =
again for each page request!  I also don't want to hammer their site, I gat=
her one can use a 'sleep' command, do you have any advice on this?=0A=0AI'v=
e managed to use HTML::tableextract to get tables I want from other reuters=
.com pages which didn't require the free logon but no joy here!  I started =
using C++/CURL/tidylib/tinyxml but moved to PERL as its so much easier to u=
se!  Once I have done this I'll want to call PERL from C++ so that I can pa=
ss my data into C++ objects; I've already looked into this and am finding i=
t tricky (running a simple perl script from C++ is fine but calling PERL wi=
th modules such as LWP has not worked for me yet; I've read the docs but no=
t managed to get the XS thing to run, Perl was saying it couldn't run dynam=
ic code in this way; does anyone know a good, easy to use Perl Wrapper for =
C++?? there are several but they all seem to be from 2003!! and not sure th=
ey will work)=0A=0AIf some kind soul would help me out or even suggest what=
 I might need to read to find my solution that would be very much appreciat=
ed!!=0A=0AThanks,=0A=0ACarl=0A=0A=0A=0A=0A