Stuck automating login to reuters.com and getting a page
[email protected] (Carl Wells) Sun, 16 Jan 2011 04:48:13 -0800 (PST)
| Newsgroups | perl.beginners.cgi |
|---|---|
| Message-ID | <[email protected]> |
Hi,=0A=0AI hope you don't mind my newbie question. I'm new to web-programm= ing (and indeed am somewhat rusty with programming in general). I'm out of= work and trying to teach myself C++, PERL, SQL and other skills and in ord= er to do this I've set myself a project. As part of this project I need to= access data from this URL:=0A=0Ahttp://www.reuters.com/finance/stocks/inco= meStatement/detail?perType=3DANN&symbol=3DBATS.L=0A=0Athe problem I'm havin= g is that this redirects to the reuters.com login page. I've tried to use = both existing cookie files from internet explorer (I had to rename these be= cause the name of the cookie involved my user name which incorporates a spa= ce and an @ e.g. fred [email protected] and Perl didn't seem to like t= hat/my syntax was wrong) and setting up perl to receive a new cookie from t= he site. Neither has worked for me. I've spent the past 3 days trying to = glue bits of code together from various googles and the cpan module descrip= tions for LWP and Mechanize. An example of code thats not working for me i= s as below:=0A=0A#!/usr/local/bin/perl -w=0Ause strict;=0Ause Crypt::SSLeay= ;=0Ause LWP::UserAgent;=0Ause LWP::Simple;=0Ause HTTP::Request::Common qw(P= OST);=0Ause HTTP::Cookies;=0A=0Amy $ua =3D LWP::UserAgent->new;=0Amy $cooki= e_jar =3D HTTP::Cookies->new(file =3D> "lwpcookies2.txt",=0Aautosave =3D> 1= );=0A$ua->cookie_jar( $cookie_jar);=0A$ua->agent('Mozilla/5.0');=0Amy $url = =3D 'https://commerce.us.reuters.com/login/pages/login/login.do';=0Amy $req= =3D POST $url, ['login' =3D> 'Fredbumblebee', 'password' =3D> 'BzzZZZ!'];= =0Amy $res =3D $ua->request($req);=0A$cookie_jar->extract_cookies($res);=0A= =0Aif ($res->is_success) {=0A# print out result to look at headers=0Aprint = $res->as_string;=0A=0A# access page with cookie secured after logged in=0Am= y $req =3D HTTP::Request->new(GET =3D> 'http://www.reuters.com/finance/stoc= ks/incomeStatement/detail?perType=3DANN&symbol=3DBATS.L');=0A$cookie_jar->a= dd_cookie_header($req);=0A$res =3D $ua->request($req);=0A#print $res->as_st= ring;=0A} else {=0Aprint "Failed: ", $res->status_line, "\n";=0A}=0A=0AThe = cookie file only contains #LWP-Cookies-1.0. I'm currently trying to use th= e Live HTTP Headers addon in firefox to figure out what is being passed to = and from the web server but I am a bit out of my depth :(.=0A=0AOnce I've d= one this for BATS I'm planning to get a few more pages for other stocks so = I'm guessing I'll want to create a session, not create a new cookie/log in = again for each page request! I also don't want to hammer their site, I gat= her one can use a 'sleep' command, do you have any advice on this?=0A=0AI'v= e managed to use HTML::tableextract to get tables I want from other reuters= .com pages which didn't require the free logon but no joy here! I started = using C++/CURL/tidylib/tinyxml but moved to PERL as its so much easier to u= se! Once I have done this I'll want to call PERL from C++ so that I can pa= ss my data into C++ objects; I've already looked into this and am finding i= t tricky (running a simple perl script from C++ is fine but calling PERL wi= th modules such as LWP has not worked for me yet; I've read the docs but no= t managed to get the XS thing to run, Perl was saying it couldn't run dynam= ic code in this way; does anyone know a good, easy to use Perl Wrapper for = C++?? there are several but they all seem to be from 2003!! and not sure th= ey will work)=0A=0AIf some kind soul would help me out or even suggest what= I might need to read to find my solution that would be very much appreciat= ed!!=0A=0AThanks,=0A=0ACarl=0A=0A=0A=0A=0A