Re: Help with Mechanize

On Mon, 15 Jan 2007 12:17:22 -0600, William Jones wrote:
> I could use some help with Mechanize and Andy Lester recommended I post an
> email on the libwww mailing list.  I am trying to do what should be a
> simple scrape of the us patent and trademark website for bibliographic
> info that they post for all patents.  Unfortunately I keep getting
> re-routed to a page that says
>   
> Basically, I am wondering how the website could know that I am using
> mechanize and not internet explorer to enter the info into the fields and
> click "submit."

You could set the user_agent, but see below.

> Here is my perl code.  Thanks.
>  
>  
> #!/usr/local/bin/perl -w
> print "Content-type: text/html\n\n";
> use strict;
> use WWW::Mechanize;
> use Crypt::SSLeay;
> my $url = "https://ramps.uspto.gov/eram/"; my $maintenancepatent =
> "5771669";
> my $maintenanceapp = "08672157";
> my $outfile = "out.htm";
> my $mech = WWW::Mechanize->new( autocheck => 1); $mech->proxy(['https'],
> '');
> $mech->get($url);
> $mech->follow_link(text => "Pay or Look up Patent Maintenance Fees", n =>
> 1);
> $mech->form_name('mfInputForm');
> $mech->field(patentNum => "$maintenancepatent");
> $mech->field(applicationNum => "$maintenanceapp"); $mech->add_header(
> Referer => $url ); $mech->click_button (number => 2);
> open(OUTFILE, ">$outfile");
> my $output_page = $mech->content();
> print OUTFILE "$output_page";
> close(OUTFILE);
> print "done";

I would say one of two things: either (a) you've made more requests than
their terms of service permit and your IP is blacklisted, or (b) you've
got something unnecessary above, because when I try it with less code than
you've got, it works:

$ perl -MWWW::Mechanize -de '$m = WWW::Mechanize->new; 1'

Loading DB routines from perl5db.pl version 1.28
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):   $m = WWW::Mechanize->new; 1
  DB<1> n
main::(-e:1):   $m = WWW::Mechanize->new; 1
  DB<1> $m->get("https://ramps.uspto.gov/eram/")

  DB<2> $m->follow_link(text => "Pay or Look up Patent Maintenance Fees", n =>1) or die

  DB<3> $m->form_name('mfInputForm')

  DB<4> $m->field(patentNum => "5771669")

  DB<5> $m->field(applicationNum => "08672157")

  DB<6> $m->click_button(number=>2)

  DB<7> p $m->content(format=>'text')
 USPTO - Patent Bibliographic Data (Patent Number: 5771669)  Patent
 Bibliographic Data01/16/2007 09:46 AMPatent Number:5771669Application
 Number:08672157Issue Date:06/30/1998Filing Date:06/27/1996Title:METHOD
 AND APPARATUS FOR MOWING IRREGULAR TURF AREAS[...]

-- 
Peter Scott
http://www.perlmedic.com/
http://www.perldebugged.com/