Re: Sudden problems with LWP::UserAgent

Keary Suska <[email protected]> Wed, 27 Oct 2010 10:38:37 -0600

Newsgroups	gmane.comp.lang.perl.modules.lwp
Message-ID	<[email protected]>

On Oct 27, 2010, at 1:15 AM, Uday Bhaskar wrote:

> I guess this is more a question of how to get around a website
> blocking LWP::UserAgent. I have been using this code for a while now
> 
> <snip>
> This code worked till yesterday. However today I am hitting the 403 error.
> "
> error in getting index 403 Forbidden
> try again...
> "
> 
> I used the same UserAgent as that of Mozilla which successfully
> retrieves the page. I tried it from different IP address to ensure
> that my IP is not blocked(I run it once a day and am not running a
> bot) Looking for some help in getting this going again.

Some sites do browser fingerprinting. Most of the time you can fool the site by sending the same headers that the browser sends. You can inspect headers using a plug=in like HttpFox, which is my favorite.

HTH,

Keary Suska