Re: Sudden problems with LWP::UserAgent

Hi Keary,

I followed your suggestion and plugged in all the headers one by one
and figured that

'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',

was expected as a request header (I did not break it down further.
Just plugged in whatever firefox was using).

Thanks a lot for your help.

Thanks
Uday

On Wed, Oct 27, 2010 at 10:08 PM, Keary Suska <[email protected]> wrote:
> On Oct 27, 2010, at 1:15 AM, Uday Bhaskar wrote:
>
>> I guess this is more a question of how to get around a website
>> blocking LWP::UserAgent. I have been using this code for a while now
>>
>> <snip>
>> This code worked till yesterday. However today I am hitting the 403 error.
>> "
>> error in getting index 403 Forbidden
>> try again...
>> "
>>
>> I used the same UserAgent as that of Mozilla which successfully
>> retrieves the page. I tried it from different IP address to ensure
>> that my IP is not blocked(I run it once a day and am not running a
>> bot) Looking for some help in getting this going again.
>
> Some sites do browser fingerprinting. Most of the time you can fool the site by sending the same headers that the browser sends. You can inspect headers using a plug=in like HttpFox, which is my favorite.
>
> HTH,
>
> Keary Suska
>
>