Java script FAQ revised

Hi Andy,

[
     This text reflects the feedback from Mike Schilli about what's needed
     for mech to handle javascript. The question about getting the request
     which mech is going to send is still open:

     Is there a preferred way to get the request which mech is going to send?

     I was able to get it by following the code into the innards of
     HTTP::Request, but that seems like the kind of stuff a $mechanize user
     won't want to do.

]

Some time a ago, I sent you the text for the FAQ on Java script. There
is a section "So what do I do". I would like to suggest that you update
the text:

Corrected Original text:

     So what can I do?

     Since Javascript is completely visible to the client, it cannot be
     used to prevent a scraper from following links. But it can make life
     difficult, and until a DOM implementation is available to complement
     the perl Javascript interpreter ( javascript::SpiderMonkey ) or unless
     somebody develops a Mechanize clone to control Firefox, there will be no
     general solution. But if you want to scrape specific pages, then a
     solution is always possible.

     One typical use of Javascript is to perform argument checking before
     posting to the server. The URL you want is probably just buried in
     the Javascript function. Do a regular expression match on
     |$mech->content()| to find the link that you want and |$mech->get|
     it directly (this assumes that you know what your are looking for in
     advance).

     In more difficult cases, the Javascript is used for URL mangling to
     satisfy the needs of some middleware. In this case you need to
     figure out what the Javascript is doing (why are these URLs always
     really long?). There is probably some function with one or more
     arguments which calculates the new URL.

     Step one: using your favorite browser, get the before and after URLs and save them to
     files. Edit each file, converting the the argument separators ('?',
     '&' or ';') into newlines. Now it is easy to use diff or comm to
     find out what Javascript did to the URL.

     Step two: find the function call which created the URL. You will
     need to parse and interpret its argument list. Using the Javascript
     Debugger Extension for Firefox may help with the analysis. At this
     point, it is fairly trivial to write your own function which emulates
     the Javascript for the pages you want to process.

Please append to it:

     An Alternative Approach (this is also an answer to the question, "It
     works in Firefox, why not in $mech?" )

     Everything the web server knows about the client is present in the
     HTTP request. If two requests are identical, the results should be
     identical. So the real question is "What is different between the
     mech request and the Firefox request?"

     The Firefox extension "Tamper Data" is an effective tool for
     examining the headers of the requests to the server. Compare that
     with what LWP is sending. Once the two are identical, the action of
     the server should be the same as well.

     I say "should", because this is an oversimplification - some values
     are naturally unique, e.g. a SessionID, but if a SessionID is
     present, that is probably sufficient, even though the value will be
     different between the LWP request and the Firefox request. The
     server could use the session to store information which is
     troublesome, but that's not the first place to look (and highly
     unlikely to be relevant when you are requesting the login page of your
     site).

     Generally the problem is to be found in missing or incorrect
     POSTDATA arguments, Cookies, User-Agents, Accepts, etc. If you are
     using mech, then redirects and cookies should not be a problem, but
     are listed here for completeness. If you are missing headers,
     $mech->add_header can be used to add the headers that you need.


Cheers,

Peter



Cahoon, Forrest wrote:
> If you're specifically looking at Yahoo! Mail, there's at least one CPAN module for that:
> http://search.cpan.org/~johnsca/MailClientYahoo-1.0/lib/Mail/Client/Yahoo.pm
>
> If it's just something similar to Yahoo!, perhaps that code will give you some clues.
> (I haven't used that module myself, just happened to notice it's existence.)
>
> Forrest
> not speaking for merrill corporation 
>
>   
>> -----Original Message-----
>> From: Roy Lor [mailto:[email protected]] 
>> Sent: Tuesday, April 04, 2006 8:21 AM
>> To: [email protected]
>> Subject: WWW::Mechanize
>>
>> can u give me a code/script that records the information in a 
>> log-in  form with javascript..like that of mail.yahoo.com? i 
>> badly need  this..thanks
>>   
>> 		
>> ---------------------------------
>> On Yahoo!7
>>   Desperate Housewives: Sneak peeks, recaps and more. 
>>
>>     
>
>
>