Re: AJAX/Google Pages

Ryan Perry wrote:
>
> On Jun 16, 2006, at 2:38 PM, Peter Stevens wrote:
>
>> Yes and no. AJAX is pretty heavily into Java script - you'll be 
>> spending a lot of time with Tamper Data to figure out what data is 
>> being sent back and forth. You can look at and reverse engineer the 
>> java script.
>>
>> Can it be done? Certainly. Is it easy? No. Is it practical? Well that 
>> depends on the complexity of the javascript environment and how much 
>> time & energy you have for the problem.
>>
>> We need a Javascript + DOM implementation on top of Mechanize!
>
> I strongly agree.  It's the future and we need to be there.  How would 
> one go about this?  I'd be happy to contribute.  Where do I start?
>
> Thanks!
Well, this may get me fried, but I do feel compelled to comment because 
my day to day job is to create code to do web scraping on some extremely 
complex web sites, many of which use a lot of AJAX and DHTML (Javascript 
and *shudder* VBS).

My first web scraper was written in Perl/Mechanize, and we managed to 
get past the DHTML portions by decoding the Javascript and returning 
responses that took into account the Javascript that *would* have run.  
It ran, but took over 3 months to write, and THAT site was well written, 
with nice ID's and no AJAX.  I still had to assume a few page locations 
and do blind posts because of complex Javascript that did page redirects 
to pages who's Javascript did MORE page re-directs.

After doing some research on upcoming projects scraping sites using AJAX 
and more complex Javascript I found a tool called Watir 
(http://openqa.org/watir/).  Watir is a library written in Ruby (which 
we were already starting to use) and gets around the whole 
Javascript/AJAX issues by automating use of Internet Explorer so all 
scripts actually run.  While having to run it under Windows and only 
having support for IE was the downside, the upside has been that our 
scrapers have been much easier to write, and the last one I did took 
about two weeks, and it does a lot more than our first one.

One thing that still gave us problems until recently was the issue of 
pop-up windows.  While Watir had a rather crude way of clicking past a 
pop-up window if you knew it was coming, modal dialogs were still hard 
to automate because they can have any valid HTML, but the IE "click" 
would block until the dialog closed, and there was no way to attach to 
the modal dialog and get access to the DOM if you did the click in 
another thread/process.  Finally someone put together an intricate 
method to attach to a modal dialog window by using the current IE's 
HWND, and then link the pointers together to get access to the modal 
dialog's DOM.

Now, I can automate a modal dialog window as easily as a normal browser 
window.  Here's the code to fire up IE to a page, click a button which 
brings up a modal dialog, attach to that dialog, fill a text box on the 
dialog, click on the dialog close box, and retrieve the entered value 
from the original window.  All in the following few lines of Ruby/Watir 
code:

    require 'watir'
    include Watir

    ie = IE.new
    ie.goto('http://SITE/modal_dialog_launcher.html')
    ie.button(:value, 'Launch Dialog').click_no_wait
    ie.modal_dialog.text_field(:name, 'modal_text').set('hello')
    ie.modal_dialog.button(:value, 'Close').click
    modal_text = ie.text_field(:name, 'modaloutput').value
    ie.close

That is code I just executed in Ruby's interactive shell (IRB) on one of 
the HTML files in the Watir unit test suite.

Now, to get that functionality you need to check out the latest 
developer versions using SVN as I just added the modal_dialog 
functionality recently, but it does work.  (I just put together the 
pieces I found in a number of places to get that functionality, so I 
can't take much credit, but I did submit it to the Watir project.)

There is a project called FireWatir which is aimed at using Firefox 
(under any O/S that Firefox runs under), but it's still lagging a bit 
behind, and performance is still very poor from what I've heard.  But 
there is hope.

For now, I'd recommend checking out Watir for your web automation 
projects, if you can get away with using IE under Windows.

David Schmidt
[email protected]