Httpunit Support for document.write
"Catalin Patulea" <[email protected]> Sat, 26 Apr 2008 19:01:21 -0400
| Newsgroups | gmane.comp.web.httpunit.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi,
I am trying to use Httpunit to scrape information off a local bus
operator's web site. I realize that httpunit wasn't really meant for
that, but it's also the best tool I've found for the job so far. I
have tried asking nicely for the information, but they were
uncooperative. With that in mind, here is the problem I am having:
The web site uses <script> blocks and document.write's to generate
<option> tags within a <select>. Httpunit seems to only apply content
generated from document.write to replace the entire page, and only
once document.close has been called. For example:
<html><head><title>httpunit-test</title></head><body>
<a href="http://www.google.ca/">Link 1</a>
<script language="javascript">
<!--
document.writeln("<a href='http://www.google.ca'>Link 2<" + "/a>");
document.close();
-->
</script>
<a href="http://www.google.ca/">Link 3</a>
</body></html>
In Httpunit, the page above generates only one link, Link 2 (as
inspected via WebResponse.getLinks). If I remove document.close, I see
only Link 1 and Link 3.
The expected behaviour is to see all three links, in the correct order
(this is what Firefox does, for instance).
Since my only objective is scraping the site, I can always just put in
a small hack in my code. But I would like to at least try to do it the
right way first.
Thanks,
Catalin
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone