Helping traversing a JavaScript table

--===============9131087604260433324==
Content-Type: multipart/alternative; boundary=000e0cd4c7649f9a0b04adf28259

--000e0cd4c7649f9a0b04adf28259
Content-Type: text/plain; charset=ISO-8859-1

Hello all,

I have a simple app I wrote with httpunit to traverse a website and extract
its tables so I can save them on a file.

The site is right here and it is an UI made by Oracle.
https://imiami.miamigov.com/OA_HTML/OA.jsp?OAFunc=PON_ABSTRACT_PAGE

I start by connecting to it and fetching all the links, among each the JS
links for the pagination of it.

WebConversation wc = new WebConversation();
        wc.setExceptionsThrownOnErrorStatus(false);

        try {
            HttpUnitOptions.setScriptingEnabled(true);
            HttpUnitOptions.setExceptionsThrownOnScriptError(false);
            HttpUnitOptions.setJavaScriptOptimizationLevel(9);
            WebRequest req = new GetMethodWebRequest(
                    "
https://imiami.miamigov.com/OA_HTML/OA.jsp?OAFunc=PON_ABSTRACT_PAGE");

            WebResponse resp;
            WebResponse respAux;

            resp = wc.getResponse(req);
            // Now I get the links
           WebLink[] links = resp.getLinks();

           // From the links, I click on each and try to extract a table
html structure:
           for (WebLink link : links) {
                        respAux = link.click();
                        HTMLElement[] tbls;
                try {
                tbls = respAux.getElementsByTagName("table");
                for (HTMLElement tbl : tbls) {
                    System.out.println(print(tbl.getNode()));
                    //result += print(tbl.getNode());
                }
                } catch (IllegalStateException e) {
                    System.out.println("There were no root tables.
"+e.getMessage());
                }
            }


     Now, here is the problem. I can get one or 2 pages at most and that
changes. Instead of following each link and move to the next page it does
not do that.

So I've tried to put the link.click call inside a recursive function so that
it would get the links again from each response and click on each. Still
does not work.

Any ideas?

Thanks.

--000e0cd4c7649f9a0b04adf28259
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello all,<br><br>I have a simple app I wrote with httpunit to traverse a w=
ebsite and extract its tables so I can save them on a file.<br><br>The site=
 is right here and it is an UI made by Oracle. <a href=3D"https://imiami.mi=
amigov.com/OA_HTML/OA.jsp?OAFunc=3DPON_ABSTRACT_PAGE">https://imiami.miamig=
ov.com/OA_HTML/OA.jsp?OAFunc=3DPON_ABSTRACT_PAGE</a><br>
<br>I start by connecting to it and fetching all the links, among each the =
JS links for the pagination of it.<br><br>WebConversation wc =3D new WebCon=
versation();<br>=A0=A0=A0 =A0=A0=A0 wc.setExceptionsThrownOnErrorStatus(fal=
se);<br><br>
=A0=A0=A0 =A0=A0=A0 try {<br>=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 HttpUnitOptions.=
setScriptingEnabled(true);<br>=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 HttpUnitOptions=
.setExceptionsThrownOnScriptError(false);<br>=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =
HttpUnitOptions.setJavaScriptOptimizationLevel(9);<br>=A0=A0=A0 =A0=A0=A0 =
=A0=A0=A0 WebRequest req =3D new GetMethodWebRequest(<br>
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 &quot;<a href=3D"https://=
imiami.miamigov.com/OA_HTML/OA.jsp?OAFunc=3DPON_ABSTRACT_PAGE">https://imia=
mi.miamigov.com/OA_HTML/OA.jsp?OAFunc=3DPON_ABSTRACT_PAGE</a>&quot;);<br>=
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 <br>=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 WebResponse=
 resp;<br>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 WebResponse respAux;<br><br>=A0=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0 resp =3D wc.getResponse(req);<br>=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0 // Now I get the links<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 We=
bLink[] links =3D resp.getLinks();<br><br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 //=
 From the links, I click on each and try to extract a table html structure:=
<br>
=A0=A0=A0=A0 =A0 =A0=A0=A0 for (WebLink link : links) {<br>=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 respAux =3D link.click();<br>=A0=
 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 HTMLElement[] tbls;<br>=A0=A0=
=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0try {<br>=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=
=A0=A0 =A0tbls =3D respAux.getElementsByTagName(&quot;table&quot;);<br>
=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0for (HTMLElement tbl : tbls) {<br>=
=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0System.out.println(print(=
tbl.getNode()));<br>=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0//res=
ult +=3D print(tbl.getNode());<br>=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0}=
<br>=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0} catch (IllegalStateException =
e) {<br>
=A0=A0=A0 =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0System.out.println(&quot;=
There were no root tables. &quot;+e.getMessage());<br>=A0=A0=A0 =A0=A0 =A0=
=A0=A0 =A0=A0=A0 =A0} =A0=A0=A0=A0=A0 <br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
 }<br><br>=A0 <br>=A0=A0=A0=A0 Now, here is the problem. I can get one or 2=
 pages at most and that changes. Instead of following each link and move to=
 the next page it does not do that.<br>
<br>So I&#39;ve tried to put the link.click call inside a recursive functio=
n so that it would get the links again from each response and click on each=
. Still does not work.<br><br>Any ideas?<br><br>Thanks.<br>

--000e0cd4c7649f9a0b04adf28259--


--===============9131087604260433324==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
--===============9131087604260433324==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Httpunit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/httpunit-develop

--===============9131087604260433324==--