Re: How do I extract JS and CSS links from a web page?

This code may be relevant for you (it's a work in progress).
I am checking for 404s, using HEAD calls for efficiency.

By the way, I would love if this kind of functionality
was built into HttpUnit.



  WebResponse resp = ...;

  try {

    // These are the CSS "<link ...>" tags, not "<a ...> tags.

    // "<link ...>" is also used for favicon, etc.  We check them all.

    HTMLElement[] links = resp.getElementsByTagName("link");

    if (links != null && links.length > 0) {

      // Create new WC with no listener or special settings

      WebConversation wc = new WebConversation();

      Set<String> hits = new HashSet<String>((links.length * 4) / 3 + 1);

      for (HTMLElement link : links) {

        String src = link.getAttribute("href");

        if (hits.contains(src))

          continue;

        hits.add(src);

        WebRequest linkRequest = new HeadMethodWebRequest(resp.getURL(), src);

        WebResponse linkResp = wc.getResponse(linkRequest);

        int linkCode = linkResp.getResponseCode();

        if (linkCode >= 400) {

          throw new RuntimeException("Error: " + linkCode + " getting " + src);

        }

      }

    }

    WebImage[] images = resp.getImages();

    if (images != null && images.length > 0) {

      // Create new WC with no listener or special settings

      WebConversation wc = new WebConversation();

      Set<String> hits = new HashSet<String>((images.length * 4) / 3 + 1);

      for (WebImage image : images) {

        String src = image.getSource();



        if (hits.contains(src))

          continue;

        hits.add(src);

        WebRequest imgRequest = new HeadMethodWebRequest(resp.getURL(), src);

        WebResponse imgResp = wc.getResponse(imgRequest);

        int imgCode = imgResp.getResponseCode();

        if (imgCode >= 400) {

          throw new RuntimeException("Error: " + imgCode + " getting " + src);

        }

      }

    }

  } catch (SAXException e) {

    log.error(e);

  }

- Dan

From: Wolfgang Fahl [mailto:[email protected]]
Sent: Thursday, August 20, 2009 3:58 AM
To: Stephen Cresswell; Discussion of use and development of HttpUnit
Subject: Re: [Httpunit-develop] How do I extract JS and CSS links from a web page?

Hi Steve,

too bad it took me so long to reply.

For the style sheet part:
WebPageTest. testGetExternalStylesheet ()
is the test case to start with and
HTMLPage.getExternalStyleSheet
would be the starting points.

For the script part getIncludedScript() is the one to look at.

To work on such a Change Request I'd need the modified test case + a patch according to the FAQ I recently sent.

Yours
  Wolfgang
>
> Hi Wolfgang,
>
> What we did was to convert the HTML to a XML Document then navigate the
> XML, extracting all the CSS & Links etc.
> If I remember it wasn't quite straight forward to get the XML Document
> though - depending on the version of httpunit we used it threw
> exceptions, but I think these have been discussed in other posts to the
> user list.
>
> Thanks for getting back to me,
>
> Steve
>
>
> 2009/2/12 Wolfgang Fahl <[email protected]>
>     Dear Stephen,
>
> thank you for your mail. I'm sorry for not getting back to you any
> sooner. Orignally i had intended to look into the details of this and
> give a response to you. Since I wasn't able to do this i'm wondering how
> you made progress in the meantime, what responses you got (if any) and
> whether others would like to get this issue solved.
>
> Yours
>  Wolfgang
> >
> >Hi,
> >
> > I'm hoping to use HttpUnit to check our application sets response
> > headers appropriately (especially for caching). To do this I expect to
> >
> > 1. Issue a GET request for http://myapp.com/index.html
> > 2. Interrogate the WebResponse to extract links to images, css and
> >javascript
> > 3. Issue GET requests for each image, css + js file
> > 4. Re-issue the GET request and assert the response code is 304, and
> > approprate expires / last modified headers are set.
> >
> > However I'm having problems obtaining the css and js urls.
> WebResponse.
> >getExternalStyleSheet() only returns the first declaired stylesheet,
> and
> > I haven't jet found a way to extract links to scripts.
> >
> > Any help appreciated,
> >
> >Thanks,
> >
> >Steve
>
>
>
>
>
> BITPlan - smart solutions
> Pater-Delp-Str. 1, D-47877 Willich Schiefbahn
> Tel. +49 2154 811-480, Fax +49 2154 811-481
> Web: http://www.bitplan.de
> bitplan GmbH, Willich - HRB 6820 Krefeld, VAT-ID: 10258040548, Geschäftsführer: Wolfgang
> Fahl
>
>
>
> ------------------------------------------------------------------------
> ------
>
> _______________________________________________
> Httpunit-develop mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/httpunit-develop
>
>
>



BITPlan - smart solutions
Pater-Delp-Str. 1, D-47877 Willich Schiefbahn
Tel. +49 2154 811-480, Fax +49 2154 811-481
Web: http://www.bitplan.de
bitplan GmbH, Willich - HRB 6820 Krefeld, VAT-ID: 10258040548, Geschäftsführer: Wolfgang Fahl

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

_______________________________________________
Httpunit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/httpunit-develop