[HtmlUnit] [htmlunit:bugs] #1975 Some urls are missing while getting request from WebClient using HtmlUnit

Sorry, for asking again. I need a way to reproduce this, any help to understand the problem is welcome.
* maybe you are able to provide the url your are calling with your initial call
* maybe you can provide some html pieces showing, how this missing stuff is called by the real browsers - is there a script tag that refers them or does some javascript do the call

And finally please place your
`webClient.waitForBackgroundJavaScript(60000);`
call after the  get class (this is not an option).


---

** [bugs:#1975] Some urls are missing while getting request from WebClient using HtmlUnit**

**Status:** open
**Group:** 2.31
**Created:** Tue Jul 24, 2018 07:50 AM UTC by Ibrahim S
**Last Updated:** Tue Jul 24, 2018 04:26 PM UTC
**Owner:** nobody


List of Issues:

Not Loading urls like facebook.com with type json, google analytics urls with type javascript, gif and xhr.

For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js


~~~
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
webClient.getCookieManager().clearCookies();
webClient.getCache().clear();

webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.getOptions().setTimeout(120000);
// to wait for AJAX

webClient.waitForBackgroundJavaScript(60000);

webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setDoNotTrackEnabled(false);

new WebConnectionWrapper(webClient) {
                int conversionUrlCount = 0;

                @Override
                public WebResponse getResponse(WebRequest request) throws IOException {
                    System.out.println(request.getUrl());
                    WebResponse response = super.getResponse(request);
                    System.out.println(response.getStatusCode());
                    if (response.getStatusCode() < 400) {
                        resourceUrls.add(request.getUrl());
                    }
                    return response;
                }
};

String url = "",
HtmlPage page = webClient.getPage(url);

// to forcibly load the link
HtmlLink link = page.getFirstByXPath("//link");
link.getWebResponse(true);

// to forcibly load the image
try {
        HtmlImage image = page.getFirstByXPath("//img");
        image.getImageReader();
} catch (IOException e) {
        //don't need to crash at this point,
        //just let the user know that a wrong file has been passed.
}
~~~



---

Sent from sourceforge.net because [email protected] is subscribed to https://sourceforge.net/p/htmlunit/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/htmlunit/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
HtmlUnit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/htmlunit-develop