[HtmlUnit] [htmlunit:bugs] #1975 Some urls are missing while getting request from WebClient using HtmlUnit

Can you please provide a sample url to make this reproducible (and a list of requests you are missing).
You are sure real browsers doing all the requrests?


---

** [bugs:#1975] Some urls are missing while getting request from WebClient using HtmlUnit**

**Status:** open
**Group:** 2.31
**Created:** Tue Jul 24, 2018 07:50 AM UTC by Ibrahim S
**Last Updated:** Tue Jul 24, 2018 07:51 AM UTC
**Owner:** nobody


List of Issues:

Not Loading urls like facebook.com with type json, google analytics urls with type javascript, gif and xhr.

For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js


 WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
            webClient.getCookieManager().clearCookies();
            webClient.getCache().clear();

            webClient.setAjaxController(new NicelyResynchronizingAjaxController());
            webClient.setCssErrorHandler(new SilentCssErrorHandler());
            webClient.getOptions().setTimeout(120000);
            // to wait for AJAX

            webClient.waitForBackgroundJavaScript(60000);

            webClient.getOptions().setRedirectEnabled(true);
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.getOptions().setPrintContentOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getOptions().setUseInsecureSSL(true);
            webClient.getOptions().setDoNotTrackEnabled(false);

            new WebConnectionWrapper(webClient) {
                int conversionUrlCount = 0;

                @Override
                public WebResponse getResponse(WebRequest request) throws IOException {
                    System.out.println(request.getUrl());
                    WebResponse response = super.getResponse(request);
                    System.out.println(response.getStatusCode());
                    if (response.getStatusCode() < 400) {
                        resourceUrls.add(request.getUrl());
                    }
                    return response;
                }
            };

String url = "",
            HtmlPage page = webClient.getPage(url);

            // to forcibly load the link
            HtmlLink link = page.getFirstByXPath("//link");
            link.getWebResponse(true);

          // to forcibly load the image
            try {
                HtmlImage image = page.getFirstByXPath("//img");
                image.getImageReader();
            } catch (IOException e) {
                //don't need to crash at this point,
                //just let the user know that a wrong file has been passed.
            }





---

Sent from sourceforge.net because [email protected] is subscribed to https://sourceforge.net/p/htmlunit/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/htmlunit/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
HtmlUnit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/htmlunit-develop