[HtmlUnit] [htmlunit:bugs] #1975 Some urls are missing while getting request from WebClient using HtmlUnit

- Description has changed:

Diff:

~~~~

--- old
+++ new
@@ -7,25 +7,25 @@
 
 ~~~
  WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
-            webClient.getCookieManager().clearCookies();
-            webClient.getCache().clear();
+webClient.getCookieManager().clearCookies();
+webClient.getCache().clear();
 
-            webClient.setAjaxController(new NicelyResynchronizingAjaxController());
-            webClient.setCssErrorHandler(new SilentCssErrorHandler());
-            webClient.getOptions().setTimeout(120000);
-            // to wait for AJAX
+webClient.setAjaxController(new NicelyResynchronizingAjaxController());
+webClient.setCssErrorHandler(new SilentCssErrorHandler());
+webClient.getOptions().setTimeout(120000);
+// to wait for AJAX
 
-            webClient.waitForBackgroundJavaScript(60000);
+webClient.waitForBackgroundJavaScript(60000);
 
-            webClient.getOptions().setRedirectEnabled(true);
-            webClient.getOptions().setJavaScriptEnabled(true);
-            webClient.getOptions().setPrintContentOnFailingStatusCode(false);
+webClient.getOptions().setRedirectEnabled(true);
+webClient.getOptions().setJavaScriptEnabled(true);
+webClient.getOptions().setPrintContentOnFailingStatusCode(false);
             webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
-            webClient.getOptions().setThrowExceptionOnScriptError(false);
-            webClient.getOptions().setUseInsecureSSL(true);
-            webClient.getOptions().setDoNotTrackEnabled(false);
+webClient.getOptions().setThrowExceptionOnScriptError(false);
+webClient.getOptions().setUseInsecureSSL(true);
+webClient.getOptions().setDoNotTrackEnabled(false);
 
-            new WebConnectionWrapper(webClient) {
+new WebConnectionWrapper(webClient) {
                 int conversionUrlCount = 0;
 
                 @Override
@@ -38,21 +38,21 @@
                     }
                     return response;
                 }
-            };
+};
 
 String url = "",
-            HtmlPage page = webClient.getPage(url);
+HtmlPage page = webClient.getPage(url);
 
-            // to forcibly load the link
-            HtmlLink link = page.getFirstByXPath("//link");
-            link.getWebResponse(true);
+// to forcibly load the link
+HtmlLink link = page.getFirstByXPath("//link");
+link.getWebResponse(true);
 
-          // to forcibly load the image
-            try {
-                HtmlImage image = page.getFirstByXPath("//img");
-                image.getImageReader();
-            } catch (IOException e) {
-                //don't need to crash at this point,
-                //just let the user know that a wrong file has been passed.
-            }
+// to forcibly load the image
+try {
+        HtmlImage image = page.getFirstByXPath("//img");
+        image.getImageReader();
+} catch (IOException e) {
+        //don't need to crash at this point,
+        //just let the user know that a wrong file has been passed.
+}
 ~~~

~~~~




---

** [bugs:#1975] Some urls are missing while getting request from WebClient using HtmlUnit**

**Status:** open
**Group:** 2.31
**Created:** Tue Jul 24, 2018 07:50 AM UTC by Ibrahim S
**Last Updated:** Tue Jul 24, 2018 04:24 PM UTC
**Owner:** nobody


List of Issues:

Not Loading urls like facebook.com with type json, google analytics urls with type javascript, gif and xhr.

For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js


~~~
 WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
webClient.getCookieManager().clearCookies();
webClient.getCache().clear();

webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.getOptions().setTimeout(120000);
// to wait for AJAX

webClient.waitForBackgroundJavaScript(60000);

webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setDoNotTrackEnabled(false);

new WebConnectionWrapper(webClient) {
                int conversionUrlCount = 0;

                @Override
                public WebResponse getResponse(WebRequest request) throws IOException {
                    System.out.println(request.getUrl());
                    WebResponse response = super.getResponse(request);
                    System.out.println(response.getStatusCode());
                    if (response.getStatusCode() < 400) {
                        resourceUrls.add(request.getUrl());
                    }
                    return response;
                }
};

String url = "",
HtmlPage page = webClient.getPage(url);

// to forcibly load the link
HtmlLink link = page.getFirstByXPath("//link");
link.getWebResponse(true);

// to forcibly load the image
try {
        HtmlImage image = page.getFirstByXPath("//img");
        image.getImageReader();
} catch (IOException e) {
        //don't need to crash at this point,
        //just let the user know that a wrong file has been passed.
}
~~~



---

Sent from sourceforge.net because [email protected] is subscribed to https://sourceforge.net/p/htmlunit/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/htmlunit/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
HtmlUnit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/htmlunit-develop