[HtmlUnit] [htmlunit:bugs] #1796 HTMLUnit lookup returns request unsuccessful with JS off

This is a multi-part message in MIME format.
--===============7184090441543001386==
Content-Type: multipart/related;
 boundary="===============0812513193358323622=="

This is a multi-part message in MIME format.
--===============0812513193358323622==
Content-Type: multipart/alternative;
 boundary="===============1986820510314109346=="
MIME-Version: 1.0

--===============1986820510314109346==
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

- **status**: open --> closed
- **assigned_to**: RBRi
- **Comment**:

Sorry, this is no longer reproducible.



---

** [bugs:#1796] HTMLUnit lookup returns request unsuccessful with JS off**

**Status:** closed
**Group:** 2.22
**Created:** Sun Jun 05, 2016 07:00 PM UTC by Manny Gites
**Last Updated:** Mon Jun 06, 2016 09:25 PM UTC
**Owner:** RBRi


If I run the following code (see that JS enabled is true), the page usually loads.  However, if I turn off JS enabled, the lookup fails. Note that I decided to turn off JS in my code b/c I'm writing a crawler and there are lots of sites that I run into that cause HTMLUnit to stall.  I just need the page source for my crawler to index.  


	public static void main(String[] args) throws Exception {

		java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF); 
		System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");

		WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);
		webClient.getOptions().setJavaScriptEnabled(true); /*test fails if false*/
		webClient.getOptions().setRedirectEnabled(true);	
		webClient.getOptions().setCssEnabled(true);
		webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);  
		webClient.getOptions().setThrowExceptionOnScriptError(false);
		webClient.getOptions().setUseInsecureSSL(true);
		webClient.getOptions().setHistorySizeLimit(1);
		webClient.getOptions().setPopupBlockerEnabled(true);

		webClient.setAjaxController(new NicelyResynchronizingAjaxController());
		webClient.setJavaScriptTimeout(2147483647);
		webClient.setCssErrorHandler(new SilentCssErrorHandler());
		webClient.setRefreshHandler(new ThreadedRefreshHandler());

		webClient.getCookieManager().setCookiesEnabled(true);

		HtmlPage pg  = webClient.getPage("http://www.justbatreviews.com");
		System.out.println(pg.asText());
		webClient.close();
		webClient.getCookieManager().clearCookies();  
		
	
	}



---

Sent from sourceforge.net because [email protected] is subscribed to https://sourceforge.net/p/htmlunit/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/htmlunit/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.
--===============1986820510314109346==
MIME-Version: 1.0
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: 7bit

<div class="markdown_content"><ul>
<li><strong>status</strong>: open --&gt; closed</li>
<li><strong>assigned_to</strong>: RBRi</li>
<li><strong>Comment</strong>:</li>
</ul>
<p>Sorry, this is no longer reproducible.</p>
<hr/>
<p><strong> <a class="alink strikethrough" href="https://sourceforge.net/p/htmlunit/bugs/1796/">[bugs:#1796]</a> HTMLUnit lookup returns request unsuccessful with JS off</strong></p>
<p><strong>Status:</strong> closed<br/>
<strong>Group:</strong> 2.22<br/>
<strong>Created:</strong> Sun Jun 05, 2016 07:00 PM UTC by Manny Gites<br/>
<strong>Last Updated:</strong> Mon Jun 06, 2016 09:25 PM UTC<br/>
<strong>Owner:</strong> RBRi</p>
<p>If I run the following code (see that JS enabled is true), the page usually loads.  However, if I turn off JS enabled, the lookup fails. Note that I decided to turn off JS in my code b/c I'm writing a crawler and there are lots of sites that I run into that cause HTMLUnit to stall.  I just need the page source for my crawler to index.  </p>
<div class="codehilite"><pre><span></span><span class="nt">public</span> <span class="nt">static</span> <span class="nt">void</span> <span class="nt">main</span><span class="o">(</span><span class="nt">String</span><span class="cp">[]</span> <span class="nt">args</span><span class="o">)</span> <span class="nt">throws</span> <span class="nt">Exception</span> <span class="p">{</span>

    <span class="err">java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF)</span><span class="p">;</span> 
    <span class="err">System.setProperty("org.apache.commons.logging.Log",</span> <span class="err">"org.apache.commons.logging.impl.NoOpLog")</span><span class="p">;</span>

    <span class="err">WebClient</span> <span class="err">webClient</span> <span class="err">=</span> <span class="err">new</span> <span class="err">WebClient(BrowserVersion.BEST_SUPPORTED)</span><span class="p">;</span>
    <span class="err">webClient.getOptions().setJavaScriptEnabled(true)</span><span class="p">;</span> <span class="c">/*test fails if false*/</span>
    <span class="err">webClient.getOptions().setRedirectEnabled(true)</span><span class="p">;</span>    
    <span class="err">webClient.getOptions().setCssEnabled(true)</span><span class="p">;</span>
    <span class="err">webClient.getOptions().setThrowExceptionOnFailingStatusCode(false)</span><span class="p">;</span>  
    <span class="err">webClient.getOptions().setThrowExceptionOnScriptError(false)</span><span class="p">;</span>
    <span class="err">webClient.getOptions().setUseInsecureSSL(true)</span><span class="p">;</span>
    <span class="err">webClient.getOptions().setHistorySizeLimit(1)</span><span class="p">;</span>
    <span class="err">webClient.getOptions().setPopupBlockerEnabled(true)</span><span class="p">;</span>

    <span class="err">webClient.setAjaxController(new</span> <span class="err">NicelyResynchronizingAjaxController())</span><span class="p">;</span>
    <span class="err">webClient.setJavaScriptTimeout(2147483647)</span><span class="p">;</span>
    <span class="err">webClient.setCssErrorHandler(new</span> <span class="err">SilentCssErrorHandler())</span><span class="p">;</span>
    <span class="err">webClient.setRefreshHandler(new</span> <span class="err">ThreadedRefreshHandler())</span><span class="p">;</span>

    <span class="err">webClient.getCookieManager().setCookiesEnabled(true)</span><span class="p">;</span>

    <span class="err">HtmlPage</span> <span class="err">pg</span>  <span class="err">=</span> <span class="err">webClient.getPage("</span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">justbatreviews</span><span class="o">.</span><span class="n">com</span><span class="err">"</span><span class="p">);</span>
    <span class="err">System.out.println(pg.asText())</span><span class="p">;</span>
    <span class="err">webClient.close()</span><span class="p">;</span>
    <span class="err">webClient.getCookieManager().clearCookies()</span><span class="p">;</span>

<span class="p">}</span>
</pre></div>


<hr/>
<p>Sent from sourceforge.net because [email protected] is subscribed to <a href="https://sourceforge.net/p/htmlunit/bugs/">https://sourceforge.net/p/htmlunit/bugs/</a></p>
<p>To unsubscribe from further messages, a project admin can change settings at <a href="https://sourceforge.net/p/htmlunit/admin/bugs/options.">https://sourceforge.net/p/htmlunit/admin/bugs/options.</a>  Or, if this is a mailing list, you can unsubscribe from the mailing list.</p></div>
--===============1986820510314109346==--
--===============0812513193358323622==--


--===============7184090441543001386==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============7184090441543001386==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
HtmlUnit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/htmlunit-develop

--===============7184090441543001386==--