Re: Two possible refinements

"James Courtney" <[email protected]> Thu, 18 Sep 2008 09:02:48 -0700
Newsgroups gmane.comp.web.httpunit.devel
Message-ID <026a01c919a8$0034f300$009ed900$@com>
Just wondering if anyone has had a chance to look at my proposed changes.

Thanks!

Jamey


-----Original Message-----
From: James Courtney [mailto:[email protected]] 
Sent: Monday, August 18, 2008 10:59 PM
To: 'Discussion of use and development of HttpUnit'
Subject: RE: [Httpunit-develop] Two possible refinements

Here's the patch file.  I'd love some feedback on the changes I made for
redirection loop detection/handling.

1) I added two ClientProperties properties to control this.  The first
(resolveRedirectLoops) tells the code to either utilize the standard URL
comparison (with name resolution) for detecting loops or to ignore this and
simply string compare the domain names.  The second property specifies a
maximum number of redirects in the chain defaulting to 10 and settable to -1
(or any negative value) to allow infinite.

2) I also changed the exception handling slightly by introducing the
RedirectionException (which RecursiveRedirectionException now extends) for
throwing if the number of redirects is exceeded or if the URL created for
the redirection is invalid.

The update for taking into account strict (or lax) domain matching for
cookies was much simpler and I was able to easily include some unit tests
for this.  The code for redirection loop handling I'm not sure how to create
unit tests for BUT the current unit tests all run save the two that were
failing even with fresh code checked out and the unit tests for my
application which depends on HTTPUnit and which originally triggered the
issues I saw now runs fine.  There I was able to test enabling and disabling
the ClientProperties parameters mentioned and these behaved as expected.

The risky changes I see for current users are:

1) The exception handling change which still throws just a subclass of
RuntimeException but now RedirectionException in addition to
RecursiveRedirectionException.  You might make the argument that the
RecursiveRedirectionException subclass is excessive and could simply be
reduced to a RedirectionException with thrown with the correct message.
This might break some existing code BUT it's explicit and forces the
developer, if they ever cared about handling those exceptions, to note the
change and make a small adaptation.

2) Defaulting to simple string comparison for redirection loop detection vs.
the previous URL.equals which attempted to resolve the domain names and
compare IPs.  While the default of not doing the resolution comparison seems
more sensible to me it is a change in behavior and maybe the existing
behavior should continue as the default.

3) The cap on redirections by quantity currently defaulted to 10 in my code.
This seems reasonable to me but who knows what silly stuff is out there and
may break with this new restriction.  This could be defaulted to -1 to agree
with current behavior.

Thanks for your consideration and assistance.

Jamey



-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of
Wolfgang Fahl - BITPlan
Sent: Monday, August 18, 2008 1:02 AM
To: Discussion of use and development of HttpUnit
Subject: Re: [Httpunit-develop] Two possible refinements

James wrote:
> Thank you Russell.  I'll take a look at making the specific changes 
> and
including unit test code for submission in the next day or two.  I trust you
want patches generated by something like the following:
> svn diff File.java >> patchfile.txt
The area you have commented on has been part a few of the patch-proposals of
the 1.7 release. Some of the proposals looked conflicting to me so not
everything was implemented as the suggestions in the patch-proposals asked
for. It's good that you are going to clarify the situation. 

You might want to not hesitate to contact me for the patch submission. I
have volunteered to do commit work like this since last year.

Wolfgang
BITPlan - smart solutions
Pater-Delp-Str. 1, D-47877 Willich-Schiefbahn Tel. +49 1805 - BITPLAN / +49
1805 248 752, Fax +49 2154
811-481
Web: http://www.bitplan.de
bitplan GmbH, Willich - HRB 6820 Krefeld, VAT-ID: 
10258040584, Geschäftsführer: Wolfgang Fahl Mit BITPlan effizient zum
Erfolg: http://www.bitplan.de


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Httpunit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/httpunit-develop

No virus found in this incoming message.
Checked by AVG - http://www.avg.com 
Version: 8.0.138 / Virus Database: 270.6.5/1618 - Release Date: 8/18/2008
6:51 AM

No virus found in this incoming message.
Checked by AVG - http://www.avg.com 
Version: 8.0.169 / Virus Database: 270.6.19/1664 - Release Date: 9/17/2008
9:33 AM

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Httpunit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/httpunit-develop
patchfile.txt (text/plain, 16.8 KB)
Index: test/com/meterware/httpunit/cookies/CookieTest.java
===================================================================
--- test/com/meterware/httpunit/cookies/CookieTest.java	(revision 989)
+++ test/com/meterware/httpunit/cookies/CookieTest.java	(working copy)
@@ -88,6 +88,9 @@
 
         checkMatching( 5, true, new URL( "http://www.meterware.com/servlets/sample" ), "www.meterware.com", "/servlets" );
         checkMatching( 6, false, new URL( "http://www.meterware.com/servlets/sample" ), "www.meterware.com", "/servlets/sample/data" );
+
+        checkMatching( 7, true, new URL( "http://www.meterware.com/servlets/sample" ), "www.MeterWare.com", "/servlets/sample" );
+        checkMatching( 8, true, new URL( "http://WWW.meterware.com/servlets/sample" ), ".MeterWare.com", "/servlets/sample" );
     }
 
 
@@ -284,6 +287,7 @@
         checkAcceptance( 1, true, "www.some.meterware.com/servlets/special", ".meterware.com", null );
         checkAcceptance( 2, false, "www.meterware.com/servlets/special", ".meterware.com", "/servlets/ordinary" );
         checkAcceptance( 3, true, "www.meterware.com/servlets/special", "www.meterware.com", null );
+        checkAcceptance( 1, true, "www.some.meterware.com/servlets/special", "meterware.com", null );
 
         CookieProperties.setPathMatchingStrict( false );
         checkAcceptance( 11, true, "www.meterware.com/servlets/special", ".meterware.com", "/servlets/ordinary" );
Index: src/com/meterware/httpunit/cookies/Cookie.java
===================================================================
--- src/com/meterware/httpunit/cookies/Cookie.java	(revision 989)
+++ src/com/meterware/httpunit/cookies/Cookie.java	(working copy)
@@ -247,8 +247,9 @@
      * @return true if there is a fit
      */
     private static boolean acceptHost( String hostPattern, String hostName ) {
-        return hostPattern.equalsIgnoreCase( hostName ) ||
-               (hostPattern.startsWith( "." ) && hostName.endsWith( hostPattern ));
+        return !CookieProperties.isDomainMatchingStrict() ||
+        	hostPattern.equalsIgnoreCase( hostName ) ||
+            (hostPattern.startsWith( "." ) && hostName.toLowerCase().endsWith(hostPattern.toLowerCase()));
     }
 
 
Index: src/com/meterware/httpunit/ClientProperties.java
===================================================================
--- src/com/meterware/httpunit/ClientProperties.java	(revision 989)
+++ src/com/meterware/httpunit/ClientProperties.java	(working copy)
@@ -180,6 +180,48 @@
 
 
     /**
+     * Returns the maximum redirects to be followed in a chain before erring out.
+     * By default, this is 10.
+     **/
+    public int getMaxRedirects() {
+        return _maxRedirects;
+    }
+
+
+    /**
+     * Sets the maximum number of redirects the client should follow in a chain
+     * before erring out.  -1 tells the client to follow an infinite number of
+     * redirects in the chain, 10 is the default.  0 has the effect of failing on
+     * any and all redirects.
+     **/
+    public void setMaxRedirects( int maxRedirects ) {
+        _maxRedirects = maxRedirects;
+    }
+
+
+    /**
+     * Returns true if the client should resolve redirect domain names in a
+     * redirect chain and compare by IP address rather than simply domain name
+     * during redirect loop detection.  If the domain names cannot be resolved
+     * the fall-back is a string comparison of domain names.
+     **/
+    public boolean resolveRedirectLoops() {
+        return _resolveRedirectLoops;
+    }
+
+
+    /**
+     * Determines whether the client should resolve redirect domain names
+     * when comparing for the detection of redirect loops.  If true domain
+     * names are resolved and the IPs compared.  If the resolution is not
+     * successful the domain names are string compared as a fall-back.
+     **/
+    public void setResolveRedirectLoops( boolean resolveRedirectLoops ) {
+    	_resolveRedirectLoops = resolveRedirectLoops;
+    }
+
+
+    /**
      * Returns true if the client should automatically follow page refresh requests.
      * By default, this is false, so that programs can verify the redirect page presented
      * to users before the browser switches to the new page.
@@ -269,20 +311,22 @@
     }
 
 
-    private String _applicationCodeName = "httpunit";
-    private String _applicationName     = "HttpUnit";
-    private String _applicationVersion  = "1.5";
+    private String _applicationCodeName   = "httpunit";
+    private String _applicationName       = "HttpUnit";
+    private String _applicationVersion    = "1.5";
     private String _userAgent;
-    private String _platform            = "Java";
-    private String _overrideContextType = null;
-    private int    _availWidth          = 800;
-    private int    _availHeight         = 600;
+    private String _platform              = "Java";
+    private String _overrideContextType   = null;
+    private int    _availWidth            = 800;
+    private int    _availHeight           = 600;
 
-    private boolean _iframeSupported = true;
-    private boolean _acceptCookies = true;
-    private boolean _acceptGzip    = true;
-    private boolean _autoRedirect  = true;
-    private boolean _autoRefresh   = false;
+    private boolean _iframeSupported      = true;
+    private boolean _acceptCookies        = true;
+    private boolean _acceptGzip           = true;
+    private boolean _autoRedirect         = true;
+    private int     _maxRedirects         = 10;
+    private boolean _resolveRedirectLoops = false;
+    private boolean _autoRefresh          = false;
 
     private DNSListener _dnsListener;
     private boolean _sendReferer;
@@ -304,18 +348,20 @@
      * @param source - the ClientProperties to copy from
      */
     private ClientProperties( ClientProperties source ) {
-        _applicationCodeName = source._applicationCodeName;
-        _applicationName     = source._applicationName;
-        _applicationVersion  = source._applicationVersion;
-        _userAgent           = source._userAgent;
-        _platform            = source._platform;
-        _overrideContextType = source._overrideContextType;
-        _iframeSupported     = source._iframeSupported;
-        _acceptCookies       = source._acceptCookies;
-        _acceptGzip          = source._acceptGzip;
-        _autoRedirect        = source._autoRedirect;
-        _autoRefresh         = source._autoRefresh;
-        _sendReferer         = source._sendReferer;
+        _applicationCodeName  = source._applicationCodeName;
+        _applicationName      = source._applicationName;
+        _applicationVersion   = source._applicationVersion;
+        _userAgent            = source._userAgent;
+        _platform             = source._platform;
+        _overrideContextType  = source._overrideContextType;
+        _iframeSupported      = source._iframeSupported;
+        _acceptCookies        = source._acceptCookies;
+        _acceptGzip           = source._acceptGzip;
+        _autoRedirect         = source._autoRedirect;
+        _maxRedirects         = source._maxRedirects;
+        _resolveRedirectLoops = source._resolveRedirectLoops;
+        _autoRefresh          = source._autoRefresh;
+        _sendReferer          = source._sendReferer;
     }
 
 
Index: src/com/meterware/httpunit/WebWindow.java
===================================================================
--- src/com/meterware/httpunit/WebWindow.java	(revision 989)
+++ src/com/meterware/httpunit/WebWindow.java	(working copy)
@@ -23,9 +23,9 @@
 import java.net.HttpURLConnection;
 import java.net.MalformedURLException;
 import java.net.URL;
-import java.util.HashSet;
+import java.util.HashMap;
 import java.util.List;
-import java.util.Set;
+import java.util.Map;
 
 import org.xml.sax.SAXException;
 import com.meterware.httpunit.scripting.ScriptingHandler;
@@ -60,7 +60,7 @@
      * of a single client-initiated request
      * @since patch [ 1155415 ] Handle redirect instructions which can lead to a loop
      */
-    private final Set _redirects;
+    private final Map _redirects;
     
     /** True if seen initial request 
      * @since patch [ 1155415 ] Handle redirect instructions which can lead to a loop
@@ -300,7 +300,7 @@
         _client = client;
         _frameContents = new FrameHolder( this );
         _name = NO_NAME + _client.getOpenWindows().length;
-        _redirects = new HashSet();
+        _redirects = new HashMap();
     }
 
 
@@ -350,7 +350,7 @@
     }
 
     /**
-     * check wether we should follow the redirect given in the response
+     * check whether we should follow the redirect given in the response
      * make sure we don't run into a recursion
      * @param response
      * @return
@@ -371,22 +371,42 @@
               url = new URL(response.getURL(), redirectLocation);
           }
       } catch (MalformedURLException e) {
-          // Fall through and allow existing exception handling code deal 
-          // with any exception - we don't know at this stage whether it is
-          // a redirect instruction, although it is highly likely, given 
-          // there is a location header present in the response!
+          throw new RedirectionException(response.getURL(),
+              "Error constructing redirect URL using location '"
+        		  + redirectLocation + "'.", e);
+    	  
       }
       
       switch (response.getResponseCode()) {
       	case HttpURLConnection.HTTP_MOVED_PERM:
   	    case HttpURLConnection.HTTP_MOVED_TEMP:	// Fall through
-  	        if (null != url && _redirects.contains(url)) {
-  	            // We have already been instructed to redirect to that location in
-  	            // the course of this attempt to resolve the resource
-  	            throw new RecursiveRedirectionException(url, 
+  	    	ClientProperties props = _client.getClientProperties();
+
+  	    	// First check for an excessive number of redirects
+  	    	if ((props.getMaxRedirects() >= 0) &&
+  	    			(_redirects.size() >= props.getMaxRedirects()))
+  	    	{
+  	    		throw new RedirectionException(url,
+  	    			"Maximum redirection chain length of "	+ props.getMaxRedirects() + " exceeded.");
+  	    	}
+  	    	
+  	    	// Now check for repeat URLs
+  	        if (null != url)
+  	        {
+  	        	URL prev = (URL) _redirects.put(url, url);
+  	        	// if we have a URL comparison match
+  	        	// if resolving redirects we're done
+  	        	// else be a little more lax and just compare host names
+  	        	if ((prev != null) &&
+  	        			(props.resolveRedirectLoops() ||
+  	        			 url.getHost().equalsIgnoreCase(prev.getHost())))
+  	        	{
+  	        		// We have already been instructed to redirect to that location in
+  	        		// the course of this attempt to resolve the resource
+  	        		throw new RecursiveRedirectionException(url, 
   	                    "Unable to process request due to redirection loop");
+  	        	}
   	        }
-  	        _redirects.add(url);
       		break;
       }    
       return redirectLocation != null;
Index: src/com/meterware/httpunit/RedirectionException.java
===================================================================
--- src/com/meterware/httpunit/RedirectionException.java	(revision 0)
+++ src/com/meterware/httpunit/RedirectionException.java	(revision 0)
@@ -0,0 +1,100 @@
+package com.meterware.httpunit;
+
+/********************************************************************************************************************
+ * $Id: RedirectionException.java 908 2008-04-05 08:24:51Z wolfgang_fahl $
+ *
+ * Copyright (c) 2002-2008, Russell Gold
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+ * documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+ * to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all copies or substantial portions
+ * of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
+ * THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+ * CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ *
+ *******************************************************************************************************************/
+import java.net.URL;
+
+/**
+ * Class used to indicate when a request to a resource resulted in an HTTP
+ * redirect response which fails.
+ * 
+ * @author <a href="mailto:[email protected]">James Abley </a>
+ */
+public class RedirectionException extends RuntimeException {
+
+    private URL url;
+
+    /**
+     * Create a new <code>RedirectionException</code> with the
+     * specified URL and cause.
+     * 
+     * @param url
+     *            the <code>URL</code> that caused the redirection issue.
+     *            The URL is saved for later retrieval by
+     *            {@link #getURL()}
+     * @param cause
+     *            the cause (which is saved for later retrieval by the
+     *            {@link #getCause()}method). (A null value is permitted, and
+     *            indicates that the cause is nonexistent or unknown.)
+     */
+    public RedirectionException(URL url, Throwable cause) {
+        super(cause);
+        this.url = url;
+    }
+
+    /**
+     * Create a new <code>RedirectionException</code> with the
+     * specified URL and detail message.
+     * 
+     * @param url
+     *            the <code>URL</code> that caused the redirection issue.
+     *            The URL is saved for later retrieval by
+     *            {@link #getURL()}
+     * @param message
+     *            the detail message. The detail message is saved for later
+     *            retrieval by {@link #getMessage()}
+     */
+    public RedirectionException(URL url, String message) {
+        super(message);
+        this.url = url;
+    }
+
+    /**
+     * Create a new <code>RedirectionException</code> with the
+     * specified URL, detail message and cause.
+     * 
+     * @param url
+     *            the <code>URL</code> that caused the redirection issue.
+     *            The URL is saved for later retrieval by
+     *            {@link #getURL()}
+     * @param message
+     *            the detail message. The detail message is saved for later
+     *            retrieval by {@link #getMessage()}
+     * @param cause
+     *            the cause (which is saved for later retrieval by the
+     *            {@link #getCause()}method). (A null value is permitted, and
+     *            indicates that the cause is nonexistent or unknown.)
+     */
+    public RedirectionException(URL url, String message,
+            Throwable cause) {
+        super(message, cause);
+        this.url = url;
+    }
+
+    /**
+     * Returns the URL that caused this exception to be thrown.
+     * 
+     * @return the <code>URL</code> that gave rise to this Exception
+     */
+    public URL getURL() {
+        return url;
+    }
+}
Index: src/com/meterware/httpunit/RecursiveRedirectionException.java
===================================================================
--- src/com/meterware/httpunit/RecursiveRedirectionException.java	(revision 989)
+++ src/com/meterware/httpunit/RecursiveRedirectionException.java	(working copy)
@@ -28,10 +28,8 @@
  * 
  * @author <a href="mailto:[email protected]">James Abley </a>
  */
-public class RecursiveRedirectionException extends RuntimeException {
+public class RecursiveRedirectionException extends RedirectionException {
 
-    private URL url;
-
     /**
      * Create a new <code>RecursiveRedirectionException</code> with the
      * specified URL and cause.
@@ -44,8 +42,7 @@
      *            indicates that the cause is nonexistent or unknown.)
      */
     public RecursiveRedirectionException(URL url, Throwable cause) {
-        super(cause);
-        this.url = url;
+        super(url, cause);
     }
 
     /**
@@ -61,8 +58,7 @@
      *            retrieval by {@link #getMessage()}
      */
     public RecursiveRedirectionException(URL url, String message) {
-        super(message);
-        this.url = url;
+        super(url, message);
     }
 
     /**
@@ -83,16 +79,6 @@
      */
     public RecursiveRedirectionException(URL url, String message,
             Throwable cause) {
-        super(message, cause);
-        this.url = url;
+        super(url, message, cause);
     }
-
-    /**
-     * Returns the URL that caused this exception to be thrown.
-     * 
-     * @return the <code>URL</code> that gave rise to this Exception
-     */
-    public URL getURL() {
-        return url;
-    }
 }