Re: Dealing with 'external' resources and Global configuration

Dear Tim

thanks for the excerpt,
IMHO theres is a big difference between what the JUnit 
testcases for httpunit do and what people can do with 
httpunit as a testing framework.

Of course you can test external resources and 
frameworks. And some people try to do just that 
effectively using httpunit as some kind of web-
macro/robot of whatever one might call it.

The JUnit test cases of httpunit don't do this so that 
you can run all JUnit testcases without any internet 
access at all.

I just double checked by physically disconnecting my 
development machine from the internet - and voila - no 
errors - 759/759 test cases run.

So as it currently is the way people use httpunit and 
the way httpunit tests itself is different. This is the 
reason why I think we should start having further tests 
that check real-world connections. A first step is to 
test against a real server on the local network. To do 
so it's important that the server is under the control 
of the httpunit development environment. That's where I 
suggested something like XAMPP where you get a full 
package with Webserver (Apache) Database (MySQL) and 
simple development environment (PHP) in one defined 
package with example applications. XAMPP does not eat 
resources unless you start it (it is not installed as a 
service)

A second step ist to test against external web servers 
that contain a stable environment which is under control 
of the httpunit project (e.g. meterware.com, bitplan.com 
or httpunit.org or such)

A third step is to test against external web servers 
that contain pretty stable environments which are not 
under our control but rarely change (e.g. google.com, 
ebay.com, you name it)

A fourth sep is to test against any external web server 
that somebody might come up with (e.g. in a bug report).

Obviously steps 1-4 need to be configurable because 
there are lots of followup-problems with any step upward 
this list - you can't expect that all tests from all 
steps run on all machines out there for all the time.


So how much do we gain / loose in this situation by 
extending what we expect from httpunit and where would 
we go in the version 2.0+ world?

In a perfect world there would be just one kind of
web server that fits to exactly one specification.

But currently there is:
	Apache 1.x, 2.x
	IIS ...
	Tomcat
	JBoss
	Websphere	
	BEA Weblogic
	... you name it


The specs are Internet RFCs, W3C documents and ...
you name it ...

In a perfect world there would be just one kind of 
browser that fits to exactly one specification.
But currently there is
	Internet Explorer x.y
	Firefox x.y
	Safari
	Mozilla

you name it ...

So when you design a web application it needs to be 
decided what you target is and that will be a 
combination of things.

E.g. you could decide to create an application using IIS 
and optimize it for Internet Explorer using the latest 
DotNet technology. If you hate Microsoft you'll never do 
that may be you'll go for a Tomcat Solution optimized 
for Firefox.

Or you'll have the dilemma of quite a few people that 
you can only influence the server side but not the 
client side of thing. You need to supply a solution that 
allows to support quite a bunch of browsers - but the 
server side can be decided by you.

Based on these decisions your possibilities get limited 
and in a real world situation you have to use lots of 
work-arounds to get around some of the limitations.

And now you'd like to test whether things run as 
expected  ... Your test framework would now ideally be 
an environment that not only fits to specifications, 
server and browser implementations but does everything 
at the same time.

Such a super-test-environment does not exist and will 
never exist. But httpunit does a pretty good job. For 
people that have a design by testability approach 
httpunit is even the limiting factor. 

For me it is as simple as this:
	What httpunit can not test i will not implement!

I'm not sure how many people are out there that have the 
same opinon but I'm pretty sure that no matter whether 
you have that opinion or not improving httpunit will 
finally improve your own results (and be it only that 
tests are more stable, test more of your application are 
quicke to implement, better to maintain or whatever is 
important for your development work)

So my main motivation to improve httpunit is to be able 
to do more things better by raising the limits. E.g. i 
have never used Ajax so far and I won't if I can't test 
it. 

Let's take that Ajax example:
Compared to Ajax httpunit is really mature. If Ajax is 
going to persist and will be nicely standardized and in 
widespread httpunit should support testing it. 

Now to the more general issue what httpunit is and 
should be in the future:

There are several approaches to accomplish what a test 
framework for web-apps should do. To fit into the web-
app picture the test-framework needs to be able to work 
with both the server and client side aspects of the 
environment. The client side is far more important since 
this is the users's perspective. So the framework needs 
to do what a browser would do. This can be done by 
emulating a browser or by using a browser (see 
selenium). What works better depends what you further 
expect - should the tests run against spec or should 
they run against the behaviour of real world browsers 
out there.

For the server side it's comparable - should the tests 
run against spec or work as a real server would do?

And how about debugging information - do I get that from 
my client and server or do I produce it by emulating and 
analyzing what goes on?

Interestingly enough I never cared for these details too 
much while I was using httpunit since 2004. I have used 
it to test my apps generated from UML and based on a 
requirements database. I'm using the Y-principle and 
differentiate totally between technology and content. So 
when I test my cancer reporting applications I simply 
set up a test like the one below that is generated from 
the requirements database. Please note that I don't have 
to care for details of the app like the Login to much - 
it's all behind the scenes so that I can try to 
concentrate on what my customers want and not so much on 
the technology that does it. The test below should also 
work after everything was changed (e.g. to Ajax) without 
changing it.

By the way the app below is implemented in PHP and the 
same test exists a second time as a PHP test checking 
only the application logic - this part is simply for 
checking the view produced - for those wondering why the 
expected Result here is so short. 

/**
 * SITUATION
 * A female patient with rectal cancer has been treated.  She shows mild 
 * vaginal discharge and patchy ulcerations in week five.  That was the 
 * most severe acute side effect. After six month she experienced 
 * stenosis and no intercourse was possible anymore. It was not 
 * correctible.
 * 
 * @throws Throwable if this method fails
 */
	protected void situation() throws Throwable {
	// >>>{SITUATION}{AC001}{wf04000037}
		helper=new GenepiHelper();
		tester=helper.tester();
		watcher=helper.watcher();
		helper.login(GenepiHelper.PARTICIPANT);
		gpic=helper.addrectalpatient("2005-05-04");
	// <<<{SITUATION}{AC001}{wf04000037}
	}

/**
 * ACTION
 * The side effect score shall be entered into the system.
 * 
 * @throws Throwable if this method fails
 */
	protected void aktion() throws Throwable {
	// >>>{ACTION}{AC001}{wf04000037}
		helper.patientlist();
		helper.selectpatient(gpic); 
		helper.clickButton("FollowupSideeffect");	
		helper.clickButton("add");	
	// <<<{ACTION}{AC001}{wf04000037}
	}

/**
 * EXPECTED RESULT
 * after entering rectal cancer and gender "f" the system will offer the 
 * list of pelvic side effects to be scored for this combination which 
 * includes the female applicable of the 16 items from rectal fistula to 
 * vaginal mucositis. and it will include "vaginal mucositis". When 
 * selecting this side effect the 6 valid scores from 0=none to 5=death 
 * will show and the explanation will be available so that the 
 * distinction between grade 2 "correctible" and 3 "not correctible" can 
 * be done with assistance of the system. The side effect will be 
 * considered as early due to the three week statement.
 * 
 * @return true if expectedResult was available
 * @throws Throwable if this method fails
 */
	protected boolean expectedResult() throws Throwable {
		boolean result=false;
	// >>>{EXPECTEDRESULT}{AC001}{wf04000037}
		tester.assertTextPresent("Vaginal mucositis");    		
		result=true;
	// <<<{EXPECTEDRESULT}{AC001}{wf04000037}
		return result;
	} // expectedResult




For those interested I might give more inside into this 
approach and supply some of the library (as I've partly 
already done via the mailing list or personal e-mails) 
One of the things the library does is creating snapshots 
and displaying them in a browser while you go. That's 
something I consider useful when talking with customers 
because I can go thru the app test by test click by 
click this way and check whether my test specs fit his 
requirements.

Just my two cents (or was that a bit more ...?)

Yours
  Wolfgang










BITPlan - smart solutions
Pater-Delp-Str. 1, D-47877 Willich-Schiefbahn
Tel. +49 1805 - BITPLAN / +49 1805 248 752, Fax +49 2154 
811-481
Web: http://www.bitplan.de
bitplan GmbH, Willich - HRB 6820 Krefeld, VAT-ID: 
10258040584, Geschäftsführer: Wolfgang Fahl
Mit BITPlan effizient zum Erfolg: http://www.bitplan.de


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace