Re: establishing logical test order?

For my own sanity, I'd like to have some ability to specify the order
that tests run in.  For example, one of the core components of our
application is a rules engine.  When we run tests with NUnit we want to
know that the three outcomes fire.  The logical order (or at least more
human-friendly) is pass, skip, and fail, but the alphabetization that is
done by NUnit (and other frameworks) doesn't run the tests in the
"right" order.

 

The tests could be named so that they run in the "right" order.  But, as
David suggests, that isn't very scalable.

 

Mike

 

From: [email protected]
[mailto:[email protected]] On Behalf Of David
Jeske
Sent: Monday, June 23, 2008 2:14 PM
To: Pigneri, Rocco
Cc: [email protected]
Subject: Re: [Nunit-users] establishing logical test order?

 


On Mon, Jun 23, 2008 at 11:53 AM, Pigneri, Rocco
<[email protected]> wrote:

	As you mentioned, that is not a natural feature of unit tests:
unit tests are by definition unrelated to each other and can be run in
any order without introducing failures. 


Yes, my tests can be run in any order, they are all independent tests of
a single piece of functionality. 

However, all tests depend on the proper functioning of "upstream code".
I like to see the failures in order from smallest to largest number of
dependencies, because that's where the core problem usually lies. For
example, if something broke the core .NET string class, wading through
all kinds of other tests is not necessary if your basic-core set of
tests right at the top said strings are somehow broken.
 

	However, if you wanted to group tests-which seems like your
goal-you have two options.  You can use namespaces to hierarchically
organize your tests into collapsible, related groups.  Alternatively,
you can join your four tests together (here, T00-T03) into one test.  In
this second option, you will see  a failure in T00 without running T01
because T00's code will be run before T01 and therefore cancel out
execution of T01.


I'm not quite sure I understand. I already use classes and namespaces to
organize my tests. The trouble is, when they come up alphabetically,
they are not organized with any kind of meaning. The namespace
"Utilities" is near the bottom while the "ApplicationTests" is near the
top, with a bunch of other labeled stuff all mixed around them.

This is causing me to have to name my tests with these weird ordering
numbers, but this is not really scalable to large projects, or projects
that include multiple components.

It seems like we could do some simple assembly dependency analysis to
order the tests into dependency groups. This way tests of bottom-level
stuff would naturally be grouped together, and levels would get
increasingly complicated (with more dependencies) as they went down. A
project which had to many cross-dependencies that it's just one big soup
is probably not layered correctly.  When doing my own tests pre-NUnit, I
build this manually in the makefile test order. However, if we're
automating all of this, this seems like the way to do it. Thoughts?

- David

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

_______________________________________________
Nunit-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-users