Re: establishing logical test order?

Responding to various comments here...
 
As it stands, the only order guarantee that NUnit provides is that multiple
assemblies in a project
will be run in the order they are specified. Running tests in alphabetical
order is not a guarantee,
just an accident of implementation. Specifically, folks like to see their
tests in alphabetical order
and (other) folks like to see the tests execute in the order they are
displayed. We could break
the ordering constraint at any time.
 
Ordering of tests is an overloaded concept.
 
1. It can mean "Run test A before test B because test B depends on test A's
side effects in order
to run correctly".
 
2. It can mean "Run test A before test B because test B - even though it is
independent - will
always fail if test A fails." 
 
3. It can mean "Run the tests in any order, but display "important" results
first."
 
We'll probably never suport the first interpretation in NUnit but the other
two could reasonably
be done.
 
Assembly level dependency analysis is easy to do, but it's also easy to run
your assemblies in
the order you manually determine to be appropriate. Analysis at a class or
method level would
be more useful, but isn't as simple.
 
Various schemes have been proposed for marking tests as "dependent" on
others. We'll
probably implement one of them in the next release.
 
Charlie


  _____  

From: [email protected]
[mailto:[email protected]] On Behalf Of Mike Chess
Sent: Monday, June 23, 2008 2:24 PM
To: David Jeske; Pigneri, Rocco
Cc: [email protected]
Subject: Re: [Nunit-users] establishing logical test order?



For my own sanity, I'd like to have some ability to specify the order that
tests run in.  For example, one of the core components of our application is
a rules engine.  When we run tests with NUnit we want to know that the three
outcomes fire.  The logical order (or at least more human-friendly) is pass,
skip, and fail, but the alphabetization that is done by NUnit (and other
frameworks) doesn't run the tests in the "right" order.

 

The tests could be named so that they run in the "right" order.  But, as
David suggests, that isn't very scalable.

 

Mike

 

From: [email protected]
[mailto:[email protected]] On Behalf Of David Jeske
Sent: Monday, June 23, 2008 2:14 PM
To: Pigneri, Rocco
Cc: [email protected]
Subject: Re: [Nunit-users] establishing logical test order?

 


On Mon, Jun 23, 2008 at 11:53 AM, Pigneri, Rocco <[email protected]>
wrote:

As you mentioned, that is not a natural feature of unit tests: unit tests
are by definition unrelated to each other and can be run in any order
without introducing failures. 


Yes, my tests can be run in any order, they are all independent tests of a
single piece of functionality. 

However, all tests depend on the proper functioning of "upstream code". I
like to see the failures in order from smallest to largest number of
dependencies, because that's where the core problem usually lies. For
example, if something broke the core .NET string class, wading through all
kinds of other tests is not necessary if your basic-core set of tests right
at the top said strings are somehow broken.
 

However, if you wanted to group tests-which seems like your goal-you have
two options.  You can use namespaces to hierarchically organize your tests
into collapsible, related groups.  Alternatively, you can join your four
tests together (here, T00-T03) into one test.  In this second option, you
will see  a failure in T00 without running T01 because T00's code will be
run before T01 and therefore cancel out execution of T01.


I'm not quite sure I understand. I already use classes and namespaces to
organize my tests. The trouble is, when they come up alphabetically, they
are not organized with any kind of meaning. The namespace "Utilities" is
near the bottom while the "ApplicationTests" is near the top, with a bunch
of other labeled stuff all mixed around them.

This is causing me to have to name my tests with these weird ordering
numbers, but this is not really scalable to large projects, or projects that
include multiple components.

It seems like we could do some simple assembly dependency analysis to order
the tests into dependency groups. This way tests of bottom-level stuff would
naturally be grouped together, and levels would get increasingly complicated
(with more dependencies) as they went down. A project which had to many
cross-dependencies that it's just one big soup is probably not layered
correctly.  When doing my own tests pre-NUnit, I build this manually in the
makefile test order. However, if we're automating all of this, this seems
like the way to do it. Thoughts?

- David

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

_______________________________________________
Nunit-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-users