Re: establishing logical test order?

On Mon, Jun 23, 2008 at 11:53 AM, Pigneri, Rocco <[email protected]>
wrote:

>  As you mentioned, that is not a natural feature of unit tests: unit tests
> are by definition unrelated to each other and can be run in any order
> without introducing failures.
>

Yes, my tests can be run in any order, they are all independent tests of a
single piece of functionality.

However, all tests depend on the proper functioning of "upstream code". I
like to see the failures in order from smallest to largest number of
dependencies, because that's where the core problem usually lies. For
example, if something broke the core .NET string class, wading through all
kinds of other tests is not necessary if your basic-core set of tests right
at the top said strings are somehow broken.


> However, if you wanted to group tests—which seems like your goal—you have
> two options.  You can use namespaces to hierarchically organize your tests
> into collapsible, related groups.  Alternatively, you can join your four
> tests together (here, T00-T03) into one test.  In this second option, you
> will see  a failure in T00 without running T01 because T00's code will be
> run before T01 and therefore cancel out execution of T01.
>

I'm not quite sure I understand. I already use classes and namespaces to
organize my tests. The trouble is, when they come up alphabetically, they
are not organized with any kind of meaning. The namespace "Utilities" is
near the bottom while the "ApplicationTests" is near the top, with a bunch
of other labeled stuff all mixed around them.

This is causing me to have to name my tests with these weird ordering
numbers, but this is not really scalable to large projects, or projects that
include multiple components.

It seems like we could do some simple assembly dependency analysis to order
the tests into dependency groups. This way tests of bottom-level stuff would
naturally be grouped together, and levels would get increasingly complicated
(with more dependencies) as they went down. A project which had to many
cross-dependencies that it's just one big soup is probably not layered
correctly.  When doing my own tests pre-NUnit, I build this manually in the
makefile test order. However, if we're automating all of this, this seems
like the way to do it. Thoughts?

- David

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

_______________________________________________
Nunit-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-users