Re: establishing logical test order?

On Mon, Jun 23, 2008 at 4:55 PM, Charlie Poole <[email protected]>
wrote:

>
> 1. It can mean "Run test A before test B because test B depends on test
> A's side effects in order to run correctly".
>
> 2. It can mean "Run test A before test B because test B - even though it is
> independent - will always fail if test A fails."
>
> 3. It can mean "Run the tests in any order, but display "important" results
> first."
>
> We'll probably never suport the first interpretation in NUnit but the
> other two could reasonably be done.
>

Agreed 100%.  I want #3.

I'd really like to see NUnit eventually allow the JUnit option to "run tests
in random order" to assure they are not dependent on each other. Which would
assure developers don't accidentally fall into the bad habit of #1.

 Assembly level dependency analysis is easy to do, but it's also easy to run
> your assemblies in the order you manually determine to be appropriate.
> Analysis at a class or method level would be more useful, but isn't as
> simple.
>

I hadn't considered this. When I  said "assembly" I was thinking "class". To
be mose useful this would need to be TestFixture and/or Test level analysis.
However, I agree it's more complicated than what I was thinking about
originally when I suggested assembly analysis. (See below)

 Various schemes have been proposed for marking tests as "dependent" on
> others. We'll
> probably implement one of them in the next release.
>

This seems reasonable. If we could specify test-class and assembly
dependencies for a test or textfixture, even manually, it would probably go
a long way.

----------

I can see two ways to do automated dependency analysis. One is using the CLR
profiling API. This would also get us the ability to do code-coverage
analysis. I think this is probably the better way, though the API is only
for unmanaged code.

  http://msdn.microsoft.com/en-us/magazine/cc188693.aspx

The other method is to do static analysis using some code that can read IL.
While that may sound scarry, we probably get by looking for the call
instructions and looking at the types of the destinations. However, this
wouldn't produce implementation dependencies across interfaces though. The
tricky part is decoding IL. It looks like there is some code in the "cosmos
OS" and there might be some code do to this in Mono/ildasm.

http://www.codeplex.com/Cosmos/SourceControl/DirectoryView.aspx?SourcePath=%24%2fIL2CPU%2fsource%2fIndy.IL2CPU.IL&changeSetId=14490

http://www.blong.com/Conferences/DCon2003/ReverseEngineering/ReverseEngineering.htm#Ref14

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

_______________________________________________
Nunit-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-users