Re: establishing logical test order?

Hi David,
 
Interesting references - I'll look through them more closely.
 
Regarding dynamic versus static analysis:
 
Actually, using the profiler API only gets us the ability to do coverage
analysis if we choose to 
implement coverage within NUnit itself. If we want to do coverage with
another product that
uses the profiler API, this presents a bit of a problem, since only one
profiling app can
be active in a process at a given time.
 
As you point out, the profiling api is COM-based and can be tricky to
program. I've written
one of them and would go through a lot to avoid having to do it again. :-)
 
For static analysis, another candidate I'm interested in is the mono cecil
project.
 
Charlie


  _____  

From: David Jeske [mailto:[email protected]] 
Sent: Monday, June 23, 2008 8:00 PM
To: Charlie Poole
Cc: [email protected]
Subject: Re: [Nunit-users] establishing logical test order?


On Mon, Jun 23, 2008 at 4:55 PM, Charlie Poole <[email protected]>
wrote:



1. It can mean "Run test A before test B because test B depends on test A's
side effects in order to run correctly".
 
2. It can mean "Run test A before test B because test B - even though it is
independent - will always fail if test A fails." 
 
3. It can mean "Run the tests in any order, but display "important" results
first."
 
We'll probably never suport the first interpretation in NUnit but the other
two could reasonably be done.


Agreed 100%.  I want #3.

I'd really like to see NUnit eventually allow the JUnit option to "run tests
in random order" to assure they are not dependent on each other. Which would
assure developers don't accidentally fall into the bad habit of #1. 




Assembly level dependency analysis is easy to do, but it's also easy to run
your assemblies in the order you manually determine to be appropriate.
Analysis at a class or method level would be more useful, but isn't as
simple.


I hadn't considered this. When I  said "assembly" I was thinking "class". To
be mose useful this would need to be TestFixture and/or Test level analysis.
However, I agree it's more complicated than what I was thinking about
originally when I suggested assembly analysis. (See below)




Various schemes have been proposed for marking tests as "dependent" on
others. We'll
probably implement one of them in the next release.


This seems reasonable. If we could specify test-class and assembly
dependencies for a test or textfixture, even manually, it would probably go
a long way.

----------

I can see two ways to do automated dependency analysis. One is using the CLR
profiling API. This would also get us the ability to do code-coverage
analysis. I think this is probably the better way, though the API is only
for unmanaged code.

  http://msdn.microsoft.com/en-us/magazine/cc188693.aspx

The other method is to do static analysis using some code that can read IL.
While that may sound scarry, we probably get by looking for the call
instructions and looking at the types of the destinations. However, this
wouldn't produce implementation dependencies across interfaces though. The
tricky part is decoding IL. It looks like there is some code in the "cosmos
OS" and there might be some code do to this in Mono/ildasm.

http://www.codeplex.com/Cosmos/SourceControl/DirectoryView.aspx?SourcePath=%
24%2fIL2CPU%2fsource%2fIndy.IL2CPU.IL
<http://www.codeplex.com/Cosmos/SourceControl/DirectoryView.aspx?SourcePath=
%24%2fIL2CPU%2fsource%2fIndy.IL2CPU.IL&changeSetId=14490> &changeSetId=14490

http://www.blong.com/Conferences/DCon2003/ReverseEngineering/ReverseEngineer
ing.htm#Ref14

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

_______________________________________________
Nunit-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-users