Re: NUnit 2.5: What's In It Now

Hi Gary,


 
> I think it's important when RUNNING a Theory for the run to be
> repeatable, thus deterministic (if my definition of deterministic is
> the same as yours). When you go exploring the space a Theory covers,
> then it's important to be non-deterministic (but report the case(s)
> where the Theory fails so that new cases can be added to the Theory
> data source.)
> 
 
I thought that a theory should be deterministic, at least with the user
supplied data. I didn't think it would be as "rigid" as the dynamic test
data. That provides the data in "parameter sets" - i.e. sets of data that we
know work together, one of which is the expected output.
i.e. MyTest(int a, int b, int expected)
 
whereas I've seen theory syntax expressed (can't remember where) - where you
specify the set of data for each parameter INDEPENDENTLY, and then the tool
exercises the theory with whatever permutations it sees fit.
  
That's how the original (Java) implementation works. I think, there's a lot
of power in that, even without actively
searching for corner cases. The only .NET implementation I know - except for
yours - is in xunit.net, which
sticks with parameter sets.
 
MbUnit, OTOH, has both types as part of its data-driven tests - and without
any particular reference to
theory. I think that speaks to mbunit's tester-oriented origins.
 
Is this what you mean by not being deterministic? I'd expect it to exercise
every possible permutation, but can see how with large sets of data that may
be unfeasible.
 
There are strategies for maximizing coverage while not using every
permutation - google pairwise testing.
 
 This does show, however, that with even user-supplied data (if you use a
scheme like this) you might want/need assumptions even for user supplied
data and not just a theory explorer - as the assumptions may be valid for
only certain combinations of user input.
 
Yes.
 
>> My implementation throws AssumeException, which is not equivalent to
> return. My implementation of the catch of AssumeException in the test
> case makes it equivalent, but I'm kind of separating Assume.That from
> the handling of the exception. If you're talking about how the
> exception is handled, then point taken. But I believe throwing the
> exception has the right implementation for Theory...
> 
 
I disagree about an exception for assumptions necessarily being the right
implementation. For a start, there's a performance hit, and secondly, it
means that we're going to bomb out of the first assumption - there's no way
of seeing whether the data would be invalid for the other assumptions that
follow the first failing assumption (I'm not necessarily saying we want
that, but exceptions make that impossible).
 
This is a subset of the general question "how do tests report results
without throwing" One way - which may not be possible in
managed code - is to do a two-level return, leaving a result on the stack.
Another is to simply store up results somewhere for
interpretation after the eventual return. I think we need to experiment with
a few options before we'll know for sure. 
 
Charlie
 
I agree that having an Assume.That() in the code is a nicer syntax, but you
could implement this by having your TheoryMethod set a static flag, then
call Invoke, have Assume.That() set the flag and then, if any exception is
thrown, it's OK if the flag is set. Of course, having statics means that we
can't run the theories in parallel on different threads, so we probably
wouldn't want to go down that route, but I'm using the example to say that
an exception isn't necessarily the best way to implement the Assume.That()
syntax.
 
Cheers,
Gary


  _____  

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

_______________________________________________
nunit-developer mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nunit-developer