Re: NUnit 2.5: What's In It Now

Hi Kelly, 

> >  > I don't know what you mean here... Assume and Assert are an  > 
> > orthogonal concern to different Test types. I might be  > missing 
> > something as my brain is a bit hammered today (dang  > ole day job 
> > anyway).
> >
> >  I'm not getting what you mean by "orthogonal concern to different  
> > test types."
> 
> What I was thinking was that both Assume and Assert should be 
> handled by Test, DataTest, IterativeTest, etc., etc. 
> independently of each of them individually writing code to 
> handle them. But that's not how the current architecture 
> works, apparently.

I believe you are thinking of extensions that are intended by
their author to be sub-types of an NUnitTestMethod. But NUnit
supports writing tests that have nothing whatsoever to do with
NUnitTestMethod and it's those types where I think it would
not make sense.

> >  IFF you want every test type - now and future - in NUnit to  
> > understand the AssumeException and handle it in the same way,  then 
> > you would need to modify the core nunit test runner.
> 
> That is what I wanted. Apparently not what you want. That is 
> a matter of opinion, which is fine.

This is a fairly large area of discussion. Who should get to
decide whether a particular extension applies to a particular
kind of test? The extension author? The author of the original
test type? The user? IMO, they should all have a say. That's
why I want to move as many such policy decisions out of the 
core as I possibly can, leaving a small set of central
concepts.

Take a simple example: Should you be able to extend
RowTest? Maybe yes, maybe no. It depends on whether
RowTest is intended to be a pure emulation of MbUnit's
RowTest or just another way to express data. There isn't
a simple answer that applies to all types of tests - heck,
there may not even be a simple answer for RowTest.
 
> >  Personally, I don't think it's desirable to allow an  extension to 
> > affect every conceivable type of test,  whether existing now or 
> > written later on. This could  be very confusing to users.
> 
> Potentially, since we don't know what types of tests there might be.
> Is it true that ALL test types understand Assert or not? Do 
> you have to write code to understand Assert in each TestCase type?

I assume by "understand Assert" you mean "interpret
an AssertionException as a test failure." The answer
is no. Each type of test has to decide whether it's
tests failed. 

We can imagine a test that returned true on success, 
false on failure. Such a test class would probably
contain code like
  if ( retval )
     result.Success();
  else
     result.Failure();
What's "core" here is the TestResult and its methods.
That's why TestResult is in nunit.core.interfaces.
 
> It would be nice to have another choice of test result types, 
> but that's the only thing needed for Theory, and you've 
> already talked about adding it, I think.

There are currently three types defined in the ResultState
enum and, logically speaking, four others in the RunState
enum. I plan to add the "implied" results (Ignored, Skipped,
Explicit, NotRunnable) to ResultState as well.  [As an example,
RunState.Igonored means "this test should be ignored" while
ResultState.Ignored meas "this test was ignored." They don't
always match, since you can ignore a test dynamically.]

I see a need for an Inconclusive result state, which may
or may not be enough for Theory - we'll see as we do it.
 
> What else would you have it do? I suppose you could collect 
> together the set of Theory runs that resulted in Assume being 
> thrown, and that's a good thing, in fact, I think I left a 
> TODO: in the code I sent you about that. What other things?

I wasn't recommending - just commenting. But in a Theory
implementation, I think we need some way to indicate that
the data was not suitable for the Theory. Either we plan to
never pass it bad data (in which case Assume should be an error)
or we filter it externally (in which case Assume might not be
needed) or we let the test filter it, in which case we need
to give some indication that says "insufficient evidence."

In the last case, there is a sub-category of possibilities,
depending on whether NUnit calls the Theory once or multiple
times - i.e. once per data point. 

There are many ways I can imagine to skin this cat and I
don't want to presume one of them too early, particularly
since it's trivial to experiment with them as addins. If
we were to add Assume.That to NUnit prematurely, we would
be stifling further investigation, IME.

 
> >  In regular tests, your Assume.That would mean something like  
> > Succeed.If. That may be useful. If so we'll add it. In fact,  we 
> > already have a request for Assert.Succeed.
> 
> I think I would like that. It would also be nice if the other 
> test result types could be thrown directly. Assert.Ignore() 
> for example?
> Maybe that's already in there, I don't recall.

In the case of Ignore, it is.

Charlie
 
> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by the 2008 JavaOne(SM) 
> Conference Don't miss this year's exciting event. There's 
> still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
> .sun.com/javaone
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone