Re: NUnit 2.5: What's In It Now

On Wed, Apr 9, 2008 at 4:46 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>
>  > That would seem to indicate that you want the Exception types
>  > to be orthogonal to the Test types. That's something worth
>  > thinking about for sure, but I don't pretend to know the
>  > answer at this point.
>
>  "Orthogonal" to me would mean that you can decide the two things
>  independently, not that all test types use the same exceptions.

Fair enough.

>  This really isn't a preference of mine - one way or the other -
>  it's just a simple fact on the ground. For example, a csUnit
>  test indicates failure by throwing csunit.SomeExceptionIForgot,
>  while NUnit test cases throw NUnit.Framework.Exception and
>  Microsoft throws one with a really long name. Would it make
>  sense to say that NUnit should "understand" the MS exception?
>  I don't think so.

No, probably not. But why impose that level of independence upon our
own set of tests? I can see when we are trying to emulate another
testing framework that it needs to be kept independent. But within our
own Test, IterativeTest, etc. why do we need that same level of
independence?

>  > Apparently the new MbUnit has much more orthogonality than
>  > NUnit, or even previous versions of MbUnit. They can combine
>  > all kinds of things in ways that the authors might not have
>  > expected. I'm not entirely sure that's a good thing, as it
>  > *might* lead to results that would be hard to predict (for
>  > the author and the user). Then again, it does give more flexibility.
>
>  I'm pretty sure that MbUnit has precisely the degree of orthogonality
>  that the authors intended - no more no less. Same for NUnit.

Their intention apparently was to have more orthogonality than NUnit does.

>  There is a difference between providing a facility for orthogonal
>  application of a feature and requiring it. What you are proposing
>  is to require all future extensions to accept your extension and
>  I disagree with that philosophically.

I agree with you now that I understand things better.

>  > Right. But they could interpret AssertionException
>  > differently, which would be confusing if you as a user
>  > thought the concepts were orthogonal when they weren't. In
>  > designing this, the principle of least surprise becomes very
>  > important. For me, I never conceived of a tie between
>  > Exception types thrown by Assert or Assume and test types.
>  > As we increase the number of test types, we risk confusing
>  > users if we don't explicitly state that each test type deals
>  > with different exception types differently. This is basic to
>  > how it all works, and I didn't get it until you beat it into
>  > my head here. I can't imagine that a casual user would get
>  > that concept, and now that we have more than one test type,
>  > it becomes a much more important concept to get.
>  > Maybe not quite so much now, because both of the supported
>  > test types do support AssertException, but that just serves
>  > to reinforce my previous idea that they were somehow
>  > orthogonal. Do you see where I'm coming from in usability terms?
>
>  Casual users don't deal with the exception mechanism. They deal
>  with tests failing and succeeding. If I have an assert fail,
>  and I'm linked with the csUnit dll, it throws a csUnit type
>  of exception. If I'm linked with the NUnit.Framework dll,
>  then it throws an nunit type of exception. This is completely
>  transparent - but it can only be transparent because the test
>  types are smart about it.

How do I, as a casual user, learn that I can't call Assume.That inside
of a Test? I could assume that it would in a particular way, and it
wouldn't. How do I discover that fact as a casual user?

>  I think the problem here is that you are thinking of extensibility
>  as applied only to NUnitTestMethods. Within that domain, what you
>  say is reasonable.

Yes, that's what I was talking about.

>  It's just not the whole domain of NUnit
>  extensibility. When you say that something should be pushed down
>  into the core of NUnit, then you are saying it is common to
>  NUnit, csUnit and VSTS tests today and possibly JUnit, xUnit.net
>  and others in the future. At that level of generality, there
>  is commonality, but it's pretty abstract.

Ok, I get that and agree.

>  Because mbUnit has never defined extensibility to include "able
>  to run tests from other frameworks" they are able to get away
>  with a simpler implementation. THere is complexity here, but
>  it isn't accidental - it's rooted in the problem being solved.

So, do we want to create an MbUnit test runner as an extension to NUnit?

>  > Then there would be isomorphism in the two structures.
>
>  There really isn't a need for isomorphism. The two structures
>  should really mean different things - one is on a test and
>  is a characteristic of the test, the other is on a result
>  and is a characteristic of a single execution of the test.

Ok.

>  > >  In the last case, there is a sub-category of possibilities,
>  > > depending on whether NUnit calls the Theory once or
>  > multiple  times -
>  > > i.e. once per data point.
>  >
>  > It doesn't feel very Theoryish to only call it once. :-)
>
>  Of course, the test method would be called many times, but
>  the Theory isn't the test method. I believe the original
>  implementation simply runs a Theory and gets a single
>  result.

Ok, that's a new idea. I figured you would want to know the specific
data that made the Theory fail so that you could do something about
it. I'm not sure how to present that if running the Theory is an
atomic operation, but it does make me think about things in a
potentially different way.

>  > Ok, does it stifle further investigation to add Assume.That
>  > to Gary's Theory Addin? I don't think so, but people might
>  > just assume that's the only way to do things.
>
>  No. We can have all sorts of addins. I think our discussion
>  suffers because context matters to me: whether we are talking
>  about 2.5 or 3.0, whether we are talking about an addin or
>  the "official" implementation of NUnit. You are more
>  pragmatic, and it makes it hard for you to see why someone
>  would care about such things - or that's how it appears.

I would agree that I am pragmatic, and trying to get things done. You
are trying to preserve the integrity of NUnit over the long haul.
These are not entirely compatible ways to look at the world, and that
leads to lively discussion. Neither is bad, just different. I do mix
up the 2.5 and 3.0 stuff fairly freely, and that causes some
confusion. The point though is that if we know where we want 3.0 to
go, it can help in figuring out what can and cannot be put into 2.5.

>  > So why not let people throw all the result kinds directly?
>  > Who knows how useful that could end up being? I don't.
>
>  I already said I was implementing  the ability to set the
>  test result to success and probably inconclusive. We don't
>  want to encourage people to throw exceptions, however,
>  because that's an implementation detail that could change.

Ok. I'd be interested to hear the alternative you may have in mind.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone