Re: Roadmap Updated

Hi Kelly,

> > Hi Kelly,
> >  > The current approach seems to be that the test cases are 
>  > created 
> > at Load... time, and then are subsequently Run when  > you select a 
> > test case to run.
> >
> >  Yes.. and in most cases, user code in the test assembly  does not 
> > participate in loading tests. The only exception  I can think of is 
> > the Suite property, which is required  to be static.
> 
> When we're talking about RowTest, IterativeTest or Theory 
> type approaches, this doesn't hold, and user code can and 
> does get involved.

That's true for your implementation of IterativeTest. It's not true
for RowTest and I can't see why it would be true for a particular
implementation of a "Theory-based" approach (the latter can mean
so many things, I don't see how we can talk about it yet).

There are dangers involved when the test object (the user class
with TestFixtureAttribute on it) is created at load time. We used
to do that and it created many bugs, especially when users - bless
their hearts - create constructors with lots of side effects. Of
course, they shouldn't do that, but NUnit still has to deal with 
it and mitigate the damage. That's why we stopped creating these
objects at load time.

In any cae, I believe the way to deal with this issue is to come
up with some use cases - stories - that drive us to implementing
the sort of solution you envision. All else is theory. :-)
 
> I must be doing more in the BuildFrom than you are expecting 
> I would do. In reading this email, and pondering the issue, 
> it strikes me that I could make my tests do lazy evaluation 
> themselves. So perhaps that's the better solution, rather 
> than having NUnit do something different.

We're getting a little confused here because we are actually
talking about two things simultaneously. The first is how to
make your extension perform well. The second is how NUnit could
be changed to make it easier to write such extensions. We have
to get the first one write before dealing with the second.

> >  Here's what I mean by multiple result sets... Let's say  I have a 
> > method that tests multiplication. I want to  run it with 
> 2x2=4 0x7=0 
> > 90x1=90 as cases.
> 
> Which could come from some kind of autogenerator, in the Theory case.
> 
> >  I can consider this as three tests, listed in the gui  tree as 
> > Multiply(2,2), Multiply(0,7) and Multiply(90,1).
> >  That's how RowTest would do it, for example.
> 
> Sure, that's one approach.
> 
> >  On the other hand, we can think about this as a single  TestMethod 
> > producing multiple results. The test tree  would list a single 
> > Multiply test. After execution,  the results would be placed in the 
> > tree under the test.
> >  They would look pretty much like the tests in the first  
> approach - 
> > it's just a matter of how we think about it.
> 
> That sounds cleaner.

I think we need both approaches for different user needs. 

> >  One advantage of this is that a TestResult is a  very lightweight, 
> > serializable object. It has no  existence in the tree of 
> tests created 
> > by the loader,  which is BTW different from the tree in the 
> Gui, so  
> > it doesn't have to be known in advance.
> >
> >  So that's an approach I plan to follow up on.
> 
> I think that's a good plan. There remains what to do in the 
> GUI tree before the test has run... do you add the + only 
> after the tests have run? How to differentiate those sorts of 
> tests from those who never have children? Just details, of course.

Indeed. The gui tree now contains both tests and results. The 
results post as the test runs. Since each test curretly only
has one result, they both live in one node of the tree. If
some sort of test could have multiple results, then the tree
would need to do something different with those results, like
creating new nodes. It's straightforward enough, but with lots
of corner cases.

> >  > It seems that you could get the name of the test from the  > 
> > metadata without actually creating the test from calling  > 
> BuildFrom. 
> > Would that constitute an improvement in your  > opinion to 
> the current 
> > approach?
> >  > This question is independent of 2.5 vs 3.0... although the  > 
> > answer might not be.
> >
> >  If we are ready to call BuildFrom, what would we do 
> instead  in order 
> > to defer it? First, we would have to know that  this is the type of 
> > test that needs to be deferred, but  we wouldn't know that until we 
> > built it.
> >
> >  Of course, the builder extension could simply save the  
> data and only 
> > build the subtests at the time it needed them,  but this has a few 
> > problems:
> >
> >  1) By that time, we have already done most of the work,  unless we 
> > are talking about very large sets of data.
> 
> I was... In my IEnumerator, I was loading the XML files I was testing.
> Now, I think I'll restructure it so that it only makes the 
> list of files, and loads them later, as necessary. My bad.

I think there is more than efficiency involved here - it's also a
case of correctness for any enumerators that return different
sets of values each time the object is instantiated. Think of
some sort of data capture application.

> >  Bear in mind, for better or worse, NUnit extensibility has  never 
> > been targeted at the general user.  In fact, that is  a valid 
> > criticism some people have made. So, you do need  to get 
> into certain 
> > internals - SimpleTestRunner and everything  it calls - to work 
> > effectively. But it's just code. :-)
> 
> I'm hoping that this is something we want to change in the 
> 3.0 time frame. I think extensibility should be targeted at 
> the general user.

I disagree with this. The general user is not going to be inventing
things like IterativeTest or Theory.

There is certainly a need for a user to easily "extend" what is
happening in a test and NUnit needs to meet that need. But that's
entirely different from writing plugins for NUnit.

I think we would mess up the plugin architecture if we attempted
to make it suitable for general use. OTOH, we could easily have
a plugin - something like the addin Roy wrote - that makes it
easy for users to do test-level extensions through inheritance.

Charlie

> It is just code, and all it takes to understand it is just time. :-)
> 
> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/