Re: Roadmap Updated

Hi Kelly, 

> The current approach seems to be that the test cases are 
> created at Load... time, and then are subsequently Run when 
> you select a test case to run.

Yes.. and in most cases, user code in the test assembly
does not participate in loading tests. The only exception
I can think of is the Suite property, which is required
to be static.

> One issue I'm having in using 
> IterativeTest is that the Load time can get unreasonably long 
> even in the case where I only want to run a single test. (It 
> also seems to occasionally gratuitously reload the tests, but 
> that's a side issue.) This seems to be the case whether I'm 
> using TestDriven.NET or the NUnit GUI, so I assume that this 
> must be a core issue, or at least both have chosen the same 
> implementation.

If tests are reloading without being told to reload, then I
would suspect your extension as being the culprit. :-)

However, there are many ways to "tell" the test to reload,
so check your NUnit options and turn off all automatic 
reloading while testing this.

> If, on the other hand, one could be lazy about test creation, 
> and only create the test when it's about to be run, that 
> would make for a more efficient system. (At least the way I'm 
> using it.) This might also be the case in general, it's just 
> more obvious for IterativeTest.

NUnit creates all "tests" at load time, but those tests are not
instances of the user test fixture class. An NUnitTestMethod
is fairly lightweight, but it does require a bit of reflection
in order to get all the properties of the test to match the
attributes you specify. Since the attributes on the method
don't change for each test case, you should check to see if
your implementation is repeating that reflection every time.
It's only needed once, but depending on your inheritance
hierarchy it could be happening more for each case.

> So, is there some way of being indeterminate until you either 
> try to expand the test in the NUnit GUI, or actually run the 
> test? Can we do lazy evaluation of the set of tests?

I'll deal with each option separately:

The Gui is completely decoupled from the test loader, which
is as it should be. The gui doesn't even have tests, just
the names of tests and other info about them. Of course,
we could have the gui drive the progressive loading of tests,
but that would not be a very clean architecture.

There is a possibility of having a test that dynamically
creates its own children when it is run. That would mean
the gui would be receiving notifications about tests that
it doesn't know about and would have to figure out where
to put them in the tree. I think we may need this at some
point, but it's much more complicated than the alternative
of using multiple result sets. I'd like to try the simpler
alternative first.

Here's what I mean by multiple result sets... Let's say
I have a method that tests multiplication. I want to 
run it with 2x2=4 0x7=0 90x1=90 as cases.

I can consider this as three tests, listed in the gui
tree as Multiply(2,2), Multiply(0,7) and Multiply(90,1).
That's how RowTest would do it, for example.

On the other hand, we can think about this as a single
TestMethod producing multiple results. The test tree
would list a single Multiply test. After execution,
the results would be placed in the tree under the test.
They would look pretty much like the tests in the first
approach - it's just a matter of how we think about it.

One advantage of this is that a TestResult is a 
very lightweight, serializable object. It has no
existence in the tree of tests created by the loader,
which is BTW different from the tree in the Gui, so
it doesn't have to be known in advance.

So that's an approach I plan to follow up on.

> It seems that you could get the name of the test from the 
> metadata without actually creating the test from calling 
> BuildFrom. Would that constitute an improvement in your 
> opinion to the current approach?
> This question is independent of 2.5 vs 3.0... although the 
> answer might not be.

If we are ready to call BuildFrom, what would we do instead
in order to defer it? First, we would have to know that
this is the type of test that needs to be deferred, but
we wouldn't know that until we built it.

Of course, the builder extension could simply save the
data and only build the subtests at the time it needed them,
but this has a few problems:

1) By that time, we have already done most of the work,
unless we are talking about very large sets of data.

2) A testcase is not a very big object anyway, so it
isn't clear we would make a net savings by caching
the info needed to create one and building it later.

> The best answer would seem to me to be that you could have 
> all the test names show up, and those that are Expandable (a 
> bool property?) would show potential expansion. Then when you 
> choose to expand it, A function is called that enumerates the 
> tests, finally, when you choose to execute a test, then 
> another function is called to prepare to run the test, 
> finally the test is run. If we could do lazy evaluation for 
> each of these steps, that would seem to be a huge improvement 
> over the current approach, at least in the cases I'm dealing with.
> 
> I'm a big fan of lazy evaluation.

Except that the entire NUnit approach to loading tests is already
based on lazy evaluation. NUnit only creates lightweight objects
that contain enough info to create your test fixture object and
invoke the tests when needed. I'm suspicious that you may be
doing something to defeat this in your extension - I'll try
to find time to look over the code.

> >  > The Theory syntax seems to merge these two approaches... 
> so  > I'm 
> > not sure whether or not it remains an unsolvable problem  > 
> to merge 
> > the two approaches.
> >
> >  No - the Theory examples we have seen use both approaches. 
> So  NUnit 
> > would need to support both of them... unless that's what  
> you mean by 
> > merge. :-)
> 
> Yes, that's what I mean. Internally this requires two 
> different approaches, of course.

And to repeat the point Andreas was making, we would like
each extension that uses data to be able to use any data
providers interchangeably. Just as Theory does.

> >  > The conversation you pointed to was technically a little 
> past  > my 
> > experience thus far (too internal), so while I'm very  > 
> interested, I 
> > couldn't make a helpful reply.
> >
> >  That's cool. Jump in where you can. We're at the implementation  
> > stage, where it's a question of how the internals will 
> work, so  it's 
> > natural that we're getting more technical. One way to read  
> this is to 
> > ask how the availabilitiy of some NUnit feature or  interface would 
> > help you to implement a given extension syntax.
> 
> I'm not especially dense, but I am unfamiliar with most of 
> the NUnit internals. I feel most comfortable and useful at 
> this point looking at things from the user's point of view. 
> By user, I mean both a general user as well as someone 
> writing extensions. I am not against getting my hands dirty 
> as might be helpful. I just recognize that there is ramp up 
> time involved to get to that point.

Bear in mind, for better or worse, NUnit extensibility has
never been targeted at the general user.  In fact, that is
a valid criticism some people have made. So, you do need
to get into certain internals - SimpleTestRunner and everything
it calls - to work effectively. But it's just code. :-)

Charlie

> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/