Re: Roadmap Updated

On Mon, Mar 10, 2008 at 6:06 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>  > The current approach seems to be that the test cases are
>  > created at Load... time, and then are subsequently Run when
>  > you select a test case to run.
>
>  Yes.. and in most cases, user code in the test assembly
>  does not participate in loading tests. The only exception
>  I can think of is the Suite property, which is required
>  to be static.

When we're talking about RowTest, IterativeTest or Theory type
approaches, this doesn't hold, and user code can and does get
involved.

>  > One issue I'm having in using
>  > IterativeTest is that the Load time can get unreasonably long
>  > even in the case where I only want to run a single test. (It
>  > also seems to occasionally gratuitously reload the tests, but
>  > that's a side issue.) This seems to be the case whether I'm
>  > using TestDriven.NET or the NUnit GUI, so I assume that this
>  > must be a core issue, or at least both have chosen the same
>  > implementation.
>
>  If tests are reloading without being told to reload, then I
>  would suspect your extension as being the culprit. :-)

Possibly. I'll look into that. Since I started with RowTest, I can't
claim to have a 100% understanding of everything that's going on right
yet.

>  However, there are many ways to "tell" the test to reload,
>  so check your NUnit options and turn off all automatic
>  reloading while testing this.

Ok, I'll look at that too.

>  > If, on the other hand, one could be lazy about test creation,
>  > and only create the test when it's about to be run, that
>  > would make for a more efficient system. (At least the way I'm
>  > using it.) This might also be the case in general, it's just
>  > more obvious for IterativeTest.
>
>  NUnit creates all "tests" at load time, but those tests are not
>  instances of the user test fixture class. An NUnitTestMethod
>  is fairly lightweight, but it does require a bit of reflection
>  in order to get all the properties of the test to match the
>  attributes you specify. Since the attributes on the method
>  don't change for each test case, you should check to see if
>  your implementation is repeating that reflection every time.
>  It's only needed once, but depending on your inheritance
>  hierarchy it could be happening more for each case.

I must be doing more in the BuildFrom than you are expecting I would
do. In reading this email, and pondering the issue, it strikes me that
I could make my tests do lazy evaluation themselves. So perhaps that's
the better solution, rather than having NUnit do something different.
I don't think it's the IterativeTest Addin, per se, but just the way
I'm using it that's inefficient.

>  > So, is there some way of being indeterminate until you either
>  > try to expand the test in the NUnit GUI, or actually run the
>  > test? Can we do lazy evaluation of the set of tests?
>
>  I'll deal with each option separately:
>
>  The Gui is completely decoupled from the test loader, which
>  is as it should be. The gui doesn't even have tests, just
>  the names of tests and other info about them. Of course,
>  we could have the gui drive the progressive loading of tests,
>  but that would not be a very clean architecture.

I can see where it could get kind of coupled...

>  There is a possibility of having a test that dynamically
>  creates its own children when it is run. That would mean
>  the gui would be receiving notifications about tests that
>  it doesn't know about and would have to figure out where
>  to put them in the tree. I think we may need this at some
>  point, but it's much more complicated than the alternative
>  of using multiple result sets. I'd like to try the simpler
>  alternative first.

Also complicated.

>  Here's what I mean by multiple result sets... Let's say
>  I have a method that tests multiplication. I want to
>  run it with 2x2=4 0x7=0 90x1=90 as cases.

Which could come from some kind of autogenerator, in the Theory case.

>  I can consider this as three tests, listed in the gui
>  tree as Multiply(2,2), Multiply(0,7) and Multiply(90,1).
>  That's how RowTest would do it, for example.

Sure, that's one approach.

>  On the other hand, we can think about this as a single
>  TestMethod producing multiple results. The test tree
>  would list a single Multiply test. After execution,
>  the results would be placed in the tree under the test.
>  They would look pretty much like the tests in the first
>  approach - it's just a matter of how we think about it.

That sounds cleaner.

>  One advantage of this is that a TestResult is a
>  very lightweight, serializable object. It has no
>  existence in the tree of tests created by the loader,
>  which is BTW different from the tree in the Gui, so
>  it doesn't have to be known in advance.
>
>  So that's an approach I plan to follow up on.

I think that's a good plan. There remains what to do in the GUI tree
before the test has run... do you add the + only after the tests have
run? How to differentiate those sorts of tests from those who never
have children? Just details, of course.

>  > It seems that you could get the name of the test from the
>  > metadata without actually creating the test from calling
>  > BuildFrom. Would that constitute an improvement in your
>  > opinion to the current approach?
>  > This question is independent of 2.5 vs 3.0... although the
>  > answer might not be.
>
>  If we are ready to call BuildFrom, what would we do instead
>  in order to defer it? First, we would have to know that
>  this is the type of test that needs to be deferred, but
>  we wouldn't know that until we built it.
>
>  Of course, the builder extension could simply save the
>  data and only build the subtests at the time it needed them,
>  but this has a few problems:
>
>  1) By that time, we have already done most of the work,
>  unless we are talking about very large sets of data.

I was... In my IEnumerator, I was loading the XML files I was testing.
Now, I think I'll restructure it so that it only makes the list of
files, and loads them later, as necessary. My bad.

>  2) A testcase is not a very big object anyway, so it
>  isn't clear we would make a net savings by caching
>  the info needed to create one and building it later.

Only if your ListBuilder is implemented inefficiently, as mine is.

>  > The best answer would seem to me to be that you could have
>  > all the test names show up, and those that are Expandable (a
>  > bool property?) would show potential expansion. Then when you
>  > choose to expand it, A function is called that enumerates the
>  > tests, finally, when you choose to execute a test, then
>  > another function is called to prepare to run the test,
>  > finally the test is run. If we could do lazy evaluation for
>  > each of these steps, that would seem to be a huge improvement
>  > over the current approach, at least in the cases I'm dealing with.
>  >
>  > I'm a big fan of lazy evaluation.
>
>  Except that the entire NUnit approach to loading tests is already
>  based on lazy evaluation. NUnit only creates lightweight objects
>  that contain enough info to create your test fixture object and
>  invoke the tests when needed. I'm suspicious that you may be
>  doing something to defeat this in your extension - I'll try
>  to find time to look over the code.

I don't think that's necessary... I think I know my problem, and it's
not in the IteratorTest code.

>  > Yes, that's what I mean. Internally this requires two
>  > different approaches, of course.
>
>  And to repeat the point Andreas was making, we would like
>  each extension that uses data to be able to use any data
>  providers interchangeably. Just as Theory does.

Yup.

>  > I'm not especially dense, but I am unfamiliar with most of
>  > the NUnit internals. I feel most comfortable and useful at
>  > this point looking at things from the user's point of view.
>  > By user, I mean both a general user as well as someone
>  > writing extensions. I am not against getting my hands dirty
>  > as might be helpful. I just recognize that there is ramp up
>  > time involved to get to that point.
>
>  Bear in mind, for better or worse, NUnit extensibility has
>  never been targeted at the general user.  In fact, that is
>  a valid criticism some people have made. So, you do need
>  to get into certain internals - SimpleTestRunner and everything
>  it calls - to work effectively. But it's just code. :-)

I'm hoping that this is something we want to change in the 3.0 time
frame. I think extensibility should be targeted at the general user.

It is just code, and all it takes to understand it is just time. :-)

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/