Re: Roadmap Updated

On Mon, Mar 10, 2008 at 2:31 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>  > >  My main conclusion so far is that there is an intrinsic
>  > difference
>  > > between extensions that provide data staticly and those
>  > where it  is
>  > > only available at run time.
>  >
>  > I would assume the difference you are referring to is the
>  > difference between the RowTest sort of approach and the
>  > IterativeTest approach.
>
>  Yes, at least in one aspect. In RowTest, we can look at
>  the source code and tell how many test cases will be
>  created. This would be true for any approach that places
>  the data in attributes. To support this sort of approach,
>  we need to be able to take a method and generate multiple
>  test cases from it.

Ok, that seems clear.

>  In your IterativeTest, we can only tell where the data
>  will come from. It's not possible, in general, to figure
>  out how many cases there will be by looking at the code.

Correct.

>  Of course, some of your examples use a constant list
>  of data points, but they would not have to be constant.

A corner case, I get it.

>  To support this approach, we need to either
>  1) allow test cases to be added at test execution time, or
>  2) allow a test to return more than one result
>  I happen to lean toward the #2, but that's a detail - you
>  can se that the two situations require different support
>  from NUnit.

Yes, I can see that.

The current approach seems to be that the test cases are created at
Load... time, and then are subsequently Run when you select a test
case to run. One issue I'm having in using IterativeTest is that the
Load time can get unreasonably long even in the case where I only want
to run a single test. (It also seems to occasionally gratuitously
reload the tests, but that's a side issue.) This seems to be the case
whether I'm using TestDriven.NET or the NUnit GUI, so I assume that
this must be a core issue, or at least both have chosen the same
implementation.

If, on the other hand, one could be lazy about test creation, and only
create the test when it's about to be run, that would make for a more
efficient system. (At least the way I'm using it.) This might also be
the case in general, it's just more obvious for IterativeTest.

So, is there some way of being indeterminate until you either try to
expand the test in the NUnit GUI, or actually run the test? Can we do
lazy evaluation of the set of tests?

It seems that you could get the name of the test from the metadata
without actually creating the test from calling BuildFrom. Would that
constitute an improvement in your opinion to the current approach?
This question is independent of 2.5 vs 3.0... although the answer
might not be.

The best answer would seem to me to be that you could have all the
test names show up, and those that are Expandable (a bool property?)
would show potential expansion. Then when you choose to expand it, A
function is called that enumerates the tests, finally, when you choose
to execute a test, then another function is called to prepare to run
the test, finally the test is run. If we could do lazy evaluation for
each of these steps, that would seem to be a huge improvement over the
current approach, at least in the cases I'm dealing with.

I'm a big fan of lazy evaluation.

>  > The Theory syntax seems to merge these two approaches... so
>  > I'm not sure whether or not it remains an unsolvable problem
>  > to merge the two approaches.
>
>  No - the Theory examples we have seen use both approaches. So
>  NUnit would need to support both of them... unless that's what
>  you mean by merge. :-)

Yes, that's what I mean. Internally this requires two different
approaches, of course.

>  > The conversation you pointed to was technically a little past
>  > my experience thus far (too internal), so while I'm very
>  > interested, I couldn't make a helpful reply.
>
>  That's cool. Jump in where you can. We're at the implementation
>  stage, where it's a question of how the internals will work, so
>  it's natural that we're getting more technical. One way to read
>  this is to ask how the availabilitiy of some NUnit feature or
>  interface would help you to implement a given extension syntax.

I'm not especially dense, but I am unfamiliar with most of the NUnit
internals. I feel most comfortable and useful at this point looking at
things from the user's point of view. By user, I mean both a general
user as well as someone writing extensions. I am not against getting
my hands dirty as might be helpful. I just recognize that there is
ramp up time involved to get to that point.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/