Re: Data Driven Addins

On Thu, Feb 14, 2008 at 11:29 AM, Charlie Poole
<[email protected]> wrote:

> Another issue to bear in mind is that there's more to it than
> the underlying implementation similarities. What I mean is...
> even if it's /possible/ to solve very different sorts of
> problems with a common implementation, that commonality may
> not be obvious to users, who may require a different syntax
> so they know what they are doing.

And, I suppose that some users might not want a big solution when a
smaller one would do.

> NUnit itself has not paid a lot of attention to this issue
> in the past: everything is a TestFixture and you have to
> bend your problem to fit that construct. I'd like to start
> thinking a bit more in terms of the problems to be solved,
> and give the user solutions that match the problem.

I agree, and that's what the conversation has been about thus far.

> Examples of what I see as different problems are:
>
> 1) I want a shorthand way to specify the same test method
> with several different arguments, so I don't have to
> type it many times.

So here, you could have a RowTest type approach, or an IterativeTest
type approach, or a Theory approach that combines the two into one
syntax.

> 2) I want to run the same test in different contexts - with
> a different database, under a different culture, etc.

This starts to feel like Gary's solutions, where the Fixture is run
multiple times within different contexts...

> 3) I want to be able to supply an open-ended list of
> parameters to a test and run all of them as if they
> were separate tests.

I'm not sure exactly what you're getting at here... do you mean a
func(a,b,c,...) parameter list? If so, I'm not sure how that's
helpful.

> 4) I want to test multiple implementations of the same
> interface using a standard set of tests.

Yes, that is where Gary's approach is good.

> 5) I want to specify general statements about how a
> method should work and provide data to test those general
> statements.

This is a bit vague to me... can you come up with an example of what
you might be thinking?

> 6) I want to specify general statements about how a method
> should work and have the framework generate data sets to
> test those statements.

I think this is the Pex approach, or the Agitator approach, right? I
guess this is part of the same general set of problems, but unlike the
other elements in this list, I think to do that in the best way
requires that you analyze the source code. Starts to sound like a
pretty big job. And then how do you repeat a failed test run?

> Add your own...

I think that about covers everything we've been talking about...

> Many of the above can be solved by a data-driven implementation,
> but I'm not convinced that it serves the user best to make
> them all /look like/ the same thing.

NUnit's style is to make one thing do lots. Assert does lots.
TestFixture does lots. If we added Assume, that would do just about
the same amount of work as Assert. I would try to persuade the group
not to create a bunch of different new things, but to see if one new
thing would cover most or all of what we want. Maybe we could even do
some of this stuff by just adding new parameters to the Test and
TextFixture attributes, then we don't have to make anything "new" at
all. That's the style of NUnit as I have seen it. Changing the style
of something as successful as NUnit seems risky to me.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/