Re: Roadmap Updated

On Tue, Mar 11, 2008 at 12:58 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>  > >  Yes.. and in most cases, user code in the test assembly  does not
>  > > participate in loading tests. The only exception  I can think of is
>  > > the Suite property, which is required  to be static.
>  >
>  > When we're talking about RowTest, IterativeTest or Theory
>  > type approaches, this doesn't hold, and user code can and
>  > does get involved.
>
>  That's true for your implementation of IterativeTest. It's not true
>  for RowTest and I can't see why it would be true for a particular
>  implementation of a "Theory-based" approach (the latter can mean
>  so many things, I don't see how we can talk about it yet).

I think we can talk about it... :-)  How else will we discover what it
should be?

>  There are dangers involved when the test object (the user class
>  with TestFixtureAttribute on it) is created at load time. We used
>  to do that and it created many bugs, especially when users - bless
>  their hearts - create constructors with lots of side effects. Of
>  course, they shouldn't do that, but NUnit still has to deal with
>  it and mitigate the damage. That's why we stopped creating these
>  objects at load time.

I can appreciate the difficulties involved with that. I wonder if
Lambda expressions might offer a solution to this. They are not
evaluated until needed, which in this case would be later. You could
pass in a Lambda expression creating the set of Tests to theorize
about, but it would not be executed at Load time, but only when the
expression is actually evaluated. It's a thought anyway.

>  In any cae, I believe the way to deal with this issue is to come
>  up with some use cases - stories - that drive us to implementing
>  the sort of solution you envision. All else is theory. :-)

Ah, then the practical approach it is to be eh? Well then, let me add
my story...

I have a set of XML files containing data that need to be checked
against a set of rules. I need to have multiple Tests or Theories run
against these multiple files (so it's a two dimensional problem) to
check them for internal consistency.

>  We're getting a little confused here because we are actually
>  talking about two things simultaneously. The first is how to
>  make your extension perform well. The second is how NUnit could
>  be changed to make it easier to write such extensions. We have
>  to get the first one write before dealing with the second.

I think I have my problem fixed for now, so let's move on to NUnit.

>  > >  Here's what I mean by multiple result sets... Let's say  I have a
>  > > method that tests multiplication. I want to  run it with
>  > 2x2=4 0x7=0
>  > > 90x1=90 as cases.
>  >
>  > Which could come from some kind of autogenerator, in the Theory case.
>  >
>  > >  I can consider this as three tests, listed in the gui  tree as
>  > > Multiply(2,2), Multiply(0,7) and Multiply(90,1).
>  > >  That's how RowTest would do it, for example.
>  >
>  > Sure, that's one approach.
>  >
>  > >  On the other hand, we can think about this as a single  TestMethod
>  > > producing multiple results. The test tree  would list a single
>  > > Multiply test. After execution,  the results would be placed in the
>  > > tree under the test.
>  > >  They would look pretty much like the tests in the first
>  > approach -
>  > > it's just a matter of how we think about it.
>  >
>  > That sounds cleaner.
>
>  I think we need both approaches for different user needs.

I guess it's back to the stories then.

>  > >  One advantage of this is that a TestResult is a  very lightweight,
>  > > serializable object. It has no  existence in the tree of
>  > tests created
>  > > by the loader,  which is BTW different from the tree in the
>  > Gui, so
>  > > it doesn't have to be known in advance.
>  > >
>  > >  So that's an approach I plan to follow up on.
>  >
>  > I think that's a good plan. There remains what to do in the
>  > GUI tree before the test has run... do you add the + only
>  > after the tests have run? How to differentiate those sorts of
>  > tests from those who never have children? Just details, of course.
>
>  Indeed. The gui tree now contains both tests and results. The
>  results post as the test runs. Since each test curretly only
>  has one result, they both live in one node of the tree. If
>  some sort of test could have multiple results, then the tree
>  would need to do something different with those results, like
>  creating new nodes. It's straightforward enough, but with lots
>  of corner cases.

Yeah, sounds like something that you would want to have lots of tests
for... ;-)

>  > >  1) By that time, we have already done most of the work,  unless we
>  > > are talking about very large sets of data.
>  >
>  > I was... In my IEnumerator, I was loading the XML files I was testing.
>  > Now, I think I'll restructure it so that it only makes the
>  > list of files, and loads them later, as necessary. My bad.
>
>  I think there is more than efficiency involved here - it's also a
>  case of correctness for any enumerators that return different
>  sets of values each time the object is instantiated. Think of
>  some sort of data capture application.

That's an interesting user story... however, I'm thinking that one of
the things that you want out of a testing framework is repeatability
and you would certainly lose that in this case. Of course, it's not
nice to be judgmental about how people use tools. They are just tools
anyway.

>  > I'm hoping that this is something we want to change in the
>  > 3.0 time frame. I think extensibility should be targeted at
>  > the general user.
>
>  I disagree with this. The general user is not going to be inventing
>  things like IterativeTest or Theory.

That's interesting. Then why go to the effort of using Mono.Addins if
not to make this type of thing easier? If you're saying "general" user
in the sense of people writing tests on a day to day basis, then I
think I see your point. However, what would change? Would it make us
less likely to provide good documentation for extension points?

>  There is certainly a need for a user to easily "extend" what is
>  happening in a test and NUnit needs to meet that need. But that's
>  entirely different from writing plugins for NUnit.

What's an example of what you're talking about here?

>  I think we would mess up the plugin architecture if we attempted
>  to make it suitable for general use. OTOH, we could easily have
>  a plugin - something like the addin Roy wrote - that makes it
>  easy for users to do test-level extensions through inheritance.

Are you referring to XtUnit? What sorts of test-level extension might
be able to be done in a system such as you are describing?

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/