Re: Roadmap Updated

Hi Kelly, 

> >  That's true for your implementation of IterativeTest. It's 
> not true  
> > for RowTest and I can't see why it would be true for a particular  
> > implementation of a "Theory-based" approach (the latter can 
> mean  so 
> > many things, I don't see how we can talk about it yet).
> 
> I think we can talk about it... :-)  How else will we 
> discover what it should be?

Then I'm in favor of the first talks being about definitions.
 
> >  There are dangers involved when the test object (the user 
> class  with 
> > TestFixtureAttribute on it) is created at load time. We used  to do 
> > that and it created many bugs, especially when users - bless  their 
> > hearts - create constructors with lots of side effects. Of  course, 
> > they shouldn't do that, but NUnit still has to deal with  it and 
> > mitigate the damage. That's why we stopped creating these  
> objects at 
> > load time.
> 
> I can appreciate the difficulties involved with that. I 
> wonder if Lambda expressions might offer a solution to this. 
> They are not evaluated until needed, which in this case would 
> be later. You could pass in a Lambda expression creating the 
> set of Tests to theorize about, but it would not be executed 
> at Load time, but only when the expression is actually 
> evaluated. It's a thought anyway.

There are lots of simpler ways than that. Otherwise, we couldn't
be doing it now - even under .NET 1.0.

Lambda expressions could provide users a way to create arbitrary
code, which NUnit would then execute inside itself - but why
would we want that?

As I said, NUnit /already/ does what you want provided your
extension - which is not "user code" by the way - doesn't
defeat it by creating things in a non-lazy way or making
it appear to NUnit that things need to be repeatedly created.

> >  In any cae, I believe the way to deal with this issue is 
> to come  up 
> > with some use cases - stories - that drive us to implementing  the 
> > sort of solution you envision. All else is theory. :-)
> 
> Ah, then the practical approach it is to be eh? Well then, 
> let me add my story...
> 
> I have a set of XML files containing data that need to be 
> checked against a set of rules. I need to have multiple Tests 
> or Theories run against these multiple files (so it's a two 
> dimensional problem) to check them for internal consistency.

So, is the set of files a static list or is it somehow
figured out dynamically?

> >  We're getting a little confused here because we are 
> actually  talking 
> > about two things simultaneously. The first is how to  make your 
> > extension perform well. The second is how NUnit could  be 
> changed to 
> > make it easier to write such extensions. We have  to get 
> the first one 
> > write before dealing with the second.
> 
> I think I have my problem fixed for now, so let's move on to NUnit.
> 
> >  > >  Here's what I mean by multiple result sets... Let's 
> say  I have 
> > a  > > method that tests multiplication. I want to  run it with  > 
> > 2x2=4 0x7=0  > > 90x1=90 as cases.
> >  >
> >  > Which could come from some kind of autogenerator, in the 
> Theory case.
> >  >
> >  > >  I can consider this as three tests, listed in the gui 
>  tree as  
> > > > Multiply(2,2), Multiply(0,7) and Multiply(90,1).
> >  > >  That's how RowTest would do it, for example.
> >  >
> >  > Sure, that's one approach.
> >  >
> >  > >  On the other hand, we can think about this as a single  
> > TestMethod  > > producing multiple results. The test tree  
> would list 
> > a single  > > Multiply test. After execution,  the results would be 
> > placed in the  > > tree under the test.
> >  > >  They would look pretty much like the tests in the first  > 
> > approach -  > > it's just a matter of how we think about it.
> >  >
> >  > That sounds cleaner.
> >
> >  I think we need both approaches for different user needs.
> 
> I guess it's back to the stories then.

I may be misreading you, but I get the idea you want to pick a best
way to implement data-provision. For design, that makes sense. But
for requirements, it doesn't. I think there is a requirement for
static data providers and dynamic data providers.

> >  > >  One advantage of this is that a TestResult is a  very 
> > lightweight,  > > serializable object. It has no  existence in the 
> > tree of  > tests created  > > by the loader,  which is BTW 
> different 
> > from the tree in the  > Gui, so  > > it doesn't have to be known in 
> > advance.
> >  > >
> >  > >  So that's an approach I plan to follow up on.
> >  >
> >  > I think that's a good plan. There remains what to do in 
> the  > GUI 
> > tree before the test has run... do you add the + only  > after the 
> > tests have run? How to differentiate those sorts of  > tests from 
> > those who never have children? Just details, of course.
> >
> >  Indeed. The gui tree now contains both tests and results. The  
> > results post as the test runs. Since each test curretly 
> only  has one 
> > result, they both live in one node of the tree. If  some 
> sort of test 
> > could have multiple results, then the tree  would need to 
> do something 
> > different with those results, like  creating new nodes. It's 
> > straightforward enough, but with lots  of corner cases.
> 
> Yeah, sounds like something that you would want to have lots 
> of tests for... ;-)

Yup.

> >  > >  1) By that time, we have already done most of the 
> work,  unless 
> > we  > > are talking about very large sets of data.
> >  >
> >  > I was... In my IEnumerator, I was loading the XML files 
> I was testing.
> >  > Now, I think I'll restructure it so that it only makes 
> the  > list 
> > of files, and loads them later, as necessary. My bad.
> >
> >  I think there is more than efficiency involved here - it's also a  
> > case of correctness for any enumerators that return 
> different  sets of 
> > values each time the object is instantiated. Think of  some sort of 
> > data capture application.
> 
> That's an interesting user story... however, I'm thinking 
> that one of the things that you want out of a testing 
> framework is repeatability and you would certainly lose that 
> in this case. Of course, it's not nice to be judgmental about 
> how people use tools. They are just tools anyway.

I want both. I'd like to have a different set of random values
use every time I run the test but the same values used when
I need to re-run it under the debugger. I'm a demanding user.

> >  > I'm hoping that this is something we want to change in 
> the  > 3.0 
> > time frame. I think extensibility should be targeted at  > 
> the general 
> > user.
> >
> >  I disagree with this. The general user is not going to be 
> inventing  
> > things like IterativeTest or Theory.
> 
> That's interesting. Then why go to the effort of using 
> Mono.Addins if not to make this type of thing easier? If 
> you're saying "general" user in the sense of people writing 
> tests on a day to day basis, then I think I see your point. 
> However, what would change? Would it make us less likely to 
> provide good documentation for extension points?

By "general user" I mean someone who writes tests and doesn't
particularly care how NUnit works. When he needs to use a
new feature, he goes to some local guru and asks about it.
That guru might very well be writing extensions - just like
he might be creating user controls for others in his group
to use. He's the target for the addin architecture.

The new architecture will have more extension points and
a better design - for example extensions on extensions - 
that makes it easier to do things that are hard now, and
possible to do things that are impossible. But it isn't
intended to lower the bar for changing how NUnit works.

> >  There is certainly a need for a user to easily "extend" what is  
> > happening in a test and NUnit needs to meet that need. But that's  
> > entirely different from writing plugins for NUnit.
> 
> What's an example of what you're talking about here?
> 
> >  I think we would mess up the plugin architecture if we 
> attempted  to 
> > make it suitable for general use. OTOH, we could easily 
> have  a plugin 
> > - something like the addin Roy wrote - that makes it  easy 
> for users 
> > to do test-level extensions through inheritance.
> 
> Are you referring to XtUnit? What sorts of test-level 
> extension might be able to be done in a system such as you 
> are describing?

Yes... take a look at how he uses a special attribute as
something a user can inherit from.

Charlie

> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/