Re: SuiteAttribute, RowTest and IterativeTest

Hi Kelly, 

> >  That's pretty much what I mean - and NUnit does - but let 
> me  clarify 
> > a bit. In the tree that is built at load time. There  is 
> nothing but 
> > the meta-information anyway. What NUnit calls  "Tests" are not the 
> > objects and methods that the user has  defined - it's just the 
> > information needed to create them.
> >
> >  So, taken in that light, any tests with "children" are 
> really  not so 
> > different from other tests.
> 
> In my current implementation of IterativeTest, it calls user 
> code to generate the list of tests. So, it is not as light 
> weight as you're describing here. That is likely a problem 
> with IterativeTest (being that it started life as a copy of 
> RowTest), but is there currently an alternative approach that 
> would delay building the list of tests until later?

The generation of tests isn't heavy-weight, but instantiating
the user object may be. Ideally, you should not be, but I 
understand that you have to because of the design. In that
case, make sure you are only creating one instance of your
user object.

An alternative would be to define a test type that creates
sub-tests when it is run.

Another would be to have your test invoke the user method
multiple times and return a bunch of results - but NUnit
will only look at the top level result as it currently works.

 
> >  To me, that says "Run time", i.e. the time when we create 
> the  user's 
> > objects and execute the code.
> 
> Ok, but is there currently an extension point that would 
> allow me to build more tests later?

No. Why would you need an extension point? Your master test
could build them.
 
> >  > What I'm suggesting specifically is breaking BuildFrom 
> into  > two 
> > functions... one could be BuildNameFrom (the first step)  > and the 
> > other would be BuildFrom.... which would be  > essentially 
> equivalent 
> > to what it is now. In any case, as you  > suggested, there probably 
> > should be a way to generate  > "children" of a Test/Theory 
> after the 
> > initial Load in some way.
> >
> >  What would a BuildNameFrom look like? Who would call it? 
> When  would 
> > it call it? Where would that caller get the data?
> 
> In the case of  IterativeTest, BuildNameFrom would just 
> return the name of the test... so for example:
> 
> ==============
>     private static ArrayList list;
> 
>     public IEnumerable FileList()
>     {
>       if (list == null)
>       {
>         list = new ArrayList();
>         list.Add(@"C:\test\test1.xml");
>         list.Add(@"C:\test\test2.xml");
>         list.Add(@"C:\test\test3.xml");
>       }
>       return list;
>     }
> 
>     [IterativeTest("FileList")]
>     public void FileNameTest(object current)
>     {
>       string curStr = (string) current;
>       Assert.IsTrue(curStr.EndsWith("xml"));
>     }
> ===========
> 
> Would result in the tree looking like this:
> +FileNameTest
> 
> ===========

The + and - make me realize you're talking about the Gui
tree. But in this context, we have to talk about the tree
of test objects created by TestRunners. There's no such
thing as expanded/unexpanded in a tree data structure - 
there are either subnodes or there are not.

Really, as I think about it, this may be one of the main
places where we are not understanding one another. I'm
talking about the tree of tests that is built by NUnit
as part of it's core internals, with the help of various
extensions if present.

The Gui tree is indeed a mapping of that tree, but only
a mapping. For example, we choose to show results as
part of the node that represents a test. But we could
equally well show them as subnodes in the gui tree.
They would have no identity in the (core) test tree,
but they would show up in the gui. Similarly, we can
choose to show every namespace as a tree node, or we
can leave them out.

I hope this distinction is helpful to you.

> After the second phase was run, as you either expanded the 
> tree, or as you ran the tests, you could get:
> 
> -FileNameTest
>  FileNameTest("C:\test\test1.xml")
>  FileNameTest("C:\test\test2.xml")
>  FileNameTest("C:\test\test3.xml")
> 
> ===========
> 
> >  We're talking about architectural changes here - and for  an 
> > "interim" release no less! Even for the 3.0 release,  I 
> would want to 
> > have a strong rationale to change the  underlying architecture.
> 
> I have no bias about when this might occur.

I wasn't supposing bias, but since we generally don't discuss
3.0 stuff on this list...

> >  Currently, the design of this part of NUnit is simple...
> 
> Ok. Keeping things simple is good. Too simple, though, and 
> it's potentially limiting.

What happened to the principle of "Simple design?" Our design
should IMO always be the simplest one that can accomplish the
functions of the system. Growing the design to accommodate 
stories is OK to me. But that's not what I'm seeing here.

> >  Various builders run at load time and create whatever  test types 
> > they desire. In order to control run time  behavior, they put that 
> > behavior has to be put into  the type of test that is created.
> >
> >  So far, this has worked for a wide range of test types  
> and it's hard 
> > to see why it won't work for the test  types you are describing.
> >
> >  Maybe I'm still not understanding what a "BuildNameFrom"
> >  would do. Let me ask it this way: If we didn't have that third 
> > phase, what features would we be missing?
> 
> It isn't a feature so much, but it would potentially lead to 
> slower Loads than might otherwise be possible.

I can't see that. The slowest part of loading is the reflection - 
that's been measured. If you build 5000 tests from one method,
you should still only need to reflect on that method once. If
you're doing it 5000 times, that's just an implementation error.

You will have a lot of overhead transmitting those 5000 tests
to the user and getting them all into the gui. And the user
will have a lot of personal overhead in looking at 5000 test
cases - which is why I don't think you should be doing this.

Long story short: If you design it right, it will only be
inefficient when you are generating large numbers of tests,
which I think is a bad idea. Instead, you should be executing
one test multiple times.

> >  OK, I'll call that "test case generation." Right now, of course,  
> > we're not talking about that but about how to feed the test 
>  cases to 
> > the methods once they are generated.
> 
> I wouldn't call it test case generation, but "test parameter 
> generation"... "test case generation" sounds like you're 
> generating new code or something like the MS Team test 
> generator thing.

In an earlier thread, we agreed - or someone did - to call
each "call" to a method with different data a test case. That
may have been on the other list.

> >  Yes... In general, all the tests in the tree are child 
> tests  except 
> > the top one. We build them based on metadata.
> 
> IterativeTest does more than that. Maybe it could be done 
> better, but I don't know at this time what I'd override to 
> create children of the root test... maybe you have some guidance.

My comment has nothing to do with IterativeTest. The term 
"root test" is used internally to refer to the test that has
a name like MyProject.nunit. It has child tests named after
assemblies. Those assemblies have child tests named after
namespaces. Finally, those tests have tests named after
classes and methods. So "child test" is a very broad
concept.

> >  > >  I have a suggestion. This talk is getting into specific  > > 
> > implementation details, which is appropriate for this list.
> >  > >  OTOH, you have said several times that you aren't 
> familiar  >  
> > with how  > > NUnit currently loads tests. I think you need  to get 
> > familiar with  > > the internals in order to fruitfully  
> discuss how 
> > we are going to  > > change them.
> >  >
> >  > Can you point me to the classes to look at? I'm making my  > 
> > comments based upon my observations of how it works and 
> what  > you've 
> > said before, as well as looking in the debugger at how  > my Addin 
> > code gets called.
> >
> >  If you looked at SimpleTestRunner and everything it calls 
> directly  
> > or indirectly you'd get a pretty clear understading of this 
> part  of 
> > NUnit. It's a fair amount of code, but significantly less than  the 
> > whole.
> 
> I will look at it now.... brb.... ok b...

You really just read all that? Wow!

> The only issue I see at this point is...
> 
> public int CountTestCases( ITestFilter filter ) {
> 	return test.CountTestCases( filter );
> }
> 
> Which would only return the number of roots, not the number 
> of roots plus children... if you did the simple meta-data 
> load without calling the user code to generate the children. 
> I don't know if that would cause a big problem or not, but it 
> wouldn't return the answer you might be accustomed to.

There's only one root - but I kow what you mean. The leaf nodes
in the tree generally only have one test case. But since we
only know what a test answers when the interface is called,
you can say it has 5000 or 50000 if you like.
 
> >  > >  Alternatively, we could dial the detail back a little  and  > 
> > talk about  > > what you would like to be able to do  eithe as a  > 
> > test-writer or as an  > > extension-writer.
> >  >
> >  > Or perhaps we could just discuss how we would want Theory to  > 
> > work, and I think things would perhaps naturally fall out 
> of  > that 
> > discussion.
> >
> >  Yes, that's basically what I meant. You, as a test writer, would  
> > like to be able to define a "theory" ... and so on...
> 
> Ok. Maybe I'll just start another thread with a detailed 
> starting point of what I see as a potential Theory implementation.

I think this is a futuristic (that is 3.0) thing and I'd recommend
doing it on the NUnitV3 list, where there are people who want to
talk about that. Of course, if someone on this list wants to be
part of it, it would be great if they joined that list.

In case your reaction is that you'd like us to work on Theory
sooner rather than later, I'll point out that I had expected
to be spending most of my nunit time on 3.0 by now, except that 
folks - you included I believe - felt we should do an interim 
2.5 release. That means you get a little bit sooner, but have
to wait longer for the full meal deal.

Charlie

> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/