Re: SuiteAttribute, RowTest and IterativeTest

Hi Kelly,

> Ok, that's a fair question. By "load the tests" I mean load the
> assembly(s) containing the tests, and query just enough 
> meta-data information to get the names of the tests so that 
> the GUI tree can be populated with those names.  I would 
> assume at this point any Tests with "children" that are 
> generated later would not be shown in an expanded state, but 
> possibly in an expandable state.

That's pretty much what I mean - and NUnit does - but let me 
clarify a bit. In the tree that is built at load time. There
is nothing but the meta-information anyway. What NUnit calls
"Tests" are not the objects and methods that the user has
defined - it's just the information needed to create them.

So, taken in that light, any tests with "children" are really
not so different from other tests.

> >  > the second stage would "load up" the Theories,
> >
> >  And what does this mean?
> 
> By this I mean generating (through whatever means the 
> particular approach would need, e.g. RowTest vs. Agitator) 
> the individual sets of parameters that would end up invoking 
> the Test multiple times. These would be the "children" Tests 
> that might show up in a GUI... In the current implementation 
> as I implemented it in IterativeTest (and Andreas in RowTest) 
> this is combined with the initial MetaData Load as part of 
> the ITestCaseBuilder.BuildFrom function.

I think this could be part of the Load (BuildFrom) if the data 
is statically available as it is for RowTest. I feel that you
are missing an intrinsic difference between your IterativeTest
and RowTest: it is not possible to get the data for an
IterativeTest without executing user code.

To me, that says "Run time", i.e. the time when we create the
user's objects and execute the code.

> What I'm suggesting specifically is breaking BuildFrom into 
> two functions... one could be BuildNameFrom (the first step) 
> and the other would be BuildFrom.... which would be 
> essentially equivalent to what it is now. In any case, as you 
> suggested, there probably should be a way to generate 
> "children" of a Test/Theory after the initial Load in some way.

What would a BuildNameFrom look like? Who would call it? When
would it call it? Where would that caller get the data?

We're talking about architectural changes here - and for
an "interim" release no less! Even for the 3.0 release,
I would want to have a strong rationale to change the
underlying architecture.

Currently, the design of this part of NUnit is simple...

Various builders run at load time and create whatever 
test types they desire. In order to control run time
behavior, they put that behavior has to be put into
the type of test that is created.

So far, this has worked for a wide range of test types
and it's hard to see why it won't work for the test
types you are describing.

Maybe I'm still not understanding what a "BuildNameFrom"
would do. Let me ask it this way: If we didn't have that
third phase, what features would we be missing?


> Maybe I misunderstand Agitator... 

Agitator is a product and may have added lots of features
since I first became aware of it. At that time, it's main
feature was that it introduced "perturbances" in java code
in order to see if the tests failed. If they didn't, you
needed another test.

I suggest we not refer to any elephants in this disussion
without saying whether we mean the trunk, the ears, the
feet, the side, etc.

> but what I mean is 
> generating sets of either totally random or semi-intelligent 
> random parameters to the tests. 

OK, I'll call that "test case generation." Right now, of course,
we're not talking about that but about how to feed the test
cases to the methods once they are generated.

> >  > This should not be in the Load stage, so it would have to be  > 
> > done later, right?
> >
> >  I don't know. I'm really not understanding your proposal.
> 
> Like you said, you don't want to call user code during the 
> first stage Load. Building "children" tests COULD, but 
> doesn't always, involve calling user code.

Yes... In general, all the tests in the tree are child tests
except the top one. We build them based on metadata.
 
> >  I have a suggestion. This talk is getting into specific  
> > implementation details, which is appropriate for this list.
> >  OTOH, you have said several times that you aren't familiar 
>  with how 
> > NUnit currently loads tests. I think you need  to get familiar with 
> > the internals in order to fruitfully  discuss how we are going to 
> > change them.
> 
> Can you point me to the classes to look at? I'm making my 
> comments based upon my observations of how it works and what 
> you've said before, as well as looking in the debugger at how 
> my Addin code gets called.

If you looked at SimpleTestRunner and everything it calls directly
or indirectly you'd get a pretty clear understading of this part
of NUnit. It's a fair amount of code, but significantly less than
the whole.

> >  Alternatively, we could dial the detail back a little  and 
> talk about 
> > what you would like to be able to do  eithe as a 
> test-writer or as an 
> > extension-writer.
> 
> Or perhaps we could just discuss how we would want Theory to 
> work, and I think things would perhaps naturally fall out of 
> that discussion.

Yes, that's basically what I meant. You, as a test writer, would
like to be able to define a "theory" ... and so on...

Charlie
 
> >  I'm OK either way.
> 
> Me too.
> 
> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/