Re: SuiteAttribute, RowTest and IterativeTest

On Wed, Mar 12, 2008 at 2:05 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>
>  > Ok, that's a fair question. By "load the tests" I mean load the
>  > assembly(s) containing the tests, and query just enough
>  > meta-data information to get the names of the tests so that
>  > the GUI tree can be populated with those names.  I would
>  > assume at this point any Tests with "children" that are
>  > generated later would not be shown in an expanded state, but
>  > possibly in an expandable state.
>
>  That's pretty much what I mean - and NUnit does - but let me
>  clarify a bit. In the tree that is built at load time. There
>  is nothing but the meta-information anyway. What NUnit calls
>  "Tests" are not the objects and methods that the user has
>  defined - it's just the information needed to create them.
>
>  So, taken in that light, any tests with "children" are really
>  not so different from other tests.

In my current implementation of IterativeTest, it calls user code to
generate the list of tests. So, it is not as light weight as you're
describing here. That is likely a problem with IterativeTest (being
that it started life as a copy of RowTest), but is there currently an
alternative approach that would delay building the list of tests until
later?

>  > >  > the second stage would "load up" the Theories,
>  > >
>  > >  And what does this mean?
>  >
>  > By this I mean generating (through whatever means the
>  > particular approach would need, e.g. RowTest vs. Agitator)
>  > the individual sets of parameters that would end up invoking
>  > the Test multiple times. These would be the "children" Tests
>  > that might show up in a GUI... In the current implementation
>  > as I implemented it in IterativeTest (and Andreas in RowTest)
>  > this is combined with the initial MetaData Load as part of
>  > the ITestCaseBuilder.BuildFrom function.
>
>  I think this could be part of the Load (BuildFrom) if the data
>  is statically available as it is for RowTest. I feel that you
>  are missing an intrinsic difference between your IterativeTest
>  and RowTest: it is not possible to get the data for an
>  IterativeTest without executing user code.

Oh, I understand that.

>  To me, that says "Run time", i.e. the time when we create the
>  user's objects and execute the code.

Ok, but is there currently an extension point that would allow me to
build more tests later?

>  > What I'm suggesting specifically is breaking BuildFrom into
>  > two functions... one could be BuildNameFrom (the first step)
>  > and the other would be BuildFrom.... which would be
>  > essentially equivalent to what it is now. In any case, as you
>  > suggested, there probably should be a way to generate
>  > "children" of a Test/Theory after the initial Load in some way.
>
>  What would a BuildNameFrom look like? Who would call it? When
>  would it call it? Where would that caller get the data?

In the case of  IterativeTest, BuildNameFrom would just return the
name of the test... so for example:

==============
    private static ArrayList list;

    public IEnumerable FileList()
    {
      if (list == null)
      {
        list = new ArrayList();
        list.Add(@"C:\test\test1.xml");
        list.Add(@"C:\test\test2.xml");
        list.Add(@"C:\test\test3.xml");
      }
      return list;
    }

    [IterativeTest("FileList")]
    public void FileNameTest(object current)
    {
      string curStr = (string) current;
      Assert.IsTrue(curStr.EndsWith("xml"));
    }
===========

Would result in the tree looking like this:
+FileNameTest

===========

After the second phase was run, as you either expanded the tree, or as
you ran the tests, you could get:

-FileNameTest
 FileNameTest("C:\test\test1.xml")
 FileNameTest("C:\test\test2.xml")
 FileNameTest("C:\test\test3.xml")

===========

>  We're talking about architectural changes here - and for
>  an "interim" release no less! Even for the 3.0 release,
>  I would want to have a strong rationale to change the
>  underlying architecture.

I have no bias about when this might occur.

>  Currently, the design of this part of NUnit is simple...

Ok. Keeping things simple is good. Too simple, though, and it's
potentially limiting.

>  Various builders run at load time and create whatever
>  test types they desire. In order to control run time
>  behavior, they put that behavior has to be put into
>  the type of test that is created.
>
>  So far, this has worked for a wide range of test types
>  and it's hard to see why it won't work for the test
>  types you are describing.
>
>  Maybe I'm still not understanding what a "BuildNameFrom"
>  would do. Let me ask it this way: If we didn't have that
>  third phase, what features would we be missing?

It isn't a feature so much, but it would potentially lead to slower
Loads than might otherwise be possible.

>  > Maybe I misunderstand Agitator...
>
>  Agitator is a product and may have added lots of features
>  since I first became aware of it. At that time, it's main
>  feature was that it introduced "perturbances" in java code
>  in order to see if the tests failed. If they didn't, you
>  needed another test.
>
>  I suggest we not refer to any elephants in this disussion
>  without saying whether we mean the trunk, the ears, the
>  feet, the side, etc.

Ok.

>  > but what I mean is
>  > generating sets of either totally random or semi-intelligent
>  > random parameters to the tests.
>
>  OK, I'll call that "test case generation." Right now, of course,
>  we're not talking about that but about how to feed the test
>  cases to the methods once they are generated.

I wouldn't call it test case generation, but "test parameter
generation"... "test case generation" sounds like you're generating
new code or something like the MS Team test generator thing.

>  > >  > This should not be in the Load stage, so it would have to be  >
>  > > done later, right?
>  > >
>  > >  I don't know. I'm really not understanding your proposal.
>  >
>  > Like you said, you don't want to call user code during the
>  > first stage Load. Building "children" tests COULD, but
>  > doesn't always, involve calling user code.
>
>  Yes... In general, all the tests in the tree are child tests
>  except the top one. We build them based on metadata.

IterativeTest does more than that. Maybe it could be done better, but
I don't know at this time what I'd override to create children of the
root test... maybe you have some guidance.

>  > >  I have a suggestion. This talk is getting into specific
>  > > implementation details, which is appropriate for this list.
>  > >  OTOH, you have said several times that you aren't familiar
>  >  with how
>  > > NUnit currently loads tests. I think you need  to get familiar with
>  > > the internals in order to fruitfully  discuss how we are going to
>  > > change them.
>  >
>  > Can you point me to the classes to look at? I'm making my
>  > comments based upon my observations of how it works and what
>  > you've said before, as well as looking in the debugger at how
>  > my Addin code gets called.
>
>  If you looked at SimpleTestRunner and everything it calls directly
>  or indirectly you'd get a pretty clear understading of this part
>  of NUnit. It's a fair amount of code, but significantly less than
>  the whole.

I will look at it now.... brb.... ok b...

The only issue I see at this point is...

public int CountTestCases( ITestFilter filter )
{
	return test.CountTestCases( filter );
}

Which would only return the number of roots, not the number of roots
plus children... if you did the simple meta-data load without calling
the user code to generate the children. I don't know if that would
cause a big problem or not, but it wouldn't return the answer you
might be accustomed to.

>  > >  Alternatively, we could dial the detail back a little  and
>  > talk about
>  > > what you would like to be able to do  eithe as a
>  > test-writer or as an
>  > > extension-writer.
>  >
>  > Or perhaps we could just discuss how we would want Theory to
>  > work, and I think things would perhaps naturally fall out of
>  > that discussion.
>
>  Yes, that's basically what I meant. You, as a test writer, would
>  like to be able to define a "theory" ... and so on...

Ok. Maybe I'll just start another thread with a detailed starting
point of what I see as a potential Theory implementation.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/