Re: SuiteAttribute, RowTest and IterativeTest

On Wed, Mar 12, 2008 at 5:35 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,

Hi Charlie!

Before replying to this post, read all the way to the end... :-)

>  The generation of tests isn't heavy-weight, but instantiating
>  the user object may be. Ideally, you should not be, but I
>  understand that you have to because of the design. In that
>  case, make sure you are only creating one instance of your
>  user object.

Yes, that helped a lot. It was a user issue, not a problem with
IterativeTest itself. The problem is that heavy-weight is currently in
the hands of the user at Load time with IterativeTest.

>  The + and - make me realize you're talking about the Gui
>  tree. But in this context, we have to talk about the tree
>  of test objects created by TestRunners. There's no such
>  thing as expanded/unexpanded in a tree data structure -
>  there are either subnodes or there are not.

Right. And that's part of the problem with Theory implementation in 2.5...

>  Really, as I think about it, this may be one of the main
>  places where we are not understanding one another. I'm
>  talking about the tree of tests that is built by NUnit
>  as part of it's core internals, with the help of various
>  extensions if present.

I can differentiate between the GUI and the objects...

>  The Gui tree is indeed a mapping of that tree, but only
>  a mapping. For example, we choose to show results as
>  part of the node that represents a test. But we could
>  equally well show them as subnodes in the gui tree.
>  They would have no identity in the (core) test tree,
>  but they would show up in the gui. Similarly, we can
>  choose to show every namespace as a tree node, or we
>  can leave them out.
>
>  I hope this distinction is helpful to you.

It is... But what I said applies to the internal objects as well.

>  What happened to the principle of "Simple design?" Our design
>  should IMO always be the simplest one that can accomplish the
>  functions of the system. Growing the design to accommodate
>  stories is OK to me. But that's not what I'm seeing here.

I hope that's what I did with the Theory thread on the N3 list.

>  > It isn't a feature so much, but it would potentially lead to
>  > slower Loads than might otherwise be possible.
>
>  I can't see that. The slowest part of loading is the reflection -
>  that's been measured. If you build 5000 tests from one method,
>  you should still only need to reflect on that method once. If
>  you're doing it 5000 times, that's just an implementation error.

If you invoke user code, that is no longer necessarily the case.

>  You will have a lot of overhead transmitting those 5000 tests
>  to the user and getting them all into the gui. And the user
>  will have a lot of personal overhead in looking at 5000 test
>  cases - which is why I don't think you should be doing this.
>
>  Long story short: If you design it right, it will only be
>  inefficient when you are generating large numbers of tests,
>  which I think is a bad idea. Instead, you should be executing
>  one test multiple times.

That's an interesting point.. . you're talking about the internal
objects here, not the user's test code, right?

>  > I wouldn't call it test case generation, but "test parameter
>  > generation"... "test case generation" sounds like you're
>  > generating new code or something like the MS Team test
>  > generator thing.
>
>  In an earlier thread, we agreed - or someone did - to call
>  each "call" to a method with different data a test case. That
>  may have been on the other list.

That's consistent with how I think of the GUI... I just didn't want to
have the confusion with a code generator.

>  > IterativeTest does more than that. Maybe it could be done
>  > better, but I don't know at this time what I'd override to
>  > create children of the root test... maybe you have some guidance.
>
>  My comment has nothing to do with IterativeTest. The term
>  "root test" is used internally to refer to the test that has
>  a name like MyProject.nunit. It has child tests named after
>  assemblies. Those assemblies have child tests named after
>  namespaces. Finally, those tests have tests named after
>  classes and methods. So "child test" is a very broad
>  concept.

I meant specifically children of individual tests, as per RowTest or
IterativeTest.

>  > I will look at it now.... brb.... ok b...
>
>  You really just read all that? Wow!

I think so... I can't say that I understand it all at a deep level,
but I think I read most of the relevant stuff.

>  > The only issue I see at this point is...
>  >
>  > public int CountTestCases( ITestFilter filter ) {
>  >       return test.CountTestCases( filter );
>  > }
>  >
>  > Which would only return the number of roots, not the number
>  > of roots plus children... if you did the simple meta-data
>  > load without calling the user code to generate the children.
>  > I don't know if that would cause a big problem or not, but it
>  > wouldn't return the answer you might be accustomed to.
>
>  There's only one root - but I kow what you mean. The leaf nodes
>  in the tree generally only have one test case. But since we
>  only know what a test answers when the interface is called,
>  you can say it has 5000 or 50000 if you like.

Right. But you might not know ahead of time.

>  > Ok. Maybe I'll just start another thread with a detailed
>  > starting point of what I see as a potential Theory implementation.
>
>  I think this is a futuristic (that is 3.0) thing and I'd recommend
>  doing it on the NUnitV3 list, where there are people who want to
>  talk about that. Of course, if someone on this list wants to be
>  part of it, it would be great if they joined that list.

I did.

>  In case your reaction is that you'd like us to work on Theory
>  sooner rather than later, I'll point out that I had expected
>  to be spending most of my nunit time on 3.0 by now, except that
>  folks - you included I believe - felt we should do an interim
>  2.5 release. That means you get a little bit sooner, but have
>  to wait longer for the full meal deal.

Let's decide what we want in the other thread, then once we've decided
where we want to go with things, we can discuss if some subset of that
could be usefully implemented in the 2.5 time frame.

As Yogi Berra said, "You've got to be very careful if you don't know
where you're going, because you might not get there."

I think I want to let this thread die down for a bit until we get
through the Theory thread on the Nunit3 list. Ok? I think there is
more concrete to discuss over there.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/