Re: SuiteAttribute, RowTest and IterativeTest

On Tue, Mar 11, 2008 at 4:10 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>
>  > >  True enough. My real preference is not to have any user objects
>  > > floating around at load time. :-)
>  >
>  > Ok, then to do Theory
>
>  Whatever that is. :-)

Yes, that's an interesting question, eh?

>  > we would have to have some kind of two
>
>  three?

Yes, possibly.

>  > stage approach. The first Load stage would load the tests,
>
>  We're thrashing here, so I need to ask some "obvious" questions.
>  First one: what do you mean by "load the tests?"

Ok, that's a fair question. By "load the tests" I mean load the
assembly(s) containing the tests, and query just enough meta-data
information to get the names of the tests so that the GUI tree can be
populated with those names.  I would assume at this point any Tests
with "children" that are generated later would not be shown in an
expanded state, but possibly in an expandable state.

>  > the second stage would "load up" the Theories,
>
>  And what does this mean?

By this I mean generating (through whatever means the particular
approach would need, e.g. RowTest vs. Agitator) the individual sets of
parameters that would end up invoking the Test multiple times. These
would be the "children" Tests that might show up in a GUI... In the
current implementation as I implemented it in IterativeTest (and
Andreas in RowTest) this is combined with the initial MetaData Load as
part of the ITestCaseBuilder.BuildFrom function.

What I'm suggesting specifically is breaking BuildFrom into two
functions... one could be BuildNameFrom (the first step) and the other
would be BuildFrom.... which would be essentially equivalent to what
it is now. In any case, as you suggested, there probably should be a
way to generate "children" of a Test/Theory after the initial Load in
some way.

>  > then the third
>  > stage would execute the tests... now the second and third
>  > stages could be interleaved, if that's helpful, but I don't
>  > see a way to get around that, and support Theory as I understand it.
>  >
>  > There has to be a stage, for example, where you could
>  > generate random or semi-random values to pass into tests to
>  > do Agitator type things.
>
>  OK - that's another term to define. In my understanding,
>  "agitator type things" would mean modifying the code to see
>  if it breaks. But I'm pretty sure that isn't what you mean.

Maybe I misunderstand Agitator... but what I mean is generating sets
of either totally random or semi-intelligent random parameters to the
tests. In Theory-type tests, you have parameters. The values of these
parameters can be generated in multiple ways. They could come from Row
attributes, be programmatically generated, or be generated after a
rather simplistic look at the code. As an example of how this could
work, suppose we have the following Theory...

[Theory]
public void NumbersLessThanPiAreHandledDifferently(double d)
{
  Passes.If(d.Is.EqualTo(3.141592).Within(0.001); // Meaning the test
passes for values close to Pi. I used a different word before...

  if (d > 3.141592)
  {
    Assert.That(function(d), Is.EqualTo(14.0).Within(0.001));
  }
  else
  {
    Assert.That(function(d), Is.EqualTo(15.0).Within(0.001));
  }
}

A code analysis of this Theory might lead a set of potentially
interesting values for d that might look something like: NaN,
-MaxDouble, +MaxDouble, 0.0, 1.0, 4.0, 3.141592, 3.1416 and 3.1414.
Those numbers could then be tested against the Theory to see if they
all pass. That's what I mean by a code analysis based "Agitator"
approach. So, what I mean by "Agitator" is a semi-inteligent method
for generating semi-random parameters to feed into Theories for their
validation. If I'm misunderstanding the original "Agitator" then maybe
someone can straighten me out, but I think this is how it was referred
to in one of the Theory papers I read.

>  > This should not be in the Load stage, so it would have to be
>  > done later, right?
>
>  I don't know. I'm really not understanding your proposal.

Like you said, you don't want to call user code during the first stage
Load. Building "children" tests COULD, but doesn't always, involve
calling user code.

>  I have a suggestion. This talk is getting into specific
>  implementation details, which is appropriate for this list.
>  OTOH, you have said several times that you aren't familiar
>  with how NUnit currently loads tests. I think you need
>  to get familiar with the internals in order to fruitfully
>  discuss how we are going to change them.

Can you point me to the classes to look at? I'm making my comments
based upon my observations of how it works and what you've said
before, as well as looking in the debugger at how my Addin code gets
called.

>  Alternatively, we could dial the detail back a little
>  and talk about what you would like to be able to do
>  eithe as a test-writer or as an extension-writer.

Or perhaps we could just discuss how we would want Theory to work, and
I think things would perhaps naturally fall out of that discussion.

>  I'm OK either way.

Me too.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/