Re: Roadmap Updated

On Wed, Mar 12, 2008 at 2:28 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>
>
>  > The only case I can think of where you might want to call
>  > user code is in generating "children" Tests. This would only
>  > be for the case where you programmatically (ala
>  > IterativeTest) generate the children.
>
>  Yes. That's the only one we have come up with. I've yet to
>  see any reason why this should not be done at the point of
>  running the tests.

In theory, I agree. The issue is whether the current architecture
supports this, or do we need an architecture change?

>  > >  Let's say I run the gui and let it stay  open for an hour.
>  > While it's
>  > > sitting there, I modify the contents  of the directory in question.
>  > >
>  > >  Now I run the tests. Run time is now. Load time was an hour ago.
>  > >
>  > >  What implications does this have for your requirements?
>  >
>  > I suppose you can guess my answer... :-)  I just want it to
>  > work the way I expect of course :-P
>  >
>  > In thinking about the implementation is interesting. If there
>  > is a separate step for generating "child" lists of tests, I
>  > suppose they could be rerun every time for certain classes of
>  > Theories...
>
>  Is there any reason you can think of not to rerun the generation
>  every time? If so, how will we let NUnit know that the old data
>  should be used?

I don't know why you would need to for RowTest, but I would see no
problem running it every time for IterativeTest...

>  > >  I don't know what you mean by "Theory" but to my
>  > definition,  it's a
>  > > lot more than a combination of two technical approaches.
>  > >  It's a whole new concept and a way of thinking about things.
>  >
>  > It is a new concept, but like BDD, it's a concept that I
>  > think merely encapsulates the RowTest and IterativeTest
>  > approaches, with the possible addition of randomly or
>  > semi-randomly generated parameter sets. If there is more to
>  > your understanding of Theory than that, I'd like to know what it is.
>
>  I disagree very strongly. It's not merely a combination of several
>  programming techniques. It's an entirely different way of thinking.
>  Certainly, data-driven techniques are a part of it. If I were the
>  guy who originated the concepts, I'd be offended, but I think he's
>  less touchy than I am. :-)

It could just be that I don't understand the entire concept. But I've
read the papers carefully, and I do think I've been thoughtful about
it. I almost NEVER intend to be offensive, although sometimes I am in
email... sometimes because of the limits of the medium ...sometimes
because I'm stupid and hit send too soon. Sorry about that.

>  Because I take this new concept very seriously, it is an impediment
>  to communication for me to have to discuss "theory-like" implementations
>  as a shorthand for "interchangeable data sources" - which is what
>  I suspect you may be talking about.

Give me an example where you couldn't implement a Theory in that way.
In reading the Theory papers, they seem to implement it that way. What
do you think I'm missing?

>  > I agree... it's not about data-driven tests at all, even
>  > though that's a side effect.
>
>  OK? But that seems to contradict your "merely" above.

Grrr. I hate not having a white board and being in the same room...
this would be so simple.

The intent of Theory is to state a mathematical theorem that the code
should fulfill in all circumstances. That's a pretty powerful idea.
You could drop down and do Dykstra proofs of each of these theorems
against your code, but that is heavy. A lighter way to indicate that
your theorem is likely true is to run a lot of different example data
sets against the Theory. This doesn't prove anything, but it does give
a good indication that you're on the right track.

RowTest and IterativeTest are two styles of providing those example
data sets to your Theory. Are there other ways? Possibly. You might,
for example have an IterativeTest that computes code coverage on the
fly, and continues to generate a valid Next() in the IEnumerable with
new semi-random data values until all or most of the code was covered.
That would be kind of interesting. You couldn't do that with the
current IterativeTest because the list is generated apriori... so that
would require a different kind of IterativeTest that recomputed Next
after each test was run. Nevertheless, I think all of Theory is
covered if you can produce the right set of values to pass into
parameterized tests.

>  > >  Aside from the definition, I haven't yet seen a proposed  API to
>  > > which I might commit. Are you suggestiong one?
>  >
>  > I am... would you like a formalized proposal? We've talked
>  > about a couple of approaches in some email threads... but I'd
>  > be happy to write a formal proposed syntax.
>
>  For theories or for data-driven tests? And you don't have to
>  be too formal - just spell it out. :-)

Ok... I'll start a new thread.

>  However, if you are talking about a full-fledged implementation
>  of the Theory concept, I think we're on the wrong list. NUnitV3
>  would be more suitable for futuristic stuff.

Alright, I'll post my proposed API thread there.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/