Re: Roadmap Updated

Hi Kelly,

> >  > The only case I can think of where you might want to 
> call  > user 
> > code is in generating "children" Tests. This would only  > 
> be for the 
> > case where you programmatically (ala  > IterativeTest) generate the 
> > children.
> >
> >  Yes. That's the only one we have come up with. I've yet to 
>  see any 
> > reason why this should not be done at the point of  running 
> the tests.
> 
> In theory, I agree. The issue is whether the current 
> architecture supports this, or do we need an architecture change?

Yes, it does. That was kind of my point. :-)

> >  > >  Let's say I run the gui and let it stay  open for an hour.
> >  > While it's
> >  > > sitting there, I modify the contents  of the directory 
> in question.
> >  > >
> >  > >  Now I run the tests. Run time is now. Load time was 
> an hour ago.
> >  > >
> >  > >  What implications does this have for your requirements?
> >  >
> >  > I suppose you can guess my answer... :-)  I just want it 
> to  > work 
> > the way I expect of course :-P  >  > In thinking about the 
> > implementation is interesting. If there  > is a separate step for 
> > generating "child" lists of tests, I  > suppose they could be rerun 
> > every time for certain classes of  > Theories...
> >
> >  Is there any reason you can think of not to rerun the generation  
> > every time? If so, how will we let NUnit know that the old data  
> > should be used?
> 
> I don't know why you would need to for RowTest, but I would 
> see no problem running it every time for IterativeTest...

There is no generation for row test - the data is just there.

> >  Because I take this new concept very seriously, it is an 
> impediment  
> > to communication for me to have to discuss "theory-like" 
> > implementations  as a shorthand for "interchangeable data 
> sources" - 
> > which is what  I suspect you may be talking about.
> 
> Give me an example where you couldn't implement a Theory in that way.
> In reading the Theory papers, they seem to implement it that 
> way. What do you think I'm missing?

If you use a data-driven approach to prove a theory, that's Theory.
If you use if for something else, that's not the case.

Example - Say I'm testing farenheight to celsius conversion...
The syntax is invented - and hopefully clear...

[Test(32,0)][
[Test(212,0)]
[Test(-40,-40)]
public void CanConvertFarenheitToCelsius(int f, inf c)
{
  Assert.That( FarenheitToCelsius(f), Is.EqualTo( c ) );
}

Nothing theory-like about that. It's just an easier 
way to write three tests.

[Test]
public void FarenheitToCelciusToFarenheitGivesOriginalValue(
	[Values(-40,0,32,212,500)]val)
{
  Assert.That( 
	CelciusToFarenheit( FarenheitToCelcius( val ) ),
	Is.EqualTo( val ) );
}

That's a theory. And I didn't even use the word. :-)

> >  > I agree... it's not about data-driven tests at all, even 
>  > though 
> > that's a side effect.
> >
> >  OK? But that seems to contradict your "merely" above.
> 
> Grrr. I hate not having a white board and being in the same room...
> this would be so simple.
> 
> The intent of Theory is to state a mathematical theorem that 
> the code should fulfill in all circumstances. That's a pretty 
> powerful idea.

Yes!!!

> You could drop down and do Dykstra proofs of each of these 
> theorems against your code, but that is heavy. A lighter way 
> to indicate that your theorem is likely true is to run a lot 
> of different example data sets against the Theory. This 
> doesn't prove anything, but it does give a good indication 
> that you're on the right track.

Exactly. It's amazing that we suddenly agree!

> RowTest and IterativeTest are two styles of providing those 
> example data sets to your Theory. 

Precisely. And I'd like to retain that precision in our discussion.
Testing a Theory against a finite set of examples requires two
things: a Theory and a way to express or generate examples.
RowTest and IterativeTest can serve in the second role, but
it would be incorrect to say that they somehow "are" Theory.
More precisely, it was incorrect when you said it. :-)

> Are there other ways? 
> Possibly. 

Indeed. And before going on, I'll say that it's sufficient
that you can /imagine/ that there might be other ways for
this to be an important distinction - a separtion of concerns
that we need to deal with.

> You might, for example have an IterativeTest that 
> computes code coverage on the fly, and continues to generate 
> a valid Next() in the IEnumerable with new semi-random data 
> values until all or most of the code was covered.

Yes.

> That would be kind of interesting. You couldn't do that with 
> the current IterativeTest because the list is generated 
> apriori... 

Which is what I've been griping about.

> so that would require a different kind of 
> IterativeTest that recomputed Next after each test was run. 
> Nevertheless, I think all of Theory is covered if you can 
> produce the right set of values to pass into parameterized tests.

Whoops. There you go again. Sounds like you're saying that
IterativeTest == Theory. I'm hoping what you mean to say is 
that "all of a particular theory" is covered by the data.

To put a finer point on my point... IF you are saying
that IterativeTest is /sufficient/ to produce an implementation
of Theories, I am in complete agreement. That's just not
an interesting statement to me. I'd like to know what
other sorts of techniques might be /useful/ in that
implementation.

Charlie



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/