Re: Extension Points in NUnit 2.5

Hi Kelly,

> >  AssemblySuiteBuilder - To recognize attributes on assemblies  and 
> > build an appropriate test suite for it. This would do  for 
> assemblies, 
> > what SuiteBuilder and TestCaseBuilder do  for types and methods.
> 
> This sounds really useful to me. I can think of all sorts of 
> tests that could be run on an assembly basis using metadata.. 
> and this would really help in building those sorts of tests 
> automatically.
> 
> >   Optional: It may be reasonable to combine all three
> >   into one extension. They all take a reflection object
> >   and produce some sort of Test object. What do folks think?
> 
> I don't have a strong opinion on this... would it change the 
> way current things work? What are the trade offs, pro and con 
> for putting them all together? Would it help in creating new variants?

Pro: It would be easier to create an extension that can work in
     multiple contexts - like on a type and on a method.

Con: Most such extensions would likely be decorators or some other
     modifier for tests, not test builders.

Con: If it added more methods, every extenson would have to implemet
     them, even if they just threw an NYI exception, and would have
     to deal with further tests in their CanBuildFrom method.

Con: The same thing can be accomplished with a single class that
     implements several interfaces.

Con: More extensions would have to be examined for each type and
     method found in a test assembly.

I think I talked myself out of it. :-)

> >  TestDataProvider - A new extension point to encapsulate  a 
> source of 
> > data to be used in instantiating a test. There  are some 
> design issues 
> > around this, which Andreas is looking  into and will write 
> about. It 
> > could end up being more than  one extension point - but 
> we'll see. In 
> > the end, this could  be exploited by various data-provider syntaxes.
> 
> Can we please have this discussion out on the list...

Yes... Andreas has volunteered to work in this area and is drafting
up some ideas for the list. I'll throw my ideas in at the same time
as I'm sure you and others will.

We'll try to focus this on architecture rather than syntax.

> >  ProjectConverter - An extension point for addins that know  how to 
> > read some particular format and produce an NUnit  test 
> project. I've 
> > already experimentally rewritten our  Visual Studio loading code to 
> > use this and it could be  used for other IDE formats as 
> well as other 
> > test framework  project formats (csUnit recipe files, for example). 
> > There  are pros and cons to doing this in 2.5:
> >
> >   Pro: This will give us a chance to experiment with our
> >   first extension point on the client side rather than in
> >   the test domain.
> >
> >   Con: This could represent a bit of infrastructure work,
> >   which would need to be redone in 3.0.
> >
> >   My Conclusion: See how much work it is.
> 
> If you looked at it as a prototype for 3.0, and thought about 
> what we might learn about the problem domain, that might tip 
> the decision point to some extent.

Exactly... trading off future usefulness versus effort.

> In general, this extension point sounds like it might be very 
> useful in terms of declarative programming tests, such as xml 
> (XAML, etc.) testing.

I believe you're thinking of something like an xml file that 
expresses the tests declaratively as well. That's a really 
good idea.

However, it's not what this is for. It's only to construct
NUnit projects, which are essentially lists of files. At this
time, those files are all assemblies.

However, in 3.0, we planned for an extension that will take
a single file and construct a test suite from it, as NUnit
currently does for assemblies. That's where your declarative
test specification - if I guessed right - would fit in. 
Other use cases for that, off the top of my head, include...
  pNUnit test specifications (which are xml)
  static checking of source using a rules file
  running junit tests (jar files)

> >  GENERAL:
> >
> >   I'd like to implement the feature whereby addins needed
> >   by a test are declared using an assembly attribute. If
> >   there is no declaration, things work just like now. If
> >   an addin is declared but is not installed, the entire
> >   test fails.
> 
> I'm not sure what this means exactly... but it doesn't sound bad. :-)

We had some discussion about this on one of the lists - I'm not
sure which. Essentially, you can optionally provide a sort of
manifest of addins needed to run a test. For example...

   [assembly:RequiredExtension("IterativeTestExtension")]

If it's not available, the whole thing fails. The reason this is
of value is that some tests will be silently ignored or even
silently succeed if the extension is not available. However,
we will still use any extensions, even if they are not declared.
That was the general consensus in the earlier discussion.

> >  Please let me know what you think of this as a plan - subject  to 
> > change as we go, of course.
> 
> Isn't that always the case? :-)

It's a good thing for us, of course, because we keep getting smarter. :-)

Charlie
 
> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/