Re: Addin Error Reporting Issue

Hi Kelly, 

> For programming errors (and NUnit is mostly about those) 
> stopping after the first failure seems right. For data tests, 
> most often it is not the right thing.

I'd put it a little differently. NUnit works in two phases:
first it loads tests, then it runs them.

When loading tests, NUnit makes an effort to list everything
that is "supposed to be" a test. That is, if you mark it
as a test, it should show up, no matter what is wrong with
it. When running tests, it stops on the first failure.

Your addin is working fine at run time. Once you have 
created a separate test case for each bit of data,
those tests run independently.

The problem is at load time. You are writing code that
can fail - since it is based on what the user provides - 
but you aren't handling the failure. So NUnit "fixes"
the problem at the first level of granularity it 
understands, which is the test case you are building.

Your load-time code (in BuildFrom) has to be able to
deal with the fact that some test cases may not be
buildable, given the data availble, OR you have to
be able to build even with bad data, passing the
responsibility for a failure to the run time, which
is in your Run method for the individual case.

Generally, if you want it to be non-runnable (yellow)
do it at load time and if you want it to fail (red)
do it at run time.
 
> > There are several things that need to be added to 
> > NUnit to make this work write:
> >
> > 1) Data arguments to tests
> > 2) Running all the data points, even if some fail
> > 3) Properly handling initialization errors.
> >
> > You implemented two out of three and were expecting the 
> NUnit core to 
> > take care of the third. ;-)
> 
> I wasn't so much expecting it to do so as I was curious about 
> what the right way to do it was. If I can handle it within 
> the Addin, I will do so. I'm just not sure how in the context 
> of building the lists of tests that I can build a test failed 
> result without running the test.
> I suppose that I could just create a fake test that would 
> fail with a somewhat meaningful message and put it into the 
> test list... would that be a proper approach without 
> modifying the core?

I think so. That's how NUnit does it itself - in the addin
that implements normal test cases.

> Marking as non-runnable seems like it might be right. What I 
> envision is something like this... When the test works, it 
> creates a node in the tree with children for each data 
> specific sub-test-instance...
> When the building of the IEnumerable fails, I would like to 
> create just that root node, marked red, with some sort of 
> message about the list building routine throwing an exception 
> or something along those lines. No children in that case.

That would be pretty easy. But what if only one of the elements
has bad data?

> The most important thing being that the other tests are run 
> normally, and display the right results without interference 
> from the fact that my list building activity faild.
> 
> So, going to the code, as you suggested... the key function 
> seems to be this one:
> 
>     public Test BuildFrom(MethodInfo method)
>     {
>       Attribute attrib = Reflect.GetAttribute(method, 
> IterativeTestAttribute, false);
>       string functionName =
> IterativeTestFramework.GetSourceEnumeratorName(attrib);
>       TestSuite suite = new TestSuite(method.Name);
> 
>       object tester = Reflect.Construct(method.DeclaringType);
>       MethodInfo m = tester.GetType().GetMethod(functionName);
>       IEnumerable returnValue = (IEnumerable) m.Invoke(tester, null);
>       if (returnValue != null)
>       {
>         foreach (object current in returnValue)
>         {
>           suite.Add(new IterativeTestCase(method, current));
>         }
>       }
> 
>       return suite;
>     }

OK, at this point, I realize that some of what I told you earlier
was off the mark. I was thinking you had a TestFixtureBuilder
as well as a TestCaseBuilder and that the fixture builder was
adding the test cases. Obviously, that's not how it works, so
igore any previous advise and lets go ahead with this code.

> So, if I try to handle the situation myself... My best guess 
> is something along these lines:
> 
>     public Test BuildFrom(MethodInfo method)
>     {
>       Attribute attrib = Reflect.GetAttribute(method, 
> IterativeTestAttribute, false);
>       string functionName =
> IterativeTestFramework.GetSourceEnumeratorName(attrib);
>       TestSuite suite = new TestSuite(method.Name);
> 
>       object tester = Reflect.Construct(method.DeclaringType);
>       MethodInfo m = tester.GetType().GetMethod(functionName);
>       IEnumerable returnValue = null;
>       try
>       {
>         returnValue = (IEnumerable) m.Invoke(tester, null);
>         if (returnValue != null)

CP: I would either not test for null and let it throw or put 
in code so that a null return is marked as non-runnable. The
former seems easiest.

>         {
>           foreach (object current in returnValue)
>           {
>             suite.Add(new IterativeTestCase(method, current));
>           }
>         }
>       }
>       catch (Exception e)
>       {
>         suite.RunState = RunState.NotRunnable;
>         suite.IgnoreReason =
>           string.Format("{0} failed to return a valid 
> IEnumerable.\n{1}", functionName, e.Message);
>       }
> 
>       return suite;
>     }
> 
> But since RunState and IgnoreReason have no intellisense... 
> I'm not sure this is exactly the right approach, but I 
> suspect it's something close to this that I want to do.

I'm not sure why the intellisense fails sometimes, but this
is how to handle the problem.

In addition, you might want to consider the special case
where the for an individual element throws. In that case,
you can either not add the element or add a special element
that will fail when executed.

Charlie

> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/