Re: Addin Error Reporting Issue

On Thu, Feb 14, 2008 at 12:49 PM, Charlie Poole
<[email protected]> wrote:
> Hi Kelly,
>
> > > FWIW, you could do the same thing like this, without the
> > extension...
> > >
> > >     [Test]
> > >     public void AllObjectsMeetCondition()
> > >     {
> > >        ArrayList rv = new ArrayList();
> > >        DirectoryInfo di = new DirectoryInfo(@"..\..\DataFiles");
> > >        FileInfo[] fi = di.GetFiles("*.*");
> > >        foreach (FileInfo info in fi)
> > >        {
> > >          MyObject s = new MyObject(info.FullName);
> > >          rv.Add(s);
> > >        }
> > >
> > >        Assert.That( rv, Has.All.Property( "Condition", true ) );
> > >     }
> >
> > Yes, you can do this. The question is how does it report a failure.
> > When Has.All.Property fails, does it output separate messages
> > for each Property in the collection that doesn't meet the
> > condition? I really need it to tell me which pieces of data
> > fail in one pass, not a simple all passed or something
> > failed. That's why I did the addin. Granted, I didn't look
> > into Has.All.Property at the time so it could do reporting
> > the way I want, but I didn't know about it.
>
> I shouldn't tell you - so that you're forced to try it out. :-)
>
> But I will. It reports what data didn't fail, but it doesn't
> go past the first failure.

Thanks for spilling the beans. :-)

For programming errors (and NUnit is mostly about those) stopping
after the first failure seems right. For data tests, most often it is
not the right thing.

> However, I think you have missed
> my point, which was stated below...

No, I got it. :-)

> > > My point is not that the addin is useless - this is a simple,
> > > contrived example after all - but that you may need to do a
> > bit more
> > > than what NUnit already does here. The substitute code above has
> > > exactly the same problem as your code... the test will fail without
> > > any indication of which data element caused the failure and none of
> > > the other tests will be run.
> >
> > This is only a problem in my code if it fails in building the
> > list. It does do proper reporting if one of the files doesn't
> > meet the condition inside the [IterativeTest]...
>
> Exactly my point. There are several things that need to be
> added to NUnit to make this work write:
>
> 1) Data arguments to tests
> 2) Running all the data points, even if some fail
> 3) Properly handling initialization errors.
>
> You implemented two out of three and were expecting the NUnit
> core to take care of the third. ;-)

I wasn't so much expecting it to do so as I was curious about what the
right way to do it was. If I can handle it within the Addin, I will do
so. I'm just not sure how in the context of building the lists of
tests that I can build a test failed result without running the test.
I suppose that I could just create a fake test that would fail with a
somewhat meaningful message and put it into the test list... would
that be a proper approach without modifying the core?

> > Sure, that's fine. The question though is how can I have it
> > show up as merely a failed test rather than a complete
> > failure to load the entire set of tests? I'm willing to do
> > some work to make this behave better, but the question is
> > whether it can be done without messing with the core of
> > NUnit, or would I have to go into the Core?
>
> I thought I said, but maybe I should go to your code to
> give it in more detail.
>
> In summary... When you are building test cases, you have to
> handle exceptions. When the attempt to create a test case
> fails because of bad data, you have to do something so that
> it will show up as a failed test case.

Right. How would you do that?

> Exactly what you do depends on what you want to happen, but
> one way would be to mark the test case as non-runnable, and
> fill in an appropriate reason. Another approach would be to
> override Run so that the test case fails when the data
> isn't correct.
>
> If you like, we can go to the code to be more specific.

Marking as non-runnable seems like it might be right. What I envision
is something like this... When the test works, it creates a node in
the tree with children for each data specific sub-test-instance...
When the building of the IEnumerable fails, I would like to create
just that root node, marked red, with some sort of message about the
list building routine throwing an exception or something along those
lines. No children in that case.

The most important thing being that the other tests are run normally,
and display the right results without interference from the fact that
my list building activity faild.

So, going to the code, as you suggested... the key function seems to
be this one:

    public Test BuildFrom(MethodInfo method)
    {
      Attribute attrib = Reflect.GetAttribute(method,
IterativeTestAttribute, false);
      string functionName =
IterativeTestFramework.GetSourceEnumeratorName(attrib);
      TestSuite suite = new TestSuite(method.Name);

      object tester = Reflect.Construct(method.DeclaringType);
      MethodInfo m = tester.GetType().GetMethod(functionName);
      IEnumerable returnValue = (IEnumerable) m.Invoke(tester, null);
      if (returnValue != null)
      {
        foreach (object current in returnValue)
        {
          suite.Add(new IterativeTestCase(method, current));
        }
      }

      return suite;
    }

So, if I try to handle the situation myself... My best guess is
something along these lines:

    public Test BuildFrom(MethodInfo method)
    {
      Attribute attrib = Reflect.GetAttribute(method,
IterativeTestAttribute, false);
      string functionName =
IterativeTestFramework.GetSourceEnumeratorName(attrib);
      TestSuite suite = new TestSuite(method.Name);

      object tester = Reflect.Construct(method.DeclaringType);
      MethodInfo m = tester.GetType().GetMethod(functionName);
      IEnumerable returnValue = null;
      try
      {
        returnValue = (IEnumerable) m.Invoke(tester, null);
        if (returnValue != null)
        {
          foreach (object current in returnValue)
          {
            suite.Add(new IterativeTestCase(method, current));
          }
        }
      }
      catch (Exception e)
      {
        suite.RunState = RunState.NotRunnable;
        suite.IgnoreReason =
          string.Format("{0} failed to return a valid
IEnumerable.\n{1}", functionName, e.Message);
      }

      return suite;
    }

But since RunState and IgnoreReason have no intellisense... I'm not
sure this is exactly the right approach, but I suspect it's something
close to this that I want to do.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/