Re: Addin Error Reporting Issue

Hi Kelly, 

> I've been using my IterativeTest extension for the last 
> couple of days pretty heavily and there's one aspect that's 
> really bothersome from a user's (my) standpoint. I'd like to 
> know how to fix it, or whether the core of how NUnit creates 
> the tests would have to change.
> 
> I will illustrate the problem by example:
> 
>     public IEnumerable FileObjects()
>     {
>       ArrayList rv = new ArrayList();
>       DirectoryInfo di = new DirectoryInfo(@"..\..\DataFiles");
>       FileInfo[] fi = di.GetFiles("*.*");
>       int i = 0;
>       foreach (FileInfo info in fi)
>       {
>         MyObject s = new MyObject(info.FullName);
>         rv.Add(s);
>       }>       return rv;
>     }
> 
>     [IterativeTest("FileObjects")]
>     public void AllObjectsMeetCondition(object current)
>     {
>       MyObject o = (MyObject) current;
>       Assert.IsTrue(o.Condition);
>     }

FWIW, you could do the same thing like this, without the
extension...

    [Test]
    public void AllObjectsMeetCondition()
    {
       ArrayList rv = new ArrayList();
       DirectoryInfo di = new DirectoryInfo(@"..\..\DataFiles");
       FileInfo[] fi = di.GetFiles("*.*");
       foreach (FileInfo info in fi)
       {
         MyObject s = new MyObject(info.FullName);
         rv.Add(s);
       }

       Assert.That( rv, Has.All.Property( "Condition", true ) );
    } 

My point is not that the addin is useless - this is a simple, contrived
example after all - but that you may need to do a bit more than what
NUnit already does here. The substitute code above has exactly the
same problem as your code... the test will fail without any indication
of which data element caused the failure and none of the other tests
will be run.

> Ok, this is a fairly simple example of the usage of the 
> IterativeTest... and pretty typical, I think. When I Open the 
> DLL that contains this test, it populates the test list by 
> calling my code; specifically, it calls FileObjects(). The 
> problem comes when say the constructor of MyObject throws an 
> exception because say the file is badly formed. This 
> exception is handled by NUnit, so that it doesn't crash, but 
> no tests are created at all in this case. It fails to load 
> the entire DLL. What I would prefer to have happen is for the 
> AllObjectsMeetCondition test to be marked as having failed. 
> Then I could have another regular test like this...
> 
>   [Test]
>   public void FileObjectsIsCorrectlyFilled()
>   {
>     IEnumerable x = FileObjects();
>   }
> 
> Which would allow me to debug the creation of the FileObjects 
> independently of my AddIn. I don't like the fact that because 
> FileObjects threw an exception none of the other tests are 
> successfully run, or even shown in the NUnit GUI. I admit to 
> still being a user of the Debugger, so the second test helps 
> me out a lot, if it can be run.
> 
> I think this could be solved by putting a try-catch around 
> the creation of the tests, and if the creation of the test 
> throws an exception, then the test is somehow created in an 
> already failing state, and doesn't actually try to run later, 
> but the entire load doesn't fail. Does that make sense? Am I 
> stating the problem clearly enough? This seems like a fairly 
> general problem for all Addins that call user code during 
> test creation.
> 
> Thanks for any suggestions!

>From NUnit's point of view, here's what is happening... barring
any false memory of how your code works since it's not in front
of me...

1) Here is a Type... lets see if someone CanBuildFrom it.
2) OK, this guy says he can, so
3) Let's call his BuildFrom...
4) Whoops... BuildFrom threw an exception, the whole thing
is in error.

Since the FixtureBuilder is responsible for calling the
TestCaseBuilder, it has to intercept any exceptions. If 
they are passed back to NUnit, it appears that the whole
fixture is in error.

Remember, when you are writing an addin, you are becoming
a part of NUnit in a certain sense, so you can no longer
completely rely on "NUnit" to catch the exceptions.

Charlie
 
> -Kelly
> 
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by: Microsoft Defy all 
> challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/