Re: Tearing down when SetUp fails?

Hi Hans Christian,

I agree. Although we *could* partition SetUps into those
found in the top level, those found in the base, those
found in the base of the base, etc. I don't think I want
to. So either all or none of the TearDowns have to run
and none is pretty useless.

However, I think we can make a distinction between
SetUp, TestFixtureSetUp and higher level SetUp.

Charlie

> -----Original Message-----
> From: [email protected] 
> [mailto:[email protected]] On 
> Behalf Of Hans Christian Falkenberg
> Sent: Friday, September 12, 2008 2:22 PM
> To: [email protected]
> Subject: Re: [nunit-developer] Tearing down when SetUp fails?
> 
> (Let me just try again with batteries in the crazy keyboard)
> 
> Ah, I thought I had some place about support for multiple 
> [SetUp], but I couldn't remember if it was decided for or 
> against, so I read the 2.5 docs (which understandably aren't 
> up to date yet then) and concluded that there would only be a 
> single [SetUp].
> 
> With multiple ones, the conclusion is a given, I think.
> Consider this:
> 
> [SetUp] public void SetUp1() {
>    Acquire1();
> }
> [SetUp] public void SetUp2() {
>    Acquire2();
> }
> 
> [TearDown] public void TearDown1() {
>    Release1();
> }
> 
> [TearDown] public void TearDown2() {
>    Release2();
> }
> 
> Now, if SetUp1 succeeds and SetUp2 throws - what are you gonna do?
> Annotate which [TearDown] belongs to which [SetUp]?
> That's the sure road to overhead, strange constraints and confusion.
> 
> To me it looks like the only intuitive and easily 
> understandable approach is:
> 1. If any [SetUp] method throws, no further [SetUp] methods will be
>     called.
> 2. All [TearDown] methods will *always* be called.
>     (Just check for null values, people) 3. Correspondingly 
> [TestFixture*], obviously
> 
> But feel free to correct me with coming up with something simpler :)
> 
> + Hans Christian
> 
> 
> On Fri, 12 Sep 2008, Charlie Poole wrote:
> 
> > OK, I re-read your first mail and I now see you did make a 
> distinction 
> > betweeen running resharper and nunit-console.
> >
> > I want to do some research before I get back to you on the "how it 
> > should be" part. I have a vague sense that we may have changed the 
> > behavior from what the docs say as the result of a user 
> request and if 
> > so I'd like to get the reasons for that request into the discussion.
> >
> > One part I can explain: in the 2.4 docs I meant the phrase 
> "any setup 
> > method" to mean "either SetUp or TestFixtureSetUp."
> > It's still not working as described, but I should have 
> spelled it out 
> > for clarity.
> >
> > In 2.5, there can be multiple SetUp methods, and the logic of which 
> > TearDowns should and should not be run still has to be determined. 
> > Since there is not likely to be another 2.4 release, it's 
> 2.5 that is 
> > the key to this discussion.
> >
> > Charlie
> >
> > Charlie
> >
> >> -----Original Message-----
> >> From: Hans Christian Falkenberg [mailto:[email protected]]
> >> Sent: Friday, September 12, 2008 1:12 PM
> >> To: Charlie Poole
> >> Cc: [email protected]
> >> Subject: RE: [nunit-developer] Tearing down when SetUp fails?
> >>
> >> Err, well sorry about the name of the test - I originally 
> used it to 
> >> submit a bug report to ReSharper because I saw different behavior 
> >> wrt. whether the [TearDown] method was invoked when I used
> >> 1) nunit-console.exe (see my exact command in orig mail) and
> >> 2) Right click -> Run Unit Tests (with ReSharper in VS)
> >>
> >> So the answer is most likely that I used both :)
> >>
> >> After submitting that bug report I noticed that even NUnit does it 
> >> different between [TearDown] and [TestFixtureTearDown] so 
> I read the 
> >> documentation and then I wrote this mail.
> >>
> >> + Hans Christian
> >>
> >> PS! I also told the ReSharper guys (who replied just an 
> hour after my 
> >> bug report that they would look on the issue) that I started a 
> >> discussion on what the behavior should be in NUnit.
> >>
> >> On Fri, 12 Sep 2008, Charlie Poole wrote:
> >>
> >>> Hi Hans Christian,
> >>>
> >>> We should discuss this behavior and try to get it right
> >> (whatever we
> >>> decide that is) for 2.5.
> >>>
> >>> But before we get into it, because of the name of your test, I'm 
> >>> wondering if you actually ran under NUnit.
> >>> When you run under resharper, no matter what version of the NUnit 
> >>> framework your test references, you get the semantics that
> >> resharper
> >>> defines. That's because this particular behavior is not
> >> implemented in
> >>> nunit.framework, but in nunit.core, which is not used in this 
> >>> situation.
> >>>
> >>> I'll run my own tests as well, but I thought I should ask.
> >>>
> >>>
> >>> Charlie.
> >>>
> >>>> -----Original Message-----
> >>>> From: [email protected]
> >>>> [mailto:[email protected]] 
> On Behalf Of 
> >>>> Hans Christian Falkenberg
> >>>> Sent: Friday, September 12, 2008 5:33 AM
> >>>> To: [email protected]
> >>>> Subject: [nunit-developer] Tearing down when SetUp fails?
> >>>>
> >>>> Hi,
> >>>>
> >>>> I was about to post a bug about this - but then I read the 
> >>>> documentation and realized this might be something up for 
> >>>> discussion. Please let me know if it's there's a previous 
> >>>> conclusion on this and I'll just submit the bug report...
> >>>>
> >>>> Issues:
> >>>> 1. TearDown behavior is inconsistent between
> >>>>     [TearDown] and [TestFixtureTearDown] 2. The documentation 
> >>>> specifies that it should
> >>>>     do the Wrong Thing (imho) :p
> >>>>
> >>>> What I think should happen:
> >>>>    If there is an exception in any [SetUp] method, still run
> >>>>    all [TearDown] methods.
> >>>>    If there is an exception in any [TestFixtureSetUp] method,
> >>>>    still run all [TestFixtureTearDown] methods.
> >>>> (I realize only one of each method is allowed, but the doc
> >> says "any")
> >>>>
> >>>> What the documentation says (in both 2.4.8 and 2.5 alpha 3):
> >>>>    So long as any SetUp method runs without error, the TearDown 
> >>>> method is
> >>>>    guaranteed to run. It will not run if a SetUp method fails or 
> >>>> throws an
> >>>>    exception.
> >>>> and
> >>>>    So long as any TestFixtureSetUp method runs without error, the
> >>>>    TestFixtureTearDown method is guaranteed to run. It 
> will not run 
> >>>> if a
> >>>>    TestFixtureSetUp method fails or throws an exception.
> >>>>
> >>>> What actually happens (in both 2.4.8 and 2.5 alpha 3):
> >>>> Running attached code:
> >>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >>>>      /nologo bin\Debug\Dummy.dll
> >>>>    Fixture Setting up
> >>>>    .Setting up
> >>>>    Tearing down
> >>>>    FFixture Tearing down
> >>>>
> >>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.030 seconds 
> >>>> Uncommenting FixtureSetUp exception:
> >>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >>>>     /nologo bin\Debug\Dummy.dll
> >>>>    Fixture Setting up
> >>>>    .F
> >>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.028 seconds
> >>>>
> >>>>
> >>>> So [TearDown] doesn't behave as specified when a [SetUp] method 
> >>>> throws an exception, that's a bug report, right?
> >>>>
> >>>> Except I think it *should* behave that way, so I was going to 
> >>>> submit a bug report on [TestFixtureTearDown] :)
> >>>>
> >>>> Here's my reasoning:
> >>>> Say you are acquiring two resources in the SetUp method:
> >>>> [SetUp] public void SetUp() {
> >>>>    r1 = Resources.Acquire1();
> >>>>    r2 = Resources.Acquire2();
> >>>> }
> >>>>
> >>>> And of course the must be released:
> >>>> [TearDown] public void TearDown() {
> >>>>    Resources.Release(r1);
> >>>>    r1 = null;
> >>>>    Resources.Release(r2);
> >>>>    r2 = null;
> >>>> }
> >>>>
> >>>> Now, since Resource.Release will just return if its argument is 
> >>>> null and never throws, everything is fine. Except if the 
> >>>> documentation is to believed: If an exception can occur while 
> >>>> acquiring r2 (and it can), we'll have to write this:
> >>>>
> >>>> [SetUp] public void SetUp() {
> >>>>    r1 = Resources.Acquire1();
> >>>>    try {
> >>>>      r2 = Resources.Acquire2();
> >>>>    } catch {
> >>>>      Resources.Release(r1);
> >>>>      throw;
> >>>>    }
> >>>> }
> >>>>
> >>>> Which I think is bad, but I guess not *that* bad... 
> until you have 
> >>>> 4-5 resources. Then there's suddenly a crazy amount of TearDown 
> >>>> code duplication in SetUp. So here is how I decided to 
> work around 
> >>>> this problem:
> >>>>
> >>>> [SetUp] public void SetUp() {
> >>>>    setUpCalled = true;
> >>>>    r1 = Resources.Acquire1();
> >>>>    r2 = Resources.Acquire2();
> >>>> }
> >>>>
> >>>> [TearDown] public void TearDown() {
> >>>>    if (!setUpCalled) return;
> >>>>    setUpCalled = false;
> >>>>    Resources.Release(r1);
> >>>>    r1 = null;
> >>>>    Resources.Release(r2);
> >>>>    r2 = null;
> >>>> }
> >>>>
> >>>> Not the worst of solution's, but now I'm basically doing NUnit's 
> >>>> work myself. And when the methods in TearDown *can* throw 
> >>>> Exceptions they will now be reported as if they were thrown in 
> >>>> SetUp, which is just to ask for someone being confused later.
> >>>> But at least the code will release resources properly in 
> both NUnit 
> >>>> 2.4.8 (and 2.5 alpha 3), ReSharper 4.1.933.3 (which behaves like 
> >>>> the NUnit doc says it should) and future NUnit versions 
> which might 
> >>>> adhere to the documentation.
> >>>>
> >>>> So... could we instead please change the NUnit behavior so that 
> >>>> other people don't have to write workarounds like this?
> >>>>
> >>>> I know junit doesn't run TearDown (or didn't when I last 
> used it a 
> >>>> year back) when SetUp throws, but junit didn't report test 
> >>>> exceptions when TearDown throwed either, so not much 
> cause to let 
> >>>> them set the standard here...
> >>>>
> >>>> A better argument against might be that people didn't 
> write code to 
> >>>> check for null in their [TearDown] and [TestFixtureTearDown] 
> >>>> methods. But then their [TearDown] methods are already 
> failing, and 
> >>>> they'll just have to fix their [TestFixtureTearDown] 
> methods when 
> >>>> upgrading to 2.5,
> >> won't they?
> >>>>
> >>>> + Hans Christian
> >>>>
> >>>>
> >>>>
> >>>> PS: If you are wondering wth. I'm thinking wrt. acquiring 4-5 
> >>>> resources before a unit tests... Well, the resources are files 
> >>>> (with Acquire==Create and Release==Delete) and it would 
> hardly be 
> >>>> proper to use mocked files for unit testing when testing 
> writing to 
> >>>> the OS - that would kind of void the entire testing.
> >>>>
> >>>> PPS: Why should a unit test need files? Well - it does.
> >>>>
> >>>
> >>>
> >>>
> >>
> >
> >
> >
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/