Re: Tearing down when SetUp fails?

Hi Guys, 

This isn't planned - it's implemented. :-)

The motivation for multiple setups is to allow use of 
inheritance, with base class setups called before
derived class and teardowns called in the reverse
order. [Note I'm using lower case to mean "all kinds
of setup.]

Having multiple setups in the same class - not just
in the base is a fallout of the implementation. In
other words, the simplest thing to do was to just
let it happen - it would take more code ot detect
and prevent it. We can do that, but I always like
to have a reason before doing extra work. :-)h

It's  possible that figuring out how to handle
failures may serve as the reason for that work,
but I don't see it yet.

If we change to always running teardown if setup
was run, then there is no problem. NUnit would
just treat multiple setups at the same level as if
it were one big setup. It's only if we have to 
discriminate among them that it gets hard. In that 
case, I'd write the code to prevent it, but otherwise,
I'd just let it happen.

Charlie
> -----Original Message-----
> From: [email protected] 
> [mailto:[email protected]] On 
> Behalf Of Olof Bjarnason
> Sent: Tuesday, September 16, 2008 12:45 PM
> To: [email protected]
> Subject: Re: [nunit-developer] Tearing down when SetUp fails?
> 
> Start simple, extent if needed ...
> 
> So one Setup, one TearDown is my vote
> 
> 2008/9/15 Kelly Anderson <[email protected]>:
> > Multiple Setups seems like a bad idea to me. If you need multiple 
> > setups, why not have multiple TestFixture classes?
> >
> > What order would they run in? How do you associate a 
> particular SetUp 
> > with a particular TearDown... this sounds like opening up a can of 
> > worms with no real payback.
> >
> > Is this really being considered seriously?
> >
> > -Kelly
> >
> > On Fri, Sep 12, 2008 at 3:22 PM, Hans Christian Falkenberg 
> > <[email protected]> wrote:
> >> (Let me just try again with batteries in the crazy keyboard)
> >>
> >> Ah, I thought I had some place about support for multiple [SetUp], 
> >> but I couldn't remember if it was decided for or against, 
> so I read 
> >> the 2.5 docs (which understandably aren't up to date yet then) and 
> >> concluded that there would only be a single [SetUp].
> >>
> >> With multiple ones, the conclusion is a given, I think.
> >> Consider this:
> >>
> >> [SetUp] public void SetUp1() {
> >>   Acquire1();
> >> }
> >> [SetUp] public void SetUp2() {
> >>   Acquire2();
> >> }
> >>
> >> [TearDown] public void TearDown1() {
> >>   Release1();
> >> }
> >>
> >> [TearDown] public void TearDown2() {
> >>   Release2();
> >> }
> >>
> >> Now, if SetUp1 succeeds and SetUp2 throws - what are you gonna do?
> >> Annotate which [TearDown] belongs to which [SetUp]?
> >> That's the sure road to overhead, strange constraints and 
> confusion.
> >>
> >> To me it looks like the only intuitive and easily understandable 
> >> approach is:
> >> 1. If any [SetUp] method throws, no further [SetUp] methods will be
> >>    called.
> >> 2. All [TearDown] methods will *always* be called.
> >>    (Just check for null values, people) 3. Correspondingly 
> >> [TestFixture*], obviously
> >>
> >> But feel free to correct me with coming up with something 
> simpler :)
> >>
> >> + Hans Christian
> >>
> >>
> >> On Fri, 12 Sep 2008, Charlie Poole wrote:
> >>
> >>> OK, I re-read your first mail and I now see you did make a 
> >>> distinction betweeen running resharper and nunit-console.
> >>>
> >>> I want to do some research before I get back to you on 
> the "how it 
> >>> should be" part. I have a vague sense that we may have 
> changed the 
> >>> behavior from what the docs say as the result of a user 
> request and 
> >>> if so I'd like to get the reasons for that request into the 
> >>> discussion.
> >>>
> >>> One part I can explain: in the 2.4 docs I meant the phrase "any 
> >>> setup method" to mean "either SetUp or TestFixtureSetUp."
> >>> It's still not working as described, but I should have spelled it 
> >>> out for clarity.
> >>>
> >>> In 2.5, there can be multiple SetUp methods, and the 
> logic of which 
> >>> TearDowns should and should not be run still has to be 
> determined. 
> >>> Since there is not likely to be another 2.4 release, it's 
> 2.5 that 
> >>> is the key to this discussion.
> >>>
> >>> Charlie
> >>>
> >>> Charlie
> >>>
> >>>> -----Original Message-----
> >>>> From: Hans Christian Falkenberg [mailto:[email protected]]
> >>>> Sent: Friday, September 12, 2008 1:12 PM
> >>>> To: Charlie Poole
> >>>> Cc: [email protected]
> >>>> Subject: RE: [nunit-developer] Tearing down when SetUp fails?
> >>>>
> >>>> Err, well sorry about the name of the test - I 
> originally used it 
> >>>> to submit a bug report to ReSharper because I saw different 
> >>>> behavior wrt. whether the [TearDown] method was invoked 
> when I used
> >>>> 1) nunit-console.exe (see my exact command in orig mail) and
> >>>> 2) Right click -> Run Unit Tests (with ReSharper in VS)
> >>>>
> >>>> So the answer is most likely that I used both :)
> >>>>
> >>>> After submitting that bug report I noticed that even 
> NUnit does it 
> >>>> different between [TearDown] and [TestFixtureTearDown] so I read 
> >>>> the documentation and then I wrote this mail.
> >>>>
> >>>> + Hans Christian
> >>>>
> >>>> PS! I also told the ReSharper guys (who replied just an 
> hour after 
> >>>> my bug report that they would look on the issue) that I 
> started a 
> >>>> discussion on what the behavior should be in NUnit.
> >>>>
> >>>> On Fri, 12 Sep 2008, Charlie Poole wrote:
> >>>>
> >>>>> Hi Hans Christian,
> >>>>>
> >>>>> We should discuss this behavior and try to get it right
> >>>> (whatever we
> >>>>> decide that is) for 2.5.
> >>>>>
> >>>>> But before we get into it, because of the name of your 
> test, I'm 
> >>>>> wondering if you actually ran under NUnit.
> >>>>> When you run under resharper, no matter what version of 
> the NUnit 
> >>>>> framework your test references, you get the semantics that
> >>>> resharper
> >>>>> defines. That's because this particular behavior is not
> >>>> implemented in
> >>>>> nunit.framework, but in nunit.core, which is not used in this 
> >>>>> situation.
> >>>>>
> >>>>> I'll run my own tests as well, but I thought I should ask.
> >>>>>
> >>>>>
> >>>>> Charlie.
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: [email protected]
> >>>>>> [mailto:[email protected]] 
> On Behalf 
> >>>>>> Of Hans Christian Falkenberg
> >>>>>> Sent: Friday, September 12, 2008 5:33 AM
> >>>>>> To: [email protected]
> >>>>>> Subject: [nunit-developer] Tearing down when SetUp fails?
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I was about to post a bug about this - but then I read the 
> >>>>>> documentation and realized this might be something up for 
> >>>>>> discussion. Please let me know if it's there's a previous 
> >>>>>> conclusion on this and I'll just submit the bug report...
> >>>>>>
> >>>>>> Issues:
> >>>>>> 1. TearDown behavior is inconsistent between
> >>>>>>     [TearDown] and [TestFixtureTearDown] 2. The documentation 
> >>>>>> specifies that it should
> >>>>>>     do the Wrong Thing (imho) :p
> >>>>>>
> >>>>>> What I think should happen:
> >>>>>>    If there is an exception in any [SetUp] method, still run
> >>>>>>    all [TearDown] methods.
> >>>>>>    If there is an exception in any [TestFixtureSetUp] method,
> >>>>>>    still run all [TestFixtureTearDown] methods.
> >>>>>> (I realize only one of each method is allowed, but the doc
> >>>> says "any")
> >>>>>>
> >>>>>> What the documentation says (in both 2.4.8 and 2.5 alpha 3):
> >>>>>>    So long as any SetUp method runs without error, the 
> TearDown 
> >>>>>> method is
> >>>>>>    guaranteed to run. It will not run if a SetUp 
> method fails or 
> >>>>>> throws an
> >>>>>>    exception.
> >>>>>> and
> >>>>>>    So long as any TestFixtureSetUp method runs without 
> error, the
> >>>>>>    TestFixtureTearDown method is guaranteed to run. It 
> will not 
> >>>>>> run if a
> >>>>>>    TestFixtureSetUp method fails or throws an exception.
> >>>>>>
> >>>>>> What actually happens (in both 2.4.8 and 2.5 alpha 3):
> >>>>>> Running attached code:
> >>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >>>>>>      /nologo bin\Debug\Dummy.dll
> >>>>>>    Fixture Setting up
> >>>>>>    .Setting up
> >>>>>>    Tearing down
> >>>>>>    FFixture Tearing down
> >>>>>>
> >>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.030 seconds 
> >>>>>> Uncommenting FixtureSetUp exception:
> >>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >>>>>>     /nologo bin\Debug\Dummy.dll
> >>>>>>    Fixture Setting up
> >>>>>>    .F
> >>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.028 seconds
> >>>>>>
> >>>>>>
> >>>>>> So [TearDown] doesn't behave as specified when a 
> [SetUp] method 
> >>>>>> throws an exception, that's a bug report, right?
> >>>>>>
> >>>>>> Except I think it *should* behave that way, so I was going to 
> >>>>>> submit a bug report on [TestFixtureTearDown] :)
> >>>>>>
> >>>>>> Here's my reasoning:
> >>>>>> Say you are acquiring two resources in the SetUp method:
> >>>>>> [SetUp] public void SetUp() {
> >>>>>>    r1 = Resources.Acquire1();
> >>>>>>    r2 = Resources.Acquire2();
> >>>>>> }
> >>>>>>
> >>>>>> And of course the must be released:
> >>>>>> [TearDown] public void TearDown() {
> >>>>>>    Resources.Release(r1);
> >>>>>>    r1 = null;
> >>>>>>    Resources.Release(r2);
> >>>>>>    r2 = null;
> >>>>>> }
> >>>>>>
> >>>>>> Now, since Resource.Release will just return if its 
> argument is 
> >>>>>> null and never throws, everything is fine. Except if the 
> >>>>>> documentation is to believed: If an exception can occur while 
> >>>>>> acquiring r2 (and it can), we'll have to write this:
> >>>>>>
> >>>>>> [SetUp] public void SetUp() {
> >>>>>>    r1 = Resources.Acquire1();
> >>>>>>    try {
> >>>>>>      r2 = Resources.Acquire2();
> >>>>>>    } catch {
> >>>>>>      Resources.Release(r1);
> >>>>>>      throw;
> >>>>>>    }
> >>>>>> }
> >>>>>>
> >>>>>> Which I think is bad, but I guess not *that* bad... until you 
> >>>>>> have 4-5 resources. Then there's suddenly a crazy amount of 
> >>>>>> TearDown code duplication in SetUp. So here is how I 
> decided to 
> >>>>>> work around this problem:
> >>>>>>
> >>>>>> [SetUp] public void SetUp() {
> >>>>>>    setUpCalled = true;
> >>>>>>    r1 = Resources.Acquire1();
> >>>>>>    r2 = Resources.Acquire2();
> >>>>>> }
> >>>>>>
> >>>>>> [TearDown] public void TearDown() {
> >>>>>>    if (!setUpCalled) return;
> >>>>>>    setUpCalled = false;
> >>>>>>    Resources.Release(r1);
> >>>>>>    r1 = null;
> >>>>>>    Resources.Release(r2);
> >>>>>>    r2 = null;
> >>>>>> }
> >>>>>>
> >>>>>> Not the worst of solution's, but now I'm basically 
> doing NUnit's 
> >>>>>> work myself. And when the methods in TearDown *can* throw 
> >>>>>> Exceptions they will now be reported as if they were thrown in 
> >>>>>> SetUp, which is just to ask for someone being confused later.
> >>>>>> But at least the code will release resources properly in both 
> >>>>>> NUnit 2.4.8 (and 2.5 alpha 3), ReSharper 4.1.933.3 
> (which behaves 
> >>>>>> like the NUnit doc says it should) and future NUnit versions 
> >>>>>> which might adhere to the documentation.
> >>>>>>
> >>>>>> So... could we instead please change the NUnit 
> behavior so that 
> >>>>>> other people don't have to write workarounds like this?
> >>>>>>
> >>>>>> I know junit doesn't run TearDown (or didn't when I 
> last used it 
> >>>>>> a year back) when SetUp throws, but junit didn't report test 
> >>>>>> exceptions when TearDown throwed either, so not much 
> cause to let 
> >>>>>> them set the standard here...
> >>>>>>
> >>>>>> A better argument against might be that people didn't 
> write code 
> >>>>>> to check for null in their [TearDown] and 
> [TestFixtureTearDown] 
> >>>>>> methods. But then their [TearDown] methods are already 
> failing, 
> >>>>>> and they'll just have to fix their 
> [TestFixtureTearDown] methods 
> >>>>>> when upgrading to 2.5,
> >>>> won't they?
> >>>>>>
> >>>>>> + Hans Christian
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> PS: If you are wondering wth. I'm thinking wrt. acquiring 4-5 
> >>>>>> resources before a unit tests... Well, the resources are files 
> >>>>>> (with Acquire==Create and Release==Delete) and it 
> would hardly be 
> >>>>>> proper to use mocked files for unit testing when 
> testing writing 
> >>>>>> to the OS - that would kind of void the entire testing.
> >>>>>>
> >>>>>> PPS: Why should a unit test need files? Well - it does.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >> 
> ---------------------------------------------------------------------
> >> ---- This SF.Net email is sponsored by the Moblin Your Move 
> >> Developer's challenge Build the coolest Linux based 
> applications with 
> >> Moblin SDK & win great prizes Grand prize is a trip for two to an 
> >> Open Source event anywhere in the world 
> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> _______________________________________________
> >> nunit-developer mailing list
> >> [email protected]
> >> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >>
> >
> > 
> ----------------------------------------------------------------------
> > --- This SF.Net email is sponsored by the Moblin Your Move 
> Developer's 
> > challenge Build the coolest Linux based applications with 
> Moblin SDK & 
> > win great prizes Grand prize is a trip for two to an Open 
> Source event 
> > anywhere in the world 
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > nunit-developer mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/