Re: Tearing down when SetUp fails?

Hi Kelly,

> Ok Charlie, I totally understand the SetUp TearDown thing in 
> the inheritance model. That makes sense, because it's just 
> like constructors and destructors (at least in C++, let's not 
> get into the weird end of life of C# objects) and the order 
> they are executed makes sense because of that prior 
> experience. I think I've done this myself, in fact. It was useful.

I undersand what you mean. And it's a fairly natural way to think
as well - which is why folks asked for it.
 
> That being said, if you have two SetUps in the same class, I 
> would assume they are all executed prior to calling each 
> test. But how do you know which one is executed first? Or 
> must the order of execution not matter to be safe?

My answer is that it must not matter. This will, of course,
need to be documented.

Charlie 
> -Kelly
> 
> On Tue, Sep 16, 2008 at 9:46 PM, Charlie Poole 
> <[email protected]> wrote:
> > Hi Guys,
> >
> > This isn't planned - it's implemented. :-)
> >
> > The motivation for multiple setups is to allow use of inheritance, 
> > with base class setups called before derived class and teardowns 
> > called in the reverse order. [Note I'm using lower case to 
> mean "all 
> > kinds of setup.]
> >
> > Having multiple setups in the same class - not just in the 
> base is a 
> > fallout of the implementation. In other words, the simplest 
> thing to 
> > do was to just let it happen - it would take more code ot 
> detect and 
> > prevent it. We can do that, but I always like to have a 
> reason before 
> > doing extra work. :-)h
> >
> > It's  possible that figuring out how to handle failures may 
> serve as 
> > the reason for that work, but I don't see it yet.
> >
> > If we change to always running teardown if setup was run, 
> then there 
> > is no problem. NUnit would just treat multiple setups at the same 
> > level as if it were one big setup. It's only if we have to 
> > discriminate among them that it gets hard. In that case, 
> I'd write the 
> > code to prevent it, but otherwise, I'd just let it happen.
> >
> > Charlie
> >> -----Original Message-----
> >> From: [email protected]
> >> [mailto:[email protected]] On 
> Behalf Of 
> >> Olof Bjarnason
> >> Sent: Tuesday, September 16, 2008 12:45 PM
> >> To: [email protected]
> >> Subject: Re: [nunit-developer] Tearing down when SetUp fails?
> >>
> >> Start simple, extent if needed ...
> >>
> >> So one Setup, one TearDown is my vote
> >>
> >> 2008/9/15 Kelly Anderson <[email protected]>:
> >> > Multiple Setups seems like a bad idea to me. If you need 
> multiple 
> >> > setups, why not have multiple TestFixture classes?
> >> >
> >> > What order would they run in? How do you associate a
> >> particular SetUp
> >> > with a particular TearDown... this sounds like opening 
> up a can of 
> >> > worms with no real payback.
> >> >
> >> > Is this really being considered seriously?
> >> >
> >> > -Kelly
> >> >
> >> > On Fri, Sep 12, 2008 at 3:22 PM, Hans Christian Falkenberg 
> >> > <[email protected]> wrote:
> >> >> (Let me just try again with batteries in the crazy keyboard)
> >> >>
> >> >> Ah, I thought I had some place about support for 
> multiple [SetUp], 
> >> >> but I couldn't remember if it was decided for or against,
> >> so I read
> >> >> the 2.5 docs (which understandably aren't up to date 
> yet then) and 
> >> >> concluded that there would only be a single [SetUp].
> >> >>
> >> >> With multiple ones, the conclusion is a given, I think.
> >> >> Consider this:
> >> >>
> >> >> [SetUp] public void SetUp1() {
> >> >>   Acquire1();
> >> >> }
> >> >> [SetUp] public void SetUp2() {
> >> >>   Acquire2();
> >> >> }
> >> >>
> >> >> [TearDown] public void TearDown1() {
> >> >>   Release1();
> >> >> }
> >> >>
> >> >> [TearDown] public void TearDown2() {
> >> >>   Release2();
> >> >> }
> >> >>
> >> >> Now, if SetUp1 succeeds and SetUp2 throws - what are 
> you gonna do?
> >> >> Annotate which [TearDown] belongs to which [SetUp]?
> >> >> That's the sure road to overhead, strange constraints and
> >> confusion.
> >> >>
> >> >> To me it looks like the only intuitive and easily 
> understandable 
> >> >> approach is:
> >> >> 1. If any [SetUp] method throws, no further [SetUp] 
> methods will be
> >> >>    called.
> >> >> 2. All [TearDown] methods will *always* be called.
> >> >>    (Just check for null values, people) 3. Correspondingly 
> >> >> [TestFixture*], obviously
> >> >>
> >> >> But feel free to correct me with coming up with something
> >> simpler :)
> >> >>
> >> >> + Hans Christian
> >> >>
> >> >>
> >> >> On Fri, 12 Sep 2008, Charlie Poole wrote:
> >> >>
> >> >>> OK, I re-read your first mail and I now see you did make a 
> >> >>> distinction betweeen running resharper and nunit-console.
> >> >>>
> >> >>> I want to do some research before I get back to you on
> >> the "how it
> >> >>> should be" part. I have a vague sense that we may have
> >> changed the
> >> >>> behavior from what the docs say as the result of a user
> >> request and
> >> >>> if so I'd like to get the reasons for that request into the 
> >> >>> discussion.
> >> >>>
> >> >>> One part I can explain: in the 2.4 docs I meant the 
> phrase "any 
> >> >>> setup method" to mean "either SetUp or TestFixtureSetUp."
> >> >>> It's still not working as described, but I should have 
> spelled it 
> >> >>> out for clarity.
> >> >>>
> >> >>> In 2.5, there can be multiple SetUp methods, and the
> >> logic of which
> >> >>> TearDowns should and should not be run still has to be
> >> determined.
> >> >>> Since there is not likely to be another 2.4 release, it's
> >> 2.5 that
> >> >>> is the key to this discussion.
> >> >>>
> >> >>> Charlie
> >> >>>
> >> >>> Charlie
> >> >>>
> >> >>>> -----Original Message-----
> >> >>>> From: Hans Christian Falkenberg [mailto:[email protected]]
> >> >>>> Sent: Friday, September 12, 2008 1:12 PM
> >> >>>> To: Charlie Poole
> >> >>>> Cc: [email protected]
> >> >>>> Subject: RE: [nunit-developer] Tearing down when SetUp fails?
> >> >>>>
> >> >>>> Err, well sorry about the name of the test - I
> >> originally used it
> >> >>>> to submit a bug report to ReSharper because I saw different 
> >> >>>> behavior wrt. whether the [TearDown] method was invoked
> >> when I used
> >> >>>> 1) nunit-console.exe (see my exact command in orig mail) and
> >> >>>> 2) Right click -> Run Unit Tests (with ReSharper in VS)
> >> >>>>
> >> >>>> So the answer is most likely that I used both :)
> >> >>>>
> >> >>>> After submitting that bug report I noticed that even
> >> NUnit does it
> >> >>>> different between [TearDown] and 
> [TestFixtureTearDown] so I read 
> >> >>>> the documentation and then I wrote this mail.
> >> >>>>
> >> >>>> + Hans Christian
> >> >>>>
> >> >>>> PS! I also told the ReSharper guys (who replied just an
> >> hour after
> >> >>>> my bug report that they would look on the issue) that I
> >> started a
> >> >>>> discussion on what the behavior should be in NUnit.
> >> >>>>
> >> >>>> On Fri, 12 Sep 2008, Charlie Poole wrote:
> >> >>>>
> >> >>>>> Hi Hans Christian,
> >> >>>>>
> >> >>>>> We should discuss this behavior and try to get it right
> >> >>>> (whatever we
> >> >>>>> decide that is) for 2.5.
> >> >>>>>
> >> >>>>> But before we get into it, because of the name of your
> >> test, I'm
> >> >>>>> wondering if you actually ran under NUnit.
> >> >>>>> When you run under resharper, no matter what version of
> >> the NUnit
> >> >>>>> framework your test references, you get the semantics that
> >> >>>> resharper
> >> >>>>> defines. That's because this particular behavior is not
> >> >>>> implemented in
> >> >>>>> nunit.framework, but in nunit.core, which is not 
> used in this 
> >> >>>>> situation.
> >> >>>>>
> >> >>>>> I'll run my own tests as well, but I thought I should ask.
> >> >>>>>
> >> >>>>>
> >> >>>>> Charlie.
> >> >>>>>
> >> >>>>>> -----Original Message-----
> >> >>>>>> From: [email protected]
> >> >>>>>> [mailto:[email protected]]
> >> On Behalf
> >> >>>>>> Of Hans Christian Falkenberg
> >> >>>>>> Sent: Friday, September 12, 2008 5:33 AM
> >> >>>>>> To: [email protected]
> >> >>>>>> Subject: [nunit-developer] Tearing down when SetUp fails?
> >> >>>>>>
> >> >>>>>> Hi,
> >> >>>>>>
> >> >>>>>> I was about to post a bug about this - but then I read the 
> >> >>>>>> documentation and realized this might be something up for 
> >> >>>>>> discussion. Please let me know if it's there's a previous 
> >> >>>>>> conclusion on this and I'll just submit the bug report...
> >> >>>>>>
> >> >>>>>> Issues:
> >> >>>>>> 1. TearDown behavior is inconsistent between
> >> >>>>>>     [TearDown] and [TestFixtureTearDown] 2. The 
> documentation 
> >> >>>>>> specifies that it should
> >> >>>>>>     do the Wrong Thing (imho) :p
> >> >>>>>>
> >> >>>>>> What I think should happen:
> >> >>>>>>    If there is an exception in any [SetUp] method, still run
> >> >>>>>>    all [TearDown] methods.
> >> >>>>>>    If there is an exception in any 
> [TestFixtureSetUp] method,
> >> >>>>>>    still run all [TestFixtureTearDown] methods.
> >> >>>>>> (I realize only one of each method is allowed, but the doc
> >> >>>> says "any")
> >> >>>>>>
> >> >>>>>> What the documentation says (in both 2.4.8 and 2.5 alpha 3):
> >> >>>>>>    So long as any SetUp method runs without error, the
> >> TearDown
> >> >>>>>> method is
> >> >>>>>>    guaranteed to run. It will not run if a SetUp
> >> method fails or
> >> >>>>>> throws an
> >> >>>>>>    exception.
> >> >>>>>> and
> >> >>>>>>    So long as any TestFixtureSetUp method runs without
> >> error, the
> >> >>>>>>    TestFixtureTearDown method is guaranteed to run. It
> >> will not
> >> >>>>>> run if a
> >> >>>>>>    TestFixtureSetUp method fails or throws an exception.
> >> >>>>>>
> >> >>>>>> What actually happens (in both 2.4.8 and 2.5 alpha 3):
> >> >>>>>> Running attached code:
> >> >>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >> >>>>>>      /nologo bin\Debug\Dummy.dll
> >> >>>>>>    Fixture Setting up
> >> >>>>>>    .Setting up
> >> >>>>>>    Tearing down
> >> >>>>>>    FFixture Tearing down
> >> >>>>>>
> >> >>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 
> 0.030 seconds 
> >> >>>>>> Uncommenting FixtureSetUp exception:
> >> >>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
> >> >>>>>>     /nologo bin\Debug\Dummy.dll
> >> >>>>>>    Fixture Setting up
> >> >>>>>>    .F
> >> >>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 
> 0.028 seconds
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> So [TearDown] doesn't behave as specified when a
> >> [SetUp] method
> >> >>>>>> throws an exception, that's a bug report, right?
> >> >>>>>>
> >> >>>>>> Except I think it *should* behave that way, so I 
> was going to 
> >> >>>>>> submit a bug report on [TestFixtureTearDown] :)
> >> >>>>>>
> >> >>>>>> Here's my reasoning:
> >> >>>>>> Say you are acquiring two resources in the SetUp method:
> >> >>>>>> [SetUp] public void SetUp() {
> >> >>>>>>    r1 = Resources.Acquire1();
> >> >>>>>>    r2 = Resources.Acquire2();
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> And of course the must be released:
> >> >>>>>> [TearDown] public void TearDown() {
> >> >>>>>>    Resources.Release(r1);
> >> >>>>>>    r1 = null;
> >> >>>>>>    Resources.Release(r2);
> >> >>>>>>    r2 = null;
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> Now, since Resource.Release will just return if its
> >> argument is
> >> >>>>>> null and never throws, everything is fine. Except if the 
> >> >>>>>> documentation is to believed: If an exception can 
> occur while 
> >> >>>>>> acquiring r2 (and it can), we'll have to write this:
> >> >>>>>>
> >> >>>>>> [SetUp] public void SetUp() {
> >> >>>>>>    r1 = Resources.Acquire1();
> >> >>>>>>    try {
> >> >>>>>>      r2 = Resources.Acquire2();
> >> >>>>>>    } catch {
> >> >>>>>>      Resources.Release(r1);
> >> >>>>>>      throw;
> >> >>>>>>    }
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> Which I think is bad, but I guess not *that* bad... 
> until you 
> >> >>>>>> have 4-5 resources. Then there's suddenly a crazy amount of 
> >> >>>>>> TearDown code duplication in SetUp. So here is how I
> >> decided to
> >> >>>>>> work around this problem:
> >> >>>>>>
> >> >>>>>> [SetUp] public void SetUp() {
> >> >>>>>>    setUpCalled = true;
> >> >>>>>>    r1 = Resources.Acquire1();
> >> >>>>>>    r2 = Resources.Acquire2();
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> [TearDown] public void TearDown() {
> >> >>>>>>    if (!setUpCalled) return;
> >> >>>>>>    setUpCalled = false;
> >> >>>>>>    Resources.Release(r1);
> >> >>>>>>    r1 = null;
> >> >>>>>>    Resources.Release(r2);
> >> >>>>>>    r2 = null;
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> Not the worst of solution's, but now I'm basically
> >> doing NUnit's
> >> >>>>>> work myself. And when the methods in TearDown *can* throw 
> >> >>>>>> Exceptions they will now be reported as if they 
> were thrown in 
> >> >>>>>> SetUp, which is just to ask for someone being 
> confused later.
> >> >>>>>> But at least the code will release resources 
> properly in both 
> >> >>>>>> NUnit 2.4.8 (and 2.5 alpha 3), ReSharper 4.1.933.3
> >> (which behaves
> >> >>>>>> like the NUnit doc says it should) and future NUnit 
> versions 
> >> >>>>>> which might adhere to the documentation.
> >> >>>>>>
> >> >>>>>> So... could we instead please change the NUnit
> >> behavior so that
> >> >>>>>> other people don't have to write workarounds like this?
> >> >>>>>>
> >> >>>>>> I know junit doesn't run TearDown (or didn't when I
> >> last used it
> >> >>>>>> a year back) when SetUp throws, but junit didn't 
> report test 
> >> >>>>>> exceptions when TearDown throwed either, so not much
> >> cause to let
> >> >>>>>> them set the standard here...
> >> >>>>>>
> >> >>>>>> A better argument against might be that people didn't
> >> write code
> >> >>>>>> to check for null in their [TearDown] and
> >> [TestFixtureTearDown]
> >> >>>>>> methods. But then their [TearDown] methods are already
> >> failing,
> >> >>>>>> and they'll just have to fix their
> >> [TestFixtureTearDown] methods
> >> >>>>>> when upgrading to 2.5,
> >> >>>> won't they?
> >> >>>>>>
> >> >>>>>> + Hans Christian
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> PS: If you are wondering wth. I'm thinking wrt. 
> acquiring 4-5 
> >> >>>>>> resources before a unit tests... Well, the 
> resources are files 
> >> >>>>>> (with Acquire==Create and Release==Delete) and it
> >> would hardly be
> >> >>>>>> proper to use mocked files for unit testing when
> >> testing writing
> >> >>>>>> to the OS - that would kind of void the entire testing.
> >> >>>>>>
> >> >>>>>> PPS: Why should a unit test need files? Well - it does.
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> 
> ---------------------------------------------------------------------
> >> >> ---- This SF.Net email is sponsored by the Moblin Your Move 
> >> >> Developer's challenge Build the coolest Linux based
> >> applications with
> >> >> Moblin SDK & win great prizes Grand prize is a trip for 
> two to an 
> >> >> Open Source event anywhere in the world 
> >> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> >> _______________________________________________
> >> >> nunit-developer mailing list
> >> >> [email protected]
> >> >> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >> >>
> >> >
> >> >
> >> 
> ---------------------------------------------------------------------
> >> -
> >> > --- This SF.Net email is sponsored by the Moblin Your Move
> >> Developer's
> >> > challenge Build the coolest Linux based applications with
> >> Moblin SDK &
> >> > win great prizes Grand prize is a trip for two to an Open
> >> Source event
> >> > anywhere in the world
> >> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> > _______________________________________________
> >> > nunit-developer mailing list
> >> > [email protected]
> >> > https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >> >
> >>
> >> --------------------------------------------------------------
> >> -----------
> >> This SF.Net email is sponsored by the Moblin Your Move Developer's 
> >> challenge Build the coolest Linux based applications with 
> Moblin SDK 
> >> & win great prizes Grand prize is a trip for two to an Open Source 
> >> event anywhere in the world 
> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> _______________________________________________
> >> nunit-developer mailing list
> >> [email protected]
> >> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >>
> >
> >
> >
> > 
> ----------------------------------------------------------------------
> > --- This SF.Net email is sponsored by the Moblin Your Move 
> Developer's 
> > challenge Build the coolest Linux based applications with 
> Moblin SDK & 
> > win great prizes Grand prize is a trip for two to an Open 
> Source event 
> > anywhere in the world 
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > nunit-developer mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/nunit-developer
> >
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/