Re: Tearing down when SetUp fails?

Multiple Setups seems like a bad idea to me. If you need multiple
setups, why not have multiple TestFixture classes?

What order would they run in? How do you associate a particular SetUp
with a particular TearDown... this sounds like opening up a can of
worms with no real payback.

Is this really being considered seriously?

-Kelly

On Fri, Sep 12, 2008 at 3:22 PM, Hans Christian Falkenberg
<[email protected]> wrote:
> (Let me just try again with batteries in the crazy keyboard)
>
> Ah, I thought I had some place about support for multiple
> [SetUp], but I couldn't remember if it was decided for or
> against, so I read the 2.5 docs (which understandably aren't
> up to date yet then) and concluded that there would only be
> a single [SetUp].
>
> With multiple ones, the conclusion is a given, I think.
> Consider this:
>
> [SetUp] public void SetUp1() {
>   Acquire1();
> }
> [SetUp] public void SetUp2() {
>   Acquire2();
> }
>
> [TearDown] public void TearDown1() {
>   Release1();
> }
>
> [TearDown] public void TearDown2() {
>   Release2();
> }
>
> Now, if SetUp1 succeeds and SetUp2 throws - what are you gonna do?
> Annotate which [TearDown] belongs to which [SetUp]?
> That's the sure road to overhead, strange constraints and confusion.
>
> To me it looks like the only intuitive and easily understandable
> approach is:
> 1. If any [SetUp] method throws, no further [SetUp] methods will be
>    called.
> 2. All [TearDown] methods will *always* be called.
>    (Just check for null values, people)
> 3. Correspondingly [TestFixture*], obviously
>
> But feel free to correct me with coming up with something simpler :)
>
> + Hans Christian
>
>
> On Fri, 12 Sep 2008, Charlie Poole wrote:
>
>> OK, I re-read your first mail and I now see you did make a
>> distinction betweeen running resharper and nunit-console.
>>
>> I want to do some research before I get back to you on the
>> "how it should be" part. I have a vague sense that we may
>> have changed the behavior from what the docs say as the
>> result of a user request and if so I'd like to get the
>> reasons for that request into the discussion.
>>
>> One part I can explain: in the 2.4 docs I meant the phrase
>> "any setup method" to mean "either SetUp or TestFixtureSetUp."
>> It's still not working as described, but I should have
>> spelled it out for clarity.
>>
>> In 2.5, there can be multiple SetUp methods, and the logic
>> of which TearDowns should and should not be run still
>> has to be determined. Since there is not likely to be
>> another 2.4 release, it's 2.5 that is the key to this
>> discussion.
>>
>> Charlie
>>
>> Charlie
>>
>>> -----Original Message-----
>>> From: Hans Christian Falkenberg [mailto:[email protected]]
>>> Sent: Friday, September 12, 2008 1:12 PM
>>> To: Charlie Poole
>>> Cc: [email protected]
>>> Subject: RE: [nunit-developer] Tearing down when SetUp fails?
>>>
>>> Err, well sorry about the name of the test - I originally
>>> used it to submit a bug report to ReSharper because I saw
>>> different behavior wrt. whether the [TearDown] method was
>>> invoked when I used
>>> 1) nunit-console.exe (see my exact command in orig mail) and
>>> 2) Right click -> Run Unit Tests (with ReSharper in VS)
>>>
>>> So the answer is most likely that I used both :)
>>>
>>> After submitting that bug report I noticed that even NUnit
>>> does it different between [TearDown] and
>>> [TestFixtureTearDown] so I read the documentation and then I
>>> wrote this mail.
>>>
>>> + Hans Christian
>>>
>>> PS! I also told the ReSharper guys (who replied just an hour
>>> after my bug report that they would look on the issue) that I
>>> started a discussion on what the behavior should be in NUnit.
>>>
>>> On Fri, 12 Sep 2008, Charlie Poole wrote:
>>>
>>>> Hi Hans Christian,
>>>>
>>>> We should discuss this behavior and try to get it right
>>> (whatever we
>>>> decide that is) for 2.5.
>>>>
>>>> But before we get into it, because of the name of your test, I'm
>>>> wondering if you actually ran under NUnit.
>>>> When you run under resharper, no matter what version of the NUnit
>>>> framework your test references, you get the semantics that
>>> resharper
>>>> defines. That's because this particular behavior is not
>>> implemented in
>>>> nunit.framework, but in nunit.core, which is not used in this
>>>> situation.
>>>>
>>>> I'll run my own tests as well, but I thought I should ask.
>>>>
>>>>
>>>> Charlie.
>>>>
>>>>> -----Original Message-----
>>>>> From: [email protected]
>>>>> [mailto:[email protected]] On
>>>>> Behalf Of Hans Christian Falkenberg
>>>>> Sent: Friday, September 12, 2008 5:33 AM
>>>>> To: [email protected]
>>>>> Subject: [nunit-developer] Tearing down when SetUp fails?
>>>>>
>>>>> Hi,
>>>>>
>>>>> I was about to post a bug about this - but then I read the
>>>>> documentation and realized this might be something up for
>>>>> discussion. Please let me know if it's there's a previous
>>>>> conclusion on this and I'll just submit the bug report...
>>>>>
>>>>> Issues:
>>>>> 1. TearDown behavior is inconsistent between
>>>>>     [TearDown] and [TestFixtureTearDown] 2. The documentation
>>>>> specifies that it should
>>>>>     do the Wrong Thing (imho) :p
>>>>>
>>>>> What I think should happen:
>>>>>    If there is an exception in any [SetUp] method, still run
>>>>>    all [TearDown] methods.
>>>>>    If there is an exception in any [TestFixtureSetUp] method,
>>>>>    still run all [TestFixtureTearDown] methods.
>>>>> (I realize only one of each method is allowed, but the doc
>>> says "any")
>>>>>
>>>>> What the documentation says (in both 2.4.8 and 2.5 alpha 3):
>>>>>    So long as any SetUp method runs without error, the
>>>>> TearDown method is
>>>>>    guaranteed to run. It will not run if a SetUp method fails
>>>>> or throws an
>>>>>    exception.
>>>>> and
>>>>>    So long as any TestFixtureSetUp method runs without error, the
>>>>>    TestFixtureTearDown method is guaranteed to run. It will
>>>>> not run if a
>>>>>    TestFixtureSetUp method fails or throws an exception.
>>>>>
>>>>> What actually happens (in both 2.4.8 and 2.5 alpha 3):
>>>>> Running attached code:
>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
>>>>>      /nologo bin\Debug\Dummy.dll
>>>>>    Fixture Setting up
>>>>>    .Setting up
>>>>>    Tearing down
>>>>>    FFixture Tearing down
>>>>>
>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.030 seconds
>>>>> Uncommenting FixtureSetUp exception:
>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
>>>>>     /nologo bin\Debug\Dummy.dll
>>>>>    Fixture Setting up
>>>>>    .F
>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.028 seconds
>>>>>
>>>>>
>>>>> So [TearDown] doesn't behave as specified when a [SetUp]
>>>>> method throws an exception, that's a bug report, right?
>>>>>
>>>>> Except I think it *should* behave that way, so I was going to
>>>>> submit a bug report on [TestFixtureTearDown] :)
>>>>>
>>>>> Here's my reasoning:
>>>>> Say you are acquiring two resources in the SetUp method:
>>>>> [SetUp] public void SetUp() {
>>>>>    r1 = Resources.Acquire1();
>>>>>    r2 = Resources.Acquire2();
>>>>> }
>>>>>
>>>>> And of course the must be released:
>>>>> [TearDown] public void TearDown() {
>>>>>    Resources.Release(r1);
>>>>>    r1 = null;
>>>>>    Resources.Release(r2);
>>>>>    r2 = null;
>>>>> }
>>>>>
>>>>> Now, since Resource.Release will just return if its argument
>>>>> is null and never throws, everything is fine. Except if the
>>>>> documentation is to believed: If an exception can occur while
>>>>> acquiring r2 (and it can), we'll have to write this:
>>>>>
>>>>> [SetUp] public void SetUp() {
>>>>>    r1 = Resources.Acquire1();
>>>>>    try {
>>>>>      r2 = Resources.Acquire2();
>>>>>    } catch {
>>>>>      Resources.Release(r1);
>>>>>      throw;
>>>>>    }
>>>>> }
>>>>>
>>>>> Which I think is bad, but I guess not *that* bad... until you
>>>>> have 4-5 resources. Then there's suddenly a crazy amount of
>>>>> TearDown code duplication in SetUp. So here is how I decided
>>>>> to work around this problem:
>>>>>
>>>>> [SetUp] public void SetUp() {
>>>>>    setUpCalled = true;
>>>>>    r1 = Resources.Acquire1();
>>>>>    r2 = Resources.Acquire2();
>>>>> }
>>>>>
>>>>> [TearDown] public void TearDown() {
>>>>>    if (!setUpCalled) return;
>>>>>    setUpCalled = false;
>>>>>    Resources.Release(r1);
>>>>>    r1 = null;
>>>>>    Resources.Release(r2);
>>>>>    r2 = null;
>>>>> }
>>>>>
>>>>> Not the worst of solution's, but now I'm basically doing
>>>>> NUnit's work myself. And when the methods in TearDown *can*
>>>>> throw Exceptions they will now be reported as if they were
>>>>> thrown in SetUp, which is just to ask for someone being
>>>>> confused later.
>>>>> But at least the code will release resources properly in both
>>>>> NUnit 2.4.8 (and 2.5 alpha 3), ReSharper 4.1.933.3 (which
>>>>> behaves like the NUnit doc says it should) and future NUnit
>>>>> versions which might adhere to the documentation.
>>>>>
>>>>> So... could we instead please change the NUnit behavior so
>>>>> that other people don't have to write workarounds like this?
>>>>>
>>>>> I know junit doesn't run TearDown (or didn't when I last used
>>>>> it a year back) when SetUp throws, but junit didn't report
>>>>> test exceptions when TearDown throwed either, so not much
>>>>> cause to let them set the standard here...
>>>>>
>>>>> A better argument against might be that people didn't write
>>>>> code to check for null in their [TearDown] and
>>>>> [TestFixtureTearDown] methods. But then their [TearDown]
>>>>> methods are already failing, and they'll just have to fix
>>>>> their [TestFixtureTearDown] methods when upgrading to 2.5,
>>> won't they?
>>>>>
>>>>> + Hans Christian
>>>>>
>>>>>
>>>>>
>>>>> PS: If you are wondering wth. I'm thinking wrt. acquiring 4-5
>>>>> resources before a unit tests... Well, the resources are
>>>>> files (with Acquire==Create and Release==Delete) and it would
>>>>> hardly be proper to use mocked files for unit testing when
>>>>> testing writing to the OS - that would kind of void the
>>>>> entire testing.
>>>>>
>>>>> PPS: Why should a unit test need files? Well - it does.
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/