Re: Tearing down when SetUp fails?

Start simple, extent if needed ...

So one Setup, one TearDown is my vote

2008/9/15 Kelly Anderson <[email protected]>:
> Multiple Setups seems like a bad idea to me. If you need multiple
> setups, why not have multiple TestFixture classes?
>
> What order would they run in? How do you associate a particular SetUp
> with a particular TearDown... this sounds like opening up a can of
> worms with no real payback.
>
> Is this really being considered seriously?
>
> -Kelly
>
> On Fri, Sep 12, 2008 at 3:22 PM, Hans Christian Falkenberg
> <[email protected]> wrote:
>> (Let me just try again with batteries in the crazy keyboard)
>>
>> Ah, I thought I had some place about support for multiple
>> [SetUp], but I couldn't remember if it was decided for or
>> against, so I read the 2.5 docs (which understandably aren't
>> up to date yet then) and concluded that there would only be
>> a single [SetUp].
>>
>> With multiple ones, the conclusion is a given, I think.
>> Consider this:
>>
>> [SetUp] public void SetUp1() {
>>   Acquire1();
>> }
>> [SetUp] public void SetUp2() {
>>   Acquire2();
>> }
>>
>> [TearDown] public void TearDown1() {
>>   Release1();
>> }
>>
>> [TearDown] public void TearDown2() {
>>   Release2();
>> }
>>
>> Now, if SetUp1 succeeds and SetUp2 throws - what are you gonna do?
>> Annotate which [TearDown] belongs to which [SetUp]?
>> That's the sure road to overhead, strange constraints and confusion.
>>
>> To me it looks like the only intuitive and easily understandable
>> approach is:
>> 1. If any [SetUp] method throws, no further [SetUp] methods will be
>>    called.
>> 2. All [TearDown] methods will *always* be called.
>>    (Just check for null values, people)
>> 3. Correspondingly [TestFixture*], obviously
>>
>> But feel free to correct me with coming up with something simpler :)
>>
>> + Hans Christian
>>
>>
>> On Fri, 12 Sep 2008, Charlie Poole wrote:
>>
>>> OK, I re-read your first mail and I now see you did make a
>>> distinction betweeen running resharper and nunit-console.
>>>
>>> I want to do some research before I get back to you on the
>>> "how it should be" part. I have a vague sense that we may
>>> have changed the behavior from what the docs say as the
>>> result of a user request and if so I'd like to get the
>>> reasons for that request into the discussion.
>>>
>>> One part I can explain: in the 2.4 docs I meant the phrase
>>> "any setup method" to mean "either SetUp or TestFixtureSetUp."
>>> It's still not working as described, but I should have
>>> spelled it out for clarity.
>>>
>>> In 2.5, there can be multiple SetUp methods, and the logic
>>> of which TearDowns should and should not be run still
>>> has to be determined. Since there is not likely to be
>>> another 2.4 release, it's 2.5 that is the key to this
>>> discussion.
>>>
>>> Charlie
>>>
>>> Charlie
>>>
>>>> -----Original Message-----
>>>> From: Hans Christian Falkenberg [mailto:[email protected]]
>>>> Sent: Friday, September 12, 2008 1:12 PM
>>>> To: Charlie Poole
>>>> Cc: [email protected]
>>>> Subject: RE: [nunit-developer] Tearing down when SetUp fails?
>>>>
>>>> Err, well sorry about the name of the test - I originally
>>>> used it to submit a bug report to ReSharper because I saw
>>>> different behavior wrt. whether the [TearDown] method was
>>>> invoked when I used
>>>> 1) nunit-console.exe (see my exact command in orig mail) and
>>>> 2) Right click -> Run Unit Tests (with ReSharper in VS)
>>>>
>>>> So the answer is most likely that I used both :)
>>>>
>>>> After submitting that bug report I noticed that even NUnit
>>>> does it different between [TearDown] and
>>>> [TestFixtureTearDown] so I read the documentation and then I
>>>> wrote this mail.
>>>>
>>>> + Hans Christian
>>>>
>>>> PS! I also told the ReSharper guys (who replied just an hour
>>>> after my bug report that they would look on the issue) that I
>>>> started a discussion on what the behavior should be in NUnit.
>>>>
>>>> On Fri, 12 Sep 2008, Charlie Poole wrote:
>>>>
>>>>> Hi Hans Christian,
>>>>>
>>>>> We should discuss this behavior and try to get it right
>>>> (whatever we
>>>>> decide that is) for 2.5.
>>>>>
>>>>> But before we get into it, because of the name of your test, I'm
>>>>> wondering if you actually ran under NUnit.
>>>>> When you run under resharper, no matter what version of the NUnit
>>>>> framework your test references, you get the semantics that
>>>> resharper
>>>>> defines. That's because this particular behavior is not
>>>> implemented in
>>>>> nunit.framework, but in nunit.core, which is not used in this
>>>>> situation.
>>>>>
>>>>> I'll run my own tests as well, but I thought I should ask.
>>>>>
>>>>>
>>>>> Charlie.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [email protected]
>>>>>> [mailto:[email protected]] On
>>>>>> Behalf Of Hans Christian Falkenberg
>>>>>> Sent: Friday, September 12, 2008 5:33 AM
>>>>>> To: [email protected]
>>>>>> Subject: [nunit-developer] Tearing down when SetUp fails?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I was about to post a bug about this - but then I read the
>>>>>> documentation and realized this might be something up for
>>>>>> discussion. Please let me know if it's there's a previous
>>>>>> conclusion on this and I'll just submit the bug report...
>>>>>>
>>>>>> Issues:
>>>>>> 1. TearDown behavior is inconsistent between
>>>>>>     [TearDown] and [TestFixtureTearDown] 2. The documentation
>>>>>> specifies that it should
>>>>>>     do the Wrong Thing (imho) :p
>>>>>>
>>>>>> What I think should happen:
>>>>>>    If there is an exception in any [SetUp] method, still run
>>>>>>    all [TearDown] methods.
>>>>>>    If there is an exception in any [TestFixtureSetUp] method,
>>>>>>    still run all [TestFixtureTearDown] methods.
>>>>>> (I realize only one of each method is allowed, but the doc
>>>> says "any")
>>>>>>
>>>>>> What the documentation says (in both 2.4.8 and 2.5 alpha 3):
>>>>>>    So long as any SetUp method runs without error, the
>>>>>> TearDown method is
>>>>>>    guaranteed to run. It will not run if a SetUp method fails
>>>>>> or throws an
>>>>>>    exception.
>>>>>> and
>>>>>>    So long as any TestFixtureSetUp method runs without error, the
>>>>>>    TestFixtureTearDown method is guaranteed to run. It will
>>>>>> not run if a
>>>>>>    TestFixtureSetUp method fails or throws an exception.
>>>>>>
>>>>>> What actually happens (in both 2.4.8 and 2.5 alpha 3):
>>>>>> Running attached code:
>>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
>>>>>>      /nologo bin\Debug\Dummy.dll
>>>>>>    Fixture Setting up
>>>>>>    .Setting up
>>>>>>    Tearing down
>>>>>>    FFixture Tearing down
>>>>>>
>>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.030 seconds
>>>>>> Uncommenting FixtureSetUp exception:
>>>>>>    c:\>"c:\Program Files\NUnit 2.4.8\bin\nunit-console.exe"
>>>>>>     /nologo bin\Debug\Dummy.dll
>>>>>>    Fixture Setting up
>>>>>>    .F
>>>>>>    Tests run: 1, Failures: 1, Not run: 0, Time: 0.028 seconds
>>>>>>
>>>>>>
>>>>>> So [TearDown] doesn't behave as specified when a [SetUp]
>>>>>> method throws an exception, that's a bug report, right?
>>>>>>
>>>>>> Except I think it *should* behave that way, so I was going to
>>>>>> submit a bug report on [TestFixtureTearDown] :)
>>>>>>
>>>>>> Here's my reasoning:
>>>>>> Say you are acquiring two resources in the SetUp method:
>>>>>> [SetUp] public void SetUp() {
>>>>>>    r1 = Resources.Acquire1();
>>>>>>    r2 = Resources.Acquire2();
>>>>>> }
>>>>>>
>>>>>> And of course the must be released:
>>>>>> [TearDown] public void TearDown() {
>>>>>>    Resources.Release(r1);
>>>>>>    r1 = null;
>>>>>>    Resources.Release(r2);
>>>>>>    r2 = null;
>>>>>> }
>>>>>>
>>>>>> Now, since Resource.Release will just return if its argument
>>>>>> is null and never throws, everything is fine. Except if the
>>>>>> documentation is to believed: If an exception can occur while
>>>>>> acquiring r2 (and it can), we'll have to write this:
>>>>>>
>>>>>> [SetUp] public void SetUp() {
>>>>>>    r1 = Resources.Acquire1();
>>>>>>    try {
>>>>>>      r2 = Resources.Acquire2();
>>>>>>    } catch {
>>>>>>      Resources.Release(r1);
>>>>>>      throw;
>>>>>>    }
>>>>>> }
>>>>>>
>>>>>> Which I think is bad, but I guess not *that* bad... until you
>>>>>> have 4-5 resources. Then there's suddenly a crazy amount of
>>>>>> TearDown code duplication in SetUp. So here is how I decided
>>>>>> to work around this problem:
>>>>>>
>>>>>> [SetUp] public void SetUp() {
>>>>>>    setUpCalled = true;
>>>>>>    r1 = Resources.Acquire1();
>>>>>>    r2 = Resources.Acquire2();
>>>>>> }
>>>>>>
>>>>>> [TearDown] public void TearDown() {
>>>>>>    if (!setUpCalled) return;
>>>>>>    setUpCalled = false;
>>>>>>    Resources.Release(r1);
>>>>>>    r1 = null;
>>>>>>    Resources.Release(r2);
>>>>>>    r2 = null;
>>>>>> }
>>>>>>
>>>>>> Not the worst of solution's, but now I'm basically doing
>>>>>> NUnit's work myself. And when the methods in TearDown *can*
>>>>>> throw Exceptions they will now be reported as if they were
>>>>>> thrown in SetUp, which is just to ask for someone being
>>>>>> confused later.
>>>>>> But at least the code will release resources properly in both
>>>>>> NUnit 2.4.8 (and 2.5 alpha 3), ReSharper 4.1.933.3 (which
>>>>>> behaves like the NUnit doc says it should) and future NUnit
>>>>>> versions which might adhere to the documentation.
>>>>>>
>>>>>> So... could we instead please change the NUnit behavior so
>>>>>> that other people don't have to write workarounds like this?
>>>>>>
>>>>>> I know junit doesn't run TearDown (or didn't when I last used
>>>>>> it a year back) when SetUp throws, but junit didn't report
>>>>>> test exceptions when TearDown throwed either, so not much
>>>>>> cause to let them set the standard here...
>>>>>>
>>>>>> A better argument against might be that people didn't write
>>>>>> code to check for null in their [TearDown] and
>>>>>> [TestFixtureTearDown] methods. But then their [TearDown]
>>>>>> methods are already failing, and they'll just have to fix
>>>>>> their [TestFixtureTearDown] methods when upgrading to 2.5,
>>>> won't they?
>>>>>>
>>>>>> + Hans Christian
>>>>>>
>>>>>>
>>>>>>
>>>>>> PS: If you are wondering wth. I'm thinking wrt. acquiring 4-5
>>>>>> resources before a unit tests... Well, the resources are
>>>>>> files (with Acquire==Create and Release==Delete) and it would
>>>>>> hardly be proper to use mocked files for unit testing when
>>>>>> testing writing to the OS - that would kind of void the
>>>>>> entire testing.
>>>>>>
>>>>>> PPS: Why should a unit test need files? Well - it does.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> nunit-developer mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/nunit-developer
>>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> nunit-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-developer
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/