Re: Value and Principles of Unit Testing.

Two thoughts:

1) if you aren't finding smells in your own code that means you've stopped
learning. Have someone else look and listen to what they say.

2) there are always too many places to look. Remember the "Boyscout Rule."
Remember Michael Feathers' rules for how to prioritize legacy refactoring.
Remember to improve incrementally.
On May 26, 2013 8:10 PM, "John Carter" <[email protected]> wrote:

> **
>
>
> The following is aimed at a my own team, but before I inflict it on them, I
> thought I would run past the wise folks of this forum.
>
> Suggestions, comments, flames all welcome.
>
> What I write below is somewhat Opinionated, slightly controversial, and
> hence potentially Combustible Material.
>
> I have no apologies for that.
>
> However, that makes me all the more determine to get it Right!
>
> The audience for the following document is a team of 20-30 experienced
> embedded C developers.
>
> The large body of embedded C software they are working on, has slowly being
> growing unit test coverage and has now reached around 16% by SLOC.
>
> I'm now calling for us rethink our Unit Testing.
>
> I have been revisiting some of our coverage, and groaning with
> embarrassment at some of the stupid things I did earlier on.
>
> I have been reading and rereading books on the subject, especially Gerard
> Meszaros' "xUnit Test Patterns, Refactoring Test Code".
>
> I been cringing as I wade through the list of "test smells", I recognize
> them all.
>
> I think we can do better. A lot better. I have learnt a lot since I first
> introduced Unit Testing, the Industry has learnt a lot.
>
> It is time we took on board that learning.
>
> The Value and Principles of Unit Testing. The Value of Unit Testing
>
> We have Unit Tests for these reasons...
>
> - They are our "always up to date" executable documentation on how to
> run our code, and executable specification of what it should do.
> - They are by far the most productive compile / run / test / debug cycle
> - They provide, by many orders of magnitude, the best and most thorough
> test coverage of all test techniques.
> - They provide direct feedback on where the bug is (Defect
> Localization), unlike other test techniques which merely indicate the
> presence of defects.
> - They are our safety net to allow change, whether for new features, or
> refactoring to reduce technical debt.
>
> To maintain the value of a Unit Test suite ...
>
> - We need to run them on every changeset. ie. They must be small and
> faster enough not to impose an intolerable burden on checkins.
> - They must be low maintenance (simple, easy to understand)
> - and robust (do not break for any reason other than change in what is
> directly under test).
> - They must be repeatable. ie. Every test run on the same code gives
> exactly the same result.
> - They must be incapable of adding risk to the customer.
>
> Principles of Unit Testing. Write the Test First!
>
> - Unit Tests save us a lot of debugging effort... but only if we haven't
> already debugged it!
> - Writing the Tests encourages Design for Testability.... Which is a
> Very Good Thing!
>
> Design for Testability and Minimize Untestable Code.
>
> In the past we wrote code as if tests didn't exist. This urgently needs to
> change.
>
> *In future we MUST change our Designs to be Testable!*
>
> Why? Because testable code is lightly coupled code. Testable code is
> understandable code. Testable code is re-usable code. Testable code is
> simpler code.
> Communicate Intent
>
> Tests are executable documentation and "worked examples". Make sure your
> tests are *Good* documents!
> Test (EACH PART IN ISOLATION) Exactly what you Fly, Integrate and Fly
> EXACTLY what you tested.
>
> ie. The difference between what you test and what you put into production
> is what it is linked to, not what it is.
>
> For the Code Under Test, exactly the same bytes should be fed by the
> preprocessor to the compiler for the test run, as are fed into production.
> Keep the Tests Independent
>
> Each test should be able to be run independently of all others.
> Test the smallest part you can.
>
> The larger part of production code under test, the weaker your test
> coverage is and the more fragile your test is.
> Minimize Test Overlap, Verify One Condition Per Test.
>
> There should only one reason for a test to fail (Defect Localization) and
> only one test to fail due to a defect (Effective, Targeted Testing).
>
> In the past we have done "Paranoid" testing... on every test verify every
> condition possible with the set up, Code Under Test and tear down. This
> makes our tests fragile with respect to change, stiff to change, and
> destroy's defect localization.
>
> In future we should aim to be testing, in each test, a single, specific
> aspect of the Code Under Test. Ideally, if that single aspect is broken,
> only one test should fail, and that should be the only reason that that
> test could fail.
> Keep Test Logic Out of Production Code.
>
> It increases risks to your customer, and doesn't test what you fly. There
> are better ways of doing this.
> Test the System Under Test, not the Framework!
>
> Far too many of our unit tests are exploring how and whether the framework
> works. This results in much additional complexity and clouds the intention
> of the test
>
> These "test the framework tests" need to be actively discarded.
>
> If you mistrust the framework, add tests to the unit tests for the
> framework.
>
> No Unit Test, except those explicitly testing the threading infrastructure,
> should ever create another thread!
>
> If you have doubts about the correctness of the threading infrastructure...
> you are more than welcome to explicitly test it. Just don't do it in any
> test that isn't explicitly solely aimed at testing the threading
> infrastructure!.
>
> In many places we have tests that verify whether we understand how I/O
> works in specific contexts. In future we should create, use and ACTIVELY
> DISCARD such tests!
>
> We should aim to capturing the results of such spikes only as tests of the
> permissible range of inputs and possible responses. We can then form tests
> that check we invoke such interfaces correctly, and handle all possible
> responses correctly.
> They may all be functions.... but Pure, Stateful, Services and I/O
> functions are very different sort of functions requiring very different
> sort of tests. PURE functions
>
> A function that modifies nothing and always returns the same answer given
> the same parameters is called a pure function.
>
> These have many nice mathematical properties and are the easiest to test,
> to analyse, to reuse and to optimize. Attempt to move as much code as
> possible into pure functions.
>
> "const" is a keyword that always make me relax and feel less stressed.
> "const" is a remarkable powerful statement. Use it where ever possible.
>
> Tests of pure functions are all about the results, never about functions
> they may invoke. ie. Never mock a pure subfunction that a pure function may
> use for implementation. Use the real one.
> Stateful functions
>
> Stateful functions results depend on some hidden internal state, or as a
> side effect modify some hidden internal state.
>
> However, unlike a service, if you can set up exactly the same starting
> state, the stateful function will have exactly the same behaviour every
> time.
>
> Often a stateful function can be refactored into a pure function where ....
>
> 1. the state is passed in as a const parameter.
> 2. The result can be assigned to the state.
>
> The best use for stateful functions is to encapsulate a bundle of related
> state (into a class). These functions (or methods) should guarantee that
> required relationships (invariants) between those items are maintained.
>
> Where you have a collection of functions (or methods) encapsulating state
> (or a class) the best unit testing strategy is...
>
> 1. Construct the object (possibly via a common test fixture).
> 2. Propagate the object to the required state via a public method (which
> has been tested in some other test)
> 3. Invoke the stateful function under test.
> 4. Verify the result.
> 5. Discard the object (possibly via a common tear down function).
> 6. Keep your tests independent, DO NOT succumb to the temptation to
> reuse this object in a subsequent test. Fragility and complexity lies that
> way.
>
> Preferably DO NOT assert on private, hidden state of the implementation,
> otherwise you couple your test to that particular implementation and
> representation, rather than the desired behaviour of the class.
> Services
>
> A service function is one whose full effect, and precise result, varies
> with things like timing and inputs and threads and loads in a too complex a
> manner to be specified in a simple test.
>
> Testing services is all about testing interface specifications. The
> services dependencies (unless PURE) must be explicitly cut and controlled
> by the test harness.
>
> We have had a strong natural inclination to test whether "this(...)" calls
> "that(...)" correctly by letting "this(...)" call "that(...)" and seeing if
> the right thing happened.
>
> However, this mostly tests whether the compiler can correctly invoke
> functions (yup, it can) rather than whether "this(...)" and "that(...)"
> agree on the interface.
>
> Code grown and tested in this manner is fragile and unreusable as it "grew
> up together". All kinds of implicit, hidden, undocumented coupling and
> preconditions may exist.
>
> We need to explicitly test our conformance to interfaces, and rely on the
> compiler to be correct.
>
> 1. Does the client make valid requests to the service?
> 2. Can the service handle those requests?
> 3. Can the client handle all possible responses from the service?
> 4. Can the service make every possible response?
>
> I/O functions
>
> I/O functions are hard because...
>
> - I/O primitives tend to be impossible to mock. (The unit test framework
> uses them at some level.)
> - Care needs to be taken to prevent tests from overwriting / corrupting
> production files.
> - File / devices tend to be "one per system" thing, so running tests in
> parallel can be problematic.
> - Some I/O is byte stream representations of highly constrained and
> complex objects.
> - Some inputs can contain line noise and/or malicious hand crafted
> attack vectors.
> - Some I/O are complex devices with sensitive time and context dependent
> behaviour.
>
> To cope with these facts, we need to alter our designs to make them
> testable. (Turns out this is actually A Good Thing!)
>
> 1. Placing a thin Facade over I/O primitives gives us a point where we
> can mock.
> 2. Always use relative path names on an explicit absolute base path
> (never use the current working directory). Pass the base path in as a
> parameter.
> 3. Move calls to "open()", "connect()", "socket()" up the call graph,
> pass the resulting io handle as a parameter.
> 4. Decouple the I/O from representation. ie. Serialization is about
> data, strings and buffers. Not I/O. Implement and test serialization
> separately from the task of placing the string on the wire.
> 5. Where input can be noisy or malicious, it is all the more important
> to be able throw random and malicious test vectors at your code!
> 6. NEVER use "sleep()" or timers in a unit test test harness. If you
> doing that, you're doing unit testing wrong!
> 7. Design your code so your test harness can explicitly (with malice
> aforethought) sequence the order of arrival of events.
> 8. Use patterns like the Humble Object and/or Reactor to move the I/O to
> the highest level in the call graph.
> 9. Focus on testing that the I/O primitives are invoked correctly, and
> that we can handle the response. Rely on the I/O primitives working.
>
> Conclusions
>
> - As our Test Coverage has grown, "smells" and weaknesses in our tests
> have emerged and need to be addressed.
> - We need to emphasize Design for Test.
> - We need to highlight the differences between pure, stateful, service
> and I/O functions and adjust our test strategies accordingly.
> - Unit Tests are about Defect Localization, not paranoia.
>
> --
> John Carter Phone : (64)(3) 358 6639
> Tait Electronics Fax : (64)(3) 359 4632
> PO Box 1645 Christchurch Email : [email protected]
> New Zealand
>
> --
>
> ------------------------------
> This email, including any attachments, is only for the intended recipient.
> It is subject to copyright, is confidential and may be the subject of
> legal
> or other privilege, none of which is waived or lost by reason of this
> transmission.
> If you are not an intended recipient, you may not use, disseminate,
> distribute or reproduce such email, any attachments, or any part thereof.
> If you have received a message in error, please notify the sender
> immediately and erase all copies of the message and any attachments.
> Unfortunately, we cannot warrant that the email has not been altered or
> corrupted during transmission nor can we guarantee that any email or any
> attachments are free from computer viruses or other conditions which may
> damage or interfere with recipient data, hardware or software. The
> recipient relies upon its own procedures and assumes all risk of use and
> of
> opening any attachments.
> ------------------------------
>
> [Non-text portions of this message have been removed]
>
>  
>


[Non-text portions of this message have been removed]



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/testdrivendevelopment/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/testdrivendevelopment/join
    (Yahoo! ID required)

<*> To change settings via email:
    [email protected] 
    [email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/