Value and Principles of Unit Testing.

The following is aimed at a my own team, but before I inflict it on them, I
thought I would run past the wise folks of this forum.

Suggestions, comments, flames all welcome.

What I write below is somewhat Opinionated, slightly controversial, and
hence potentially Combustible Material.

I have no apologies for that.

However, that makes me all the more determine to get it Right!

The audience for the following document is a team of 20-30 experienced
embedded C developers.

The large body of embedded C software they are working on, has slowly being
growing unit test coverage and has now reached around 16% by SLOC.

I'm now calling for us rethink our Unit Testing.

I have been revisiting some of our coverage, and groaning with
embarrassment at some of the stupid things I did earlier on.

I have been reading and rereading books on the subject, especially  Gerard
Meszaros' "xUnit Test Patterns, Refactoring Test Code".

I been cringing as I wade through the list of "test smells", I recognize
them all.

I think we can do better. A lot better. I have learnt a lot since I first
introduced Unit Testing, the Industry has learnt a lot.

It is time we took on board that learning.

The Value and Principles of Unit Testing. The Value of Unit Testing

We have Unit Tests for these reasons...

   - They are our "always up to date" executable documentation on how to
   run our code, and executable specification of what it should do.
   - They are by far the most productive compile / run / test / debug cycle
   - They provide, by many orders of magnitude, the best and most thorough
   test coverage of all test techniques.
   - They provide direct feedback on where the bug is (Defect
   Localization), unlike other test techniques which merely indicate the
   presence of defects.
   - They are our safety net to allow change, whether for new features, or
   refactoring to reduce technical debt.

 To maintain the value of a Unit Test suite ...

   - We need to run them on every changeset. ie. They must be small and
   faster enough not to impose an intolerable burden on checkins.
   - They must be low maintenance (simple, easy to understand)
   - and robust (do not break for any reason other than change in what is
   directly under test).
   - They must be repeatable. ie. Every test run on the same code gives
   exactly the same result.
   - They must be incapable of adding risk to the customer.

Principles of Unit Testing. Write the Test First!

   - Unit Tests save us a lot of debugging effort... but only if we haven't
   already debugged it!
   - Writing the Tests encourages Design for Testability.... Which is a
   Very Good Thing!

Design for Testability and Minimize Untestable Code.

In the past we wrote code as if tests didn't exist. This urgently needs to
change.

*In future we MUST change our Designs to be Testable!*

Why? Because testable code is lightly coupled code. Testable code is
understandable code. Testable code is re-usable code. Testable code is
simpler code.
Communicate Intent

Tests are executable documentation and "worked examples". Make sure your
tests are *Good* documents!
Test (EACH PART IN ISOLATION) Exactly what you Fly, Integrate and Fly
EXACTLY what you tested.

ie. The difference between what you test and what you put into production
is what it is linked to, not what it is.

For the Code Under Test, exactly the same bytes should be fed by the
preprocessor to the compiler for the test run, as are fed into production.
Keep the Tests Independent

Each test should be able to be run independently of all others.
Test the smallest part you can.

The larger part of production code under test, the weaker your test
coverage is and the more fragile your test is.
Minimize Test Overlap, Verify One Condition Per Test.

There should only one reason for a test to fail (Defect Localization) and
only one test to fail due to a defect (Effective, Targeted Testing).

In the past we have done "Paranoid" testing... on every test verify every
condition possible with the set up, Code Under Test and tear down. This
makes our tests fragile with respect to change, stiff to change, and
destroy's defect localization.

In future we should aim to be testing, in each test, a single, specific
aspect of the Code Under Test. Ideally, if that single aspect is broken,
only one test should fail, and that should be the only reason that that
test could fail.
Keep Test Logic Out of Production Code.

It increases risks to your customer, and doesn't test what you fly. There
are better ways of doing this.
Test the System Under Test, not the Framework!

Far too many of our unit tests are exploring how and whether the framework
works. This results in much additional complexity and clouds the intention
of the test

These "test the framework tests" need to be actively discarded.

If you mistrust the framework, add tests to the unit tests for the
framework.

No Unit Test, except those explicitly testing the threading infrastructure,
should ever create another thread!

If you have doubts about the correctness of the threading infrastructure...
you are more than welcome to explicitly test it. Just don't do it in any
test that isn't explicitly solely aimed at testing the threading
infrastructure!.

In many places we have tests that verify whether we understand how I/O
works in specific contexts. In future we should create, use and ACTIVELY
DISCARD such tests!

We should aim to capturing the results of such spikes only as tests of the
permissible range of inputs and possible responses. We can then form tests
that check we invoke such interfaces correctly, and handle all possible
responses correctly.
They may all be functions.... but Pure, Stateful, Services and I/O
functions are very different sort of functions requiring very different
sort of tests. PURE functions

A function that modifies nothing and always returns the same answer given
the same parameters is called a pure function.

These have many nice mathematical properties and are the easiest to test,
to analyse, to reuse and to optimize. Attempt to move as much code as
possible into pure functions.

"const" is a keyword that always make me relax and feel less stressed.
"const" is a remarkable powerful statement. Use it where ever possible.

Tests of pure functions are all about the results, never about functions
they may invoke. ie. Never mock a pure subfunction that a pure function may
use for implementation. Use the real one.
Stateful functions

Stateful functions results depend on some hidden internal state, or as a
side effect modify some hidden internal state.

However, unlike a service, if you can set up exactly the same starting
state, the stateful function will have exactly the same behaviour every
time.

Often a stateful function can be refactored into a pure function where ....

   1. the state is passed in as a const parameter.
   2. The result can be assigned to the state.

 The best use for stateful functions is to encapsulate a bundle of related
state (into a class). These functions (or methods) should guarantee that
required relationships (invariants) between those items are maintained.

Where you have a collection of functions (or methods) encapsulating state
(or a class) the best unit testing strategy is...

   1. Construct the object (possibly via a common test fixture).
   2. Propagate the object to the required state via a public method (which
   has been tested in some other test)
   3. Invoke the stateful function under test.
   4. Verify the result.
   5. Discard the object (possibly via a common tear down function).
   6. Keep your tests independent, DO NOT succumb to the temptation to
   reuse this object in a subsequent test. Fragility and complexity lies that
   way.

 Preferably DO NOT assert on private, hidden state of the implementation,
otherwise you couple your test to that particular implementation and
representation, rather than the desired behaviour of the class.
Services

A service function is one whose full effect, and precise result, varies
with things like timing and inputs and threads and loads in a too complex a
manner to be specified in a simple test.

Testing services is all about testing interface specifications. The
services dependencies (unless PURE) must be explicitly cut and controlled
by the test harness.

We have had a strong natural inclination to test whether "this(...)" calls
"that(...)" correctly by letting "this(...)" call "that(...)" and seeing if
the right thing happened.

However, this mostly tests whether the compiler can correctly invoke
functions (yup, it can) rather than whether "this(...)" and "that(...)"
agree on the interface.

Code grown and tested in this manner is fragile and unreusable as it "grew
up together". All kinds of implicit, hidden, undocumented coupling and
preconditions may exist.

We need to explicitly test our conformance to interfaces, and rely on the
compiler to be correct.

   1. Does the client make valid requests to the service?
   2. Can the service handle those requests?
   3. Can the client handle all possible responses from the service?
   4. Can the service make every possible response?

I/O functions

I/O functions are hard because...

   - I/O primitives tend to be impossible to mock. (The unit test framework
   uses them at some level.)
   - Care needs to be taken to prevent tests from overwriting / corrupting
   production files.
   - File / devices tend to be "one per system" thing, so running tests in
   parallel can be problematic.
   - Some I/O is byte stream representations of highly constrained and
   complex objects.
   - Some inputs can contain line noise and/or malicious hand crafted
   attack vectors.
   - Some I/O are complex devices with sensitive time and context dependent
   behaviour.

 To cope with these facts, we need to alter our designs to make them
testable. (Turns out this is actually A Good Thing!)

   1. Placing a thin Facade over I/O primitives gives us a point where we
   can mock.
   2. Always use relative path names on an explicit absolute base path
   (never use the current working directory). Pass the base path in as a
   parameter.
   3. Move calls to "open()", "connect()", "socket()" up the call graph,
   pass the resulting io handle as a parameter.
   4. Decouple the I/O from representation. ie. Serialization is about
   data, strings and buffers. Not I/O. Implement and test serialization
   separately from the task of placing the string on the wire.
   5. Where input can be noisy or malicious, it is all the more important
   to be able throw random and malicious test vectors at your code!
   6. NEVER use "sleep()" or timers in a unit test test harness. If you
   doing that, you're doing unit testing wrong!
   7. Design your code so your test harness can explicitly (with malice
   aforethought) sequence the order of arrival of events.
   8. Use patterns like the Humble Object and/or Reactor to move the I/O to
   the highest level in the call graph.
   9. Focus on testing that the I/O primitives are invoked correctly, and
   that we can handle the response. Rely on the I/O primitives working.

Conclusions

   - As our Test Coverage has grown, "smells" and weaknesses in our tests
   have emerged and need to be addressed.
   - We need to emphasize Design for Test.
   - We need to highlight the differences between pure, stateful, service
   and I/O functions and adjust our test strategies accordingly.
   - Unit Tests are about Defect Localization, not paranoia.




-- 
John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                       Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch       Email : [email protected]
New Zealand

-- 

------------------------------
This email, including any attachments, is only for the intended recipient. 
It is subject to copyright, is confidential and may be the subject of legal 
or other privilege, none of which is waived or lost by reason of this 
transmission.
If you are not an intended recipient, you may not use, disseminate, 
distribute or reproduce such email, any attachments, or any part thereof. 
If you have received a message in error, please notify the sender 
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or 
corrupted during transmission nor can we guarantee that any email or any 
attachments are free from computer viruses or other conditions which may 
damage or interfere with recipient data, hardware or software. The 
recipient relies upon its own procedures and assumes all risk of use and of 
opening any attachments.
------------------------------


[Non-text portions of this message have been removed]



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/testdrivendevelopment/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/testdrivendevelopment/join
    (Yahoo! ID required)

<*> To change settings via email:
    [email protected] 
    [email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/