Unit Tests for embedded C coders, what and why.

Here is a little reminder about the "what and the why" of Unit Test
for a bunch of experienced embedded C programmers.

Perhaps someone else may find it useful.


There is nothing new and special about Unit Tests.

You have always done them since you first learnt to program.

You have always written a bit of code, entered some values, looked at
the results on the screen, and said "That worked!" or more likely
"Shit! I have a bug".

The only difference is you are being asked to record, in an tiny test
harness that information of "if I enter this value and that happens,
my program is working".

Why? Because the next guy, or yourself 6 months later will no long
know what the inputs should be, nor have a clue what a correct output
looks like.

So when you, or the next guy, changes something, he merely needs to
rerun the test to check if he broke something.

A really very very simple idea.


Of course, like any Simple powerful idea... there are easy ways of
doing this and hard ways....

The world is divided into 10 kinds of people.... those who understand
binary and those who don't.


Now let's carve up the world of bugs a little.

There are integration bugs and there are programming bugs.

If the routine I have just written is defective, that's a programming bug.

If the routine I have just written incorrectly invokes other routines,
or is incorrectly invoked by other routines, that's an Integration Bug.

Unit Testing is about catching programming bugs, _not_ integration bugs.

When the only tool you have is a hammer... all the world looks like a nail.

Catching integration bugs is another problem for another tool set and
another discussion.

I'll talk about how precondition asserts and invariant checks catch
integration bugs later.

Yes, integration bugs are a largish part of our defect
count. Especially as we're accreting small chunks on new code onto a
huge ball of old.

So if you're disappointed that unit testing doesn't seem to be
catching that class of bugs, don't be.

It was never designed to.


A common error I see (and made myself at first) is to test too much at
once.

This is deadly on productivity if the code suffers from fan
out. (Fan out: if the body of code under test calls other functions
which call even more functions so the call tree fans outward until it
encompasses the whole product)

The more code we effectively have under test at once, the more set up
and initialization we have to do, and more fragile the test is.

And hence the greater the maintenance burden compared to the value.

In some cases the code under test is a leaf function. Or is perhaps a
layer or two above the leaf functions.

Then it's no problem, the call tree converges rapidly and we can test
our function and the code it calls in one go.

In other cases we have somehow cut out our code from the rest, and
just test that.

This is where mocks and stubs come in.

If we replace the functions our code under test calls with mock
versions, we can test _just_ our small chunk of code.

No setup, no complicated multi-threading, just a couple of lines of
code that does a few things, and a couple of tests to prove it works.

We can even control our mock functions to stimulate our code from
inside, or assert that it calls the lower functions with the right
values.

So there you have it. The scale of a UNIT TEST is typically a single
function. A single function that does one or two things, and one or
two tests to verify it.

Simple, easy, no hard slog.


For some reason we have this curious notion that embedded devices are
special.

That somehow we haven't tested until we have it running on the target
device.

Perhaps we imagine that code that is wrong on a desktop might
magically become right if we squeeze it into an embedded device?

Perhaps there is some truth in that... it's so hard to see anything
what's going on, we can't see it's wrong :-))



The other plaint I hear is that my code does nothing, so testing it is
a waste of time...

There is an obvious sarcastic answer to that that I won't give....

...but I understand that there is not much point in testing getters and
setters, and will admit to skipping them myself.

However, if you have a single "if", a single "?:" expression, a single
short circuit evaluated boolean expression....

What better opportunity are you going to have to exercise both
branches of execution?

I bet you a pint of beer it will be more difficult, often _much_ more
difficult, and sometimes impossible to get that coverage at system
test time.



Some additional points on Unit Testing.....

Unit Testing improves your design because it forces you to think
concretely about what this code must do.

Writing the Unit Test first improves your design because Fan Out makes
your test harder to do, so you write code with less fan out!

The Unit Test is the best, concrete, most accurate and up-to-date
documentation your code will ever have.

Don't get bogged down testing. Alternate between write test, write
code, run test, fix, write test, write code, run test,...

Once you get the urge to try it out on the real device... go ahead,
but when you spot a bug, go back to the unit test.

Write the test that will trigger the bug. Run the test to prove you
have cornered it, THEN fix it.

Why?
   * Just good experimental technique. Prove you understand the problem.

   * The existence of the bug proves you have a gap in your test
      coverage, there are probably more bugs down that path.

   * Bugs, especially when the requirements are fuzzy, have a bad habit
     of coming back time and time again over the years of
     maintenance. If you have that one cornered in a unit test, you
     will pick it up straight away next time it reappears.

   * In most embedded systems the compile, link, download to target,
     navigate through menu structure, perform actions to stimulate bug
     cycle is _long_.

     It just takes a lot of wall clock time to do that.

     A well written unit test should take under 2 seconds to hit a bug.

   * Embedded devices are cranky and uncommunicative objects. It's just
     plain easier to debug on a desktop.



John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : [email protected]
New Zealand