Re: Value and Principles of Unit Testing.

Jonh,

In the GOOS list someone asked it:

>>Do you still feel like the Unit tests have their own value to make them
worth it?

The context was that, for him, may be only ATs (Acceptance Tests) was
enough.

Follows my answer why I still value unit tests. As you write about the
value of unit tests too, may be interest you. Some points may be slightly
controversial.

-- 
Abraços,
Josué
http://twitter.com/josuesantos


>>So, have you ever had a time where you were tempted to JUST write the
AT's, and NOT tests lower down?
...
>>When you know that you have an acceptance test covering the code, does
that affect when you make a decision to write a unit test or not?

No. When I write one AT I always want to write the unit tests too. But
there was rare cases were we do not follow this rule. Follows one case I am
remembering:

In some systems we used GWT and DTOs to transfer data between the browser
and the server. And we need to copy the data from DTO to server side
classes (Entities). To do this we created converters and its respective
unit testes. Example: ClientDTO, Client, ClientConverter,
ClientConverterTests. Each new converter has its unit tests.

As the DTOs and Entities in many cases had properties with the same names,
we created a generic converter with, of course, unit tests. We could create
unit tests to prove that the conversion was possible. The only way that the
conversion will not work is when the properties names did not match, for
example, DTO has getName() and ENTITY has setNname(). For this case, the
team judged that the risk of this kind of error was very low and, if it
happens, the AT would take it.

Even in this case, if it was decided to create the tests, it would be no
problem. It would serve as a unit test documentation of the dependency
between the DTO and the Entity.

>>Do you still feel like the Unit tests have their own value to make them
worth it?

Yes. Mainly to design. May be you are a good programmer and you can test
drive only using ATs and keep the design nice. Maybe I could also. I don´t
know I never did the experience. But with beginners to intermediate
experienced developers I do not have doubt that it helps a lot. I coach and
help teams in my company in this area and always some came to me and ask:
"I think this test is not good. What do you think about". And generally we
change the design (break a big class in smaller ones(SRP), eliminate
duplication etc.) to what we think is a better design. The pain in the unit
test is an indication of bad design. I think the GOOS book talk with much
more details about it. :)

Others advantages related to unit testing:

FAST FEEDBACK

If you use only ATs to develop you will stay with a red (broken) test for a
bigger time than if you test each class in isolation. I think I would not
like this kind of development: red code refactor; red  code refactor; red
code refactor...green.  I think it is more pleasurable something like: red
green refactor; red green refactor; ...green. The TDD mantra. I have fun
with it. Each little green is a fast feedback of progress. And it is good.

FAST FEEDBACK II

Not everyone will create ATs to run as fast as the unit tests. In this case
the unit tests are essential. For example there is a system here where the
direct calls tests uses the real database. This tests executes in ten
minutes. The web tests in 2 hours. And the unit tests in less than 5s.

DESIGN(again)

Note everyone will create two kinds of ATs as we do. In many projects they
only automate AT at the external interface(Web for example). In this case
the rule to create unit tests to every code helps inexperienced developers
to not put logic in the interface component. It happened here: "How I would
test it if it is in the interface and I cant instantiate the interface?".
Answer: "Extract all the logic and code as possible from the view. Use, for
example the presenter pattern. Hey, what this sql code is doing here?! Have
you ever heard about repositories or DAOs?"

FASTER PROBLEM DETECTION

If you are creating isolated unit testing ala GOOS (interactive tests) then
when your build breaks there was, theoretically, only one (or few ones)
tests in red state indicating very precisely the error origin. If you use
only ATs all the stack of objects used can be a potential candidate to the
problem. When you work alone it is not a big problem, as probably the last
updated place is the place of the error, but when you work on a team it
starts to make difference.

DEFECT PREVENTION

"The act of “designing test” rather than the act of testing (i.e. knowing
what to test and how to test) is one of the best known defect
preventers[33]. Well I am not so smart to state this, this statement is
here:

"Analysis and Quantification of Test Driven Development Approach - Boby
George
A thesis submitted to the Graduate Faculty of North Carolina State
University
in partial fulfillment of the requirements for the Degree of Master of
Science"

and [33]: Beizer, B., Software Testing Techniques. ITP, 2nd Edition. 1990.

I have a feeling that it is true. :)

DOCUMENTATION

The unit tests documents the api to each class used in the project. So if
you are going to maintain a project that is new to you,  look at the unit
tests to know how to use the classes. For example, when a search is made in
the ClientSearcher.search(clienteId)  and there no itens what happens? It
returns null? It returns an empty list? It throws an
ClientNotFoundException? Look in the unit tests and discover the answer.

SIZE/COMPLEXITY METRICS

The number of unit tests can be o good indicator of the size and complexity
of an application. Suppose an application with no restriction of time to do
a client search:

Story: Search for one client
Given a client with name Josué Santos is registered
When search by name for Josué Santos
Then should find the Josué Santos

And now we change the application to restrict the time to do the
search(adding a new story):

Story: Search for one client must be in less than a one second
Given there are a million clients registered.
And a client with name Josué Santos is registered
When search by name for Josué Santos
Then should search in less than a second

The only new AT that is added verifies the time that takes to finish the
search. But probably much more unit tests is created to build the solution.
If you will use cache in the client side you will have unit tests do this
client side code. If you also will write caching in the server side you
will have unit tests to this too. So using only ATs to measure the
size/complexity of an application I think is not a good idea. See more
about this metrics using ATs e UTs in this paper(Agile Metrics at the
Israeli Air Force) :
http://www.davidtalby.com/papers/Agile_Metrics_AgileUnited05.pdf

Side note: Here we use function points :( to pay the outsourced
development. This paper gives me an idea to perhaps change to use tests
points. But it is to a future far far away. If possible.

ENABLE BETTER TEAM WORK

Suppose we are in a team focused to finish the first AT from the first
story. Suppose the following stack (C is class) AT->C1->C2.  You are
working in C2. If we use only the AT to drive development we are using the
feedback from the error that is presented when AT is executed. But this
error can change in your next update because of changes made in C1 by other
team member.  And you will be distracted by a thing that is not your focus
on the moment.

Other problem. Suppose AT->C1->C2. You are working on C1 and C2 is a class
that is supposed to be finished. If some change in  C2 breaks it, your work
in C1 will be injured because you need C2 working properly to finish C1 as
you are only using the AT as guide, and ATs needs the real classes. If we
were using unit tests and mock to C2 this would not be a problem to finish
C1.


ATs DON´T GUIDE DEVELOPMENT

As we are using only ATs to guide the development we will probably use the
bottom-up strategy. So if C1->C2->C3 then we will build C3, then C2 and
then C1. And ironically, the ATs will drive less the code than when you use
unit tests ala GOSS. The first code you build (C3) is not directly related
wit the AT. And so may be you are doing code that is not required by the
AT. When you use need driven development (GOOS) all the code is justified
in a top down way by the ATs.

I don´t know if all what I said is right. It is just what I think.





On Mon, May 27, 2013 at 12:10 AM, John Carter <[email protected]>wrote:

> **
>
>
> The following is aimed at a my own team, but before I inflict it on them, I
> thought I would run past the wise folks of this forum.
>
> Suggestions, comments, flames all welcome.
>
> What I write below is somewhat Opinionated, slightly controversial, and
> hence potentially Combustible Material.
>
> I have no apologies for that.
>
> However, that makes me all the more determine to get it Right!
>
> The audience for the following document is a team of 20-30 experienced
> embedded C developers.
>
> The large body of embedded C software they are working on, has slowly being
> growing unit test coverage and has now reached around 16% by SLOC.
>
> I'm now calling for us rethink our Unit Testing.
>
> I have been revisiting some of our coverage, and groaning with
> embarrassment at some of the stupid things I did earlier on.
>
> I have been reading and rereading books on the subject, especially Gerard
> Meszaros' "xUnit Test Patterns, Refactoring Test Code".
>
> I been cringing as I wade through the list of "test smells", I recognize
> them all.
>
> I think we can do better. A lot better. I have learnt a lot since I first
> introduced Unit Testing, the Industry has learnt a lot.
>
> It is time we took on board that learning.
>
> The Value and Principles of Unit Testing. The Value of Unit Testing
>
> We have Unit Tests for these reasons...
>
> - They are our "always up to date" executable documentation on how to
> run our code, and executable specification of what it should do.
> - They are by far the most productive compile / run / test / debug cycle
> - They provide, by many orders of magnitude, the best and most thorough
> test coverage of all test techniques.
> - They provide direct feedback on where the bug is (Defect
> Localization), unlike other test techniques which merely indicate the
> presence of defects.
> - They are our safety net to allow change, whether for new features, or
> refactoring to reduce technical debt.
>
> To maintain the value of a Unit Test suite ...
>
> - We need to run them on every changeset. ie. They must be small and
> faster enough not to impose an intolerable burden on checkins.
> - They must be low maintenance (simple, easy to understand)
> - and robust (do not break for any reason other than change in what is
> directly under test).
> - They must be repeatable. ie. Every test run on the same code gives
> exactly the same result.
> - They must be incapable of adding risk to the customer.
>
> Principles of Unit Testing. Write the Test First!
>
> - Unit Tests save us a lot of debugging effort... but only if we haven't
> already debugged it!
> - Writing the Tests encourages Design for Testability.... Which is a
> Very Good Thing!
>
> Design for Testability and Minimize Untestable Code.
>
> In the past we wrote code as if tests didn't exist. This urgently needs to
> change.
>
> *In future we MUST change our Designs to be Testable!*
>
> Why? Because testable code is lightly coupled code. Testable code is
> understandable code. Testable code is re-usable code. Testable code is
> simpler code.
> Communicate Intent
>
> Tests are executable documentation and "worked examples". Make sure your
> tests are *Good* documents!
> Test (EACH PART IN ISOLATION) Exactly what you Fly, Integrate and Fly
> EXACTLY what you tested.
>
> ie. The difference between what you test and what you put into production
> is what it is linked to, not what it is.
>
> For the Code Under Test, exactly the same bytes should be fed by the
> preprocessor to the compiler for the test run, as are fed into production.
> Keep the Tests Independent
>
> Each test should be able to be run independently of all others.
> Test the smallest part you can.
>
> The larger part of production code under test, the weaker your test
> coverage is and the more fragile your test is.
> Minimize Test Overlap, Verify One Condition Per Test.
>
> There should only one reason for a test to fail (Defect Localization) and
> only one test to fail due to a defect (Effective, Targeted Testing).
>
> In the past we have done "Paranoid" testing... on every test verify every
> condition possible with the set up, Code Under Test and tear down. This
> makes our tests fragile with respect to change, stiff to change, and
> destroy's defect localization.
>
> In future we should aim to be testing, in each test, a single, specific
> aspect of the Code Under Test. Ideally, if that single aspect is broken,
> only one test should fail, and that should be the only reason that that
> test could fail.
> Keep Test Logic Out of Production Code.
>
> It increases risks to your customer, and doesn't test what you fly. There
> are better ways of doing this.
> Test the System Under Test, not the Framework!
>
> Far too many of our unit tests are exploring how and whether the framework
> works. This results in much additional complexity and clouds the intention
> of the test
>
> These "test the framework tests" need to be actively discarded.
>
> If you mistrust the framework, add tests to the unit tests for the
> framework.
>
> No Unit Test, except those explicitly testing the threading infrastructure,
> should ever create another thread!
>
> If you have doubts about the correctness of the threading infrastructure...
> you are more than welcome to explicitly test it. Just don't do it in any
> test that isn't explicitly solely aimed at testing the threading
> infrastructure!.
>
> In many places we have tests that verify whether we understand how I/O
> works in specific contexts. In future we should create, use and ACTIVELY
> DISCARD such tests!
>
> We should aim to capturing the results of such spikes only as tests of the
> permissible range of inputs and possible responses. We can then form tests
> that check we invoke such interfaces correctly, and handle all possible
> responses correctly.
> They may all be functions.... but Pure, Stateful, Services and I/O
> functions are very different sort of functions requiring very different
> sort of tests. PURE functions
>
> A function that modifies nothing and always returns the same answer given
> the same parameters is called a pure function.
>
> These have many nice mathematical properties and are the easiest to test,
> to analyse, to reuse and to optimize. Attempt to move as much code as
> possible into pure functions.
>
> "const" is a keyword that always make me relax and feel less stressed.
> "const" is a remarkable powerful statement. Use it where ever possible.
>
> Tests of pure functions are all about the results, never about functions
> they may invoke. ie. Never mock a pure subfunction that a pure function may
> use for implementation. Use the real one.
> Stateful functions
>
> Stateful functions results depend on some hidden internal state, or as a
> side effect modify some hidden internal state.
>
> However, unlike a service, if you can set up exactly the same starting
> state, the stateful function will have exactly the same behaviour every
> time.
>
> Often a stateful function can be refactored into a pure function where ....
>
> 1. the state is passed in as a const parameter.
> 2. The result can be assigned to the state.
>
> The best use for stateful functions is to encapsulate a bundle of related
> state (into a class). These functions (or methods) should guarantee that
> required relationships (invariants) between those items are maintained.
>
> Where you have a collection of functions (or methods) encapsulating state
> (or a class) the best unit testing strategy is...
>
> 1. Construct the object (possibly via a common test fixture).
> 2. Propagate the object to the required state via a public method (which
> has been tested in some other test)
> 3. Invoke the stateful function under test.
> 4. Verify the result.
> 5. Discard the object (possibly via a common tear down function).
> 6. Keep your tests independent, DO NOT succumb to the temptation to
> reuse this object in a subsequent test. Fragility and complexity lies that
> way.
>
> Preferably DO NOT assert on private, hidden state of the implementation,
> otherwise you couple your test to that particular implementation and
> representation, rather than the desired behaviour of the class.
> Services
>
> A service function is one whose full effect, and precise result, varies
> with things like timing and inputs and threads and loads in a too complex a
> manner to be specified in a simple test.
>
> Testing services is all about testing interface specifications. The
> services dependencies (unless PURE) must be explicitly cut and controlled
> by the test harness.
>
> We have had a strong natural inclination to test whether "this(...)" calls
> "that(...)" correctly by letting "this(...)" call "that(...)" and seeing if
> the right thing happened.
>
> However, this mostly tests whether the compiler can correctly invoke
> functions (yup, it can) rather than whether "this(...)" and "that(...)"
> agree on the interface.
>
> Code grown and tested in this manner is fragile and unreusable as it "grew
> up together". All kinds of implicit, hidden, undocumented coupling and
> preconditions may exist.
>
> We need to explicitly test our conformance to interfaces, and rely on the
> compiler to be correct.
>
> 1. Does the client make valid requests to the service?
> 2. Can the service handle those requests?
> 3. Can the client handle all possible responses from the service?
> 4. Can the service make every possible response?
>
> I/O functions
>
> I/O functions are hard because...
>
> - I/O primitives tend to be impossible to mock. (The unit test framework
> uses them at some level.)
> - Care needs to be taken to prevent tests from overwriting / corrupting
> production files.
> - File / devices tend to be "one per system" thing, so running tests in
> parallel can be problematic.
> - Some I/O is byte stream representations of highly constrained and
> complex objects.
> - Some inputs can contain line noise and/or malicious hand crafted
> attack vectors.
> - Some I/O are complex devices with sensitive time and context dependent
> behaviour.
>
> To cope with these facts, we need to alter our designs to make them
> testable. (Turns out this is actually A Good Thing!)
>
> 1. Placing a thin Facade over I/O primitives gives us a point where we
> can mock.
> 2. Always use relative path names on an explicit absolute base path
> (never use the current working directory). Pass the base path in as a
> parameter.
> 3. Move calls to "open()", "connect()", "socket()" up the call graph,
> pass the resulting io handle as a parameter.
> 4. Decouple the I/O from representation. ie. Serialization is about
> data, strings and buffers. Not I/O. Implement and test serialization
> separately from the task of placing the string on the wire.
> 5. Where input can be noisy or malicious, it is all the more important
> to be able throw random and malicious test vectors at your code!
> 6. NEVER use "sleep()" or timers in a unit test test harness. If you
> doing that, you're doing unit testing wrong!
> 7. Design your code so your test harness can explicitly (with malice
> aforethought) sequence the order of arrival of events.
> 8. Use patterns like the Humble Object and/or Reactor to move the I/O to
> the highest level in the call graph.
> 9. Focus on testing that the I/O primitives are invoked correctly, and
> that we can handle the response. Rely on the I/O primitives working.
>
> Conclusions
>
> - As our Test Coverage has grown, "smells" and weaknesses in our tests
> have emerged and need to be addressed.
> - We need to emphasize Design for Test.
> - We need to highlight the differences between pure, stateful, service
> and I/O functions and adjust our test strategies accordingly.
> - Unit Tests are about Defect Localization, not paranoia.
>
> --
> John Carter Phone : (64)(3) 358 6639
> Tait Electronics Fax : (64)(3) 359 4632
> PO Box 1645 Christchurch Email : [email protected]
> New Zealand
>
> --
>
> ------------------------------
> This email, including any attachments, is only for the intended recipient.
> It is subject to copyright, is confidential and may be the subject of
> legal
> or other privilege, none of which is waived or lost by reason of this
> transmission.
> If you are not an intended recipient, you may not use, disseminate,
> distribute or reproduce such email, any attachments, or any part thereof.
> If you have received a message in error, please notify the sender
> immediately and erase all copies of the message and any attachments.
> Unfortunately, we cannot warrant that the email has not been altered or
> corrupted during transmission nor can we guarantee that any email or any
> attachments are free from computer viruses or other conditions which may
> damage or interfere with recipient data, hardware or software. The
> recipient relies upon its own procedures and assumes all risk of use and
> of
> opening any attachments.
> ------------------------------
>
> [Non-text portions of this message have been removed]
>
>  
>


[Non-text portions of this message have been removed]



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/testdrivendevelopment/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/testdrivendevelopment/join
    (Yahoo! ID required)

<*> To change settings via email:
    [email protected] 
    [email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/