Re: RE: Tool to create assertions from a running Java application

There is a siren that I call 100% code coverage.

You hear the siren call most strongly from practitioners in those languages
that have dynamic/weak typing, a global variable scope and are interpreted.

When you are dealing with a static/semi-strongly typed language such as
Java, which is compiled and does not have a real global variable scope
(closest thing is System.getProperties() which can only hold String
instances), the compiler itself ensures that 100% of lines are subject to
some tests (i.e. namely is it syntactically correct and does it match the
typing rules)... on top of which you can add more "tests" using tooling
such as findbugs, PMD and checkstyle (assuming you use a zero-failures
policy)

Is that enough tests for Java code... hell no... but you have 100% code
coverage... the code coverage you measure with tooling such as
cobertura/clover/emma/jacoco is the additional coverage you get from the
additional tests you have written.

When a dynamic typing interpretive language person asks: "are you
comfortable releasing your 10,000 lines of code with 1% of those lines not
tested? that's 100 lines without tests?" they are asking because there
could be a syntax or type error in one of those 100 lines that causes their
code to blow up. I would not be comfortable with such.

With a compiled static typing language, however, the answer is "yes we do
have tests... the compiler gives us tests... they may not be our best
tests... but we have tests... shall we take a look at all your tests and
see how good they are in order to get to 100% coverage?"

My point is this:

With a language like Java, 100% code coverage is not the goal. Quality
tests is the goal.

It is easy to "generate" tests from code... the kind of tests you end up
with will verify that the code behaves the same as before... so when you
change *anything meaningful* some tests will fail... only now you don't
know why the tests are failing... is it because you changed the code to
match the new requirement... or is it because you broke something else in
the process?

Quality tests are about meaningful test names and small test case size with
at most 1-2 asserts per test case.

The test name should tell you what is being tested and why. The 1-2 asserts
(should really be just 1) should validate the actual outcome.

@Test
void lookUpWidgetById_nullId() {
  assertThat(instance.lookup(null), is(nullValue());
}

@Test
void lookUpWidgetById_standardWidget() {
  assertThat(instance.lookup(STANDARD_WIDGET_ID),
is(hasProperty("id",STANDARD_WIDGET_ID)));
}

is a better quality of tests than

@Test
void lookUpWidgetById() {
  assertThat(instance.lookup(null), is(nullValue());
  assertThat(instance.lookup(STANDARD_WIDGET_ID),
is(hasProperty("id",STANDARD_WIDGET_ID)));
}

Similarly

@Test
void manchuify_hasFrongles() {
  Foo instance = new Foo();
  assertThat(instance.getFrongleCount(), is(0));
  instance.manchuify();
  assertThat(instance.getFrongleCount(), is(greaterThan(0)));
}

@Test
void manchuify_isRed() {
  Foo instance = new Foo();
  assertThat(instance.getColor(), not(is(Color.RED)));
  instance.manchuify();
  assertThat(instance.getColor(), is(Color.RED));
}

is better quality than

@Test
void manchuify() {
  Foo instance = new Foo();
  assertThat(instance.getFrongleCount(), is(0));
  assertThat(instance.getColor(), not(is(Color.RED)));
  instance.manchuify();
  assertThat(instance.getFrongleCount(), is(greaterThan(0)));
  assertThat(instance.getColor(), is(Color.RED));
}

because by splitting out the tests we know that these are individual
requirements... in addition we also know the scope of the problem... if
something happens the Foo.manchuify() method that it no longer makes the
Foo instances red, in both cases we get two failing tests... but the better
quality test tells us exactly what is wrong.

With automated test generation based off either the compiled code or the
running system, the tool cannot tell you why things are the way they are,
so hence the tool cannot give the tests good names, and cannot
differentiate the related elements... the frongle count being non zero and
the colour being red are strongly correlated, so how can the system know
that these are a side-effect of the original spec and that the new spec
allows for non-red Foo instances after manchuification?

If you are a smart developer, you will understand all of the above... and
you will be saying... this is just a tool... I am the person who will step
in and create the context... I just want some automation to take out the
grunt work...

Well that is fine as long as you don't ever face a manager who fears the
even 1 untested line of code in their source release, or who receives an
edict from the on-high CTO (who was raised on dynamic interpreted
languages) and now has to get to 100% coverage... sees the tooling and sees
it as a route to 100% coverage...

The end effect of that tooling is that the entire code base is frozen in
place, as any change results in a complete cascade of test failures... and
you run off analysing each and every one just to determine whether the
change is the intended effect and the test needs updating, or whether the
test is right and you broke something you shouldn't have.

It's not about the % coverage... it's about the quality of your test
cases... automated tooling will not produce quality test cases (unless they
are processing an independently derived specification... in which case who
tests the specification to be correct... Quis custodiet ipsos custodes?)

-Stephen


On 3 February 2014 07:41, <[email protected]> wrote:

>
>
> Hi there
>
> Yes, i totally agree with you: the TDD way is a much more useful way to
> create software. From my point of view this is the way to go for a
> greenfield project.
>
> However, while working with legacy code this is not always possible. The
> described tool is a simple helper class to create a good test coverage for
> existing codebases before changing something, so basically it is a TDD
> helper. Nothing more. Of course it is up to the designer to check what is
> tested and create a good test strategy. For our current project, this
> simple helper saved us a lot of time, so i want to share it with the
> community.
>
> Regards
> Mirko
>
> 
>