Re: RE: Tool to create assertions from a running Java application

On 4 February 2014 00:58, Teemu Kanstrén <[email protected]> wrote:

>
>
>
> On 3 February 2014 07:57, Stephen Connolly <
> [email protected]> wrote:
>
>>
>>
>> Quality tests are about meaningful test names and small test case size
>> with at most 1-2 asserts per test case.
>>
>>
> Nice ideal. In practice having a good coverage with at least somehow
> meaningful tests would be often a good start.. Just saying.
>

And I often am a bold boy and write one test case called `smokes` with
50-60 assertThat statements in a row... but, and this is the important bit,
I know I am storing up technical debt by not factoring those tests out into
individual test cases and it is a deliberate decision that I make to just
write the one smoke test-case "for now".

We need to keep the ideal in our head so that we can remember to strive for
it.


>
> The end effect of that tooling is that the entire code base is frozen in
>> place, as any change results in a complete cascade of test failures... and
>> you run off analysing each and every one just to determine whether the
>> change is the intended effect and the test needs updating, or whether the
>> test is right and you broke something you shouldn't have.
>>
>
> I think you just made a great usecase for implementation based test
> generators. Re-generate after each change and show off your 100% coverage.
> Even if the tests are pointless, manager is happy :)
>

Until somebody points out that your test cases are not checked into source
control... I've been down that path before... never ends well...

There was even the case where we were driving an effort to bump code
coverage of a legacy code base from 66% to 75%...

"Bob" was the database guy, he made a change to some infrastructure stuff
that broke 5 of the test cases (we had about 10,000 test cases) on the
Friday when going on vacation for two weeks. So our build was unstable as
none of us knew exactly how to fix those test cases or how to fix the
infrastructure they were testing against... but he'd be back in 2 weeks...
we were writing away our tests.

The Friday before "Bob" came back we had our sprint review with management,
they looked at the Code coverage graph in Jenkins... it was sitting nice
and near our end-game target at 73%... pats on back all round.

"Bob" comes back, fixes the infrastructure (some SQL Server permission
issue), and we're back to a "stable" build. (I should point out that
everyone was using local code coverage tooling to write *targeted* tests...
we had a list of the least covered classes and you would just focus on
writing *unit* tests for that class, so nobody was looking at the test
coverage graph...

Those 5 failing test cases were tripping every single exception path that
the regular test cases were missing... When "Bob" fixed the database, the
code coverage dropped from 73% back down to 66.4%

Explaining that to management, i.e. a 6 week concentrated concerted effort
of the entire team to drive test coverage from 66% to 75% only actually
gained us 0.4% extra coverage...

Well if you have ever wondered why the default of the Jenkins Cobertura
plugin is to ignore unstable builds... that experience is the reason.

We have to live in the real world, where you sometimes get managers (or
worse still - the boss of your manager) who do not understand the technical
things...


>
>
>> automated tooling will not produce quality test cases (unless they are
>> processing an independently derived specification... in which case who
>> tests the specification to be correct... Quis custodiet ipsos custodes?)
>>
>>
> This is how techniques like model-based testing work. In my view your
> "independently derived specification" is in such case tested by the
> implementation. Because your implementation is from the viewpoint of your
> tests another "independently derived specification". When you generate
> tests and run those, they will tell you if there is a mismatch. Which of
> your interpretation of system intent is at fault is then to be
> investigated..
>

Quis custodiet ipsos custodes?

Who will guard the guards?

If you generate code tests from a specification, that is great. We now know
that the code implements the specification correctly... we have guards that
check the code... but who will check that the guards are enforcing the
correct things?

I would argue that it is somewhat easier to verify code than verify a
specification... at least for a lot of the specifications you will see out
there.

Some specifications are use-case based, with wooly descriptions of
behaviour. These tend to generate example based test-cases... which give
good confidence that the use-case(s) are met... but no confidence when you
walk off the tested path.

Some specifications mix requirements with design, e.g. the UML based type.

I think you can generate test cases for a parser based on the specification
of the grammar that it is required to parse.

I think you can generate test cases for a performance specification...

But most of the rest of the time the specification is just too high a level
to be useful for generating *unit* tests... unless you are dealing with
very junior developers who have been given a rigid spec for the 10 line
method that they need to write... and I would argue that the architect
should just have written the 10 lines themselves rather than the 3 page
spec ;-)

-Stephen


>
> -Teemu
>
>
> 
>