Re: RE: Tool to create assertions from a running Java application

On 4 February 2014 04:30, Stephen Connolly
<[email protected]>wrote:

>  I think you just made a great usecase for implementation based test
>> generators. Re-generate after each change and show off your 100% coverage.
>> Even if the tests are pointless, manager is happy :)
>>
>
> Until somebody points out that your test cases are not checked into source
> control... I've been down that path before... never ends well...
>
>
Is there some specific benefit to checking them in source control if you
can just regenerate the same from another artefact?


> There was even the case where we were driving an effort to bump code
> coverage of a legacy code base from 66% to 75%...
>
> Those 5 failing test cases were tripping every single exception path that
> the regular test cases were missing... When "Bob" fixed the database, the
> code coverage dropped from 73% back down to 66.4%
>
> Explaining that to management, i.e. a 6 week concentrated concerted effort
> of the entire team to drive test coverage from 66% to 75% only actually
> gained us 0.4% extra coverage...
>
>
Ouch :). So how did that end? Did you make those 5 failure scenarios part
of the test suite?

If you generate code tests from a specification, that is great. We now know
> that the code implements the specification correctly... we have guards that
> check the code... but who will check that the guards are enforcing the
> correct things?
>
> I would argue that it is somewhat easier to verify code than verify a
> specification... at least for a lot of the specifications you will see out
> there.
>
> Some specifications are use-case based, with wooly descriptions of
> behaviour. These tend to generate example based test-cases... which give
> good confidence that the use-case(s) are met... but no confidence when you
> walk off the tested path.
>
>
Is this not the case whatever method you apply to test creation? Your idea
of what you test for needs to come from somewhere. Regardless of the method
of creating those test cases. Verifying what you actually need to test is
always an issue. I find the questions in test creation or test modelling to
produce one useful way to create some discussion on that.. Most of the time
this is not unit testing though.

I guess you mean generating tests from some use cases or natural language
spec. I was thinking more about some more formalized version for a test
generator. Which requires some effort of course, much as writing many
tests..

Teemu