Re: RE: Tool to create assertions from a running Java application

On 5 February 2014 03:59, Teemu Kanstrén <[email protected]> wrote:

>
>
>
> On 4 February 2014 04:30, Stephen Connolly <
> [email protected]> wrote:
>
>>  I think you just made a great usecase for implementation based test
>>> generators. Re-generate after each change and show off your 100% coverage.
>>> Even if the tests are pointless, manager is happy :)
>>>
>>
>> Until somebody points out that your test cases are not checked into
>> source control... I've been down that path before... never ends well...
>>
>>
> Is there some specific benefit to checking them in source control if you
> can just regenerate the same from another artefact?
>

If you are generating the "coverage fakers" from your code under test, at
some point somebody with an ISO 900x bent will force the generated "tests"
into source control and then a review of the changes to the tests over
time... in order to ensure that they are "meaningful" tests...

Walk that road if you want to... it's not a safe neighbourhood though...


>
>
>> There was even the case where we were driving an effort to bump code
>> coverage of a legacy code base from 66% to 75%...
>>
>> Those 5 failing test cases were tripping every single exception path that
>> the regular test cases were missing... When "Bob" fixed the database, the
>> code coverage dropped from 73% back down to 66.4%
>>
>> Explaining that to management, i.e. a 6 week concentrated concerted
>> effort of the entire team to drive test coverage from 66% to 75% only
>> actually gained us 0.4% extra coverage...
>>
>>
> Ouch :). So how did that end? Did you make those 5 failure scenarios part
> of the test suite?
>

Oh it was worse than that, there was no easy way to get the system to fail
through all those exceptions as it was not designed for testability... in
the end management just switched to 80% coverage for all new code... which
can be rather hard to measure... so hence easy to fluff!


>
> If you generate code tests from a specification, that is great. We now
>> know that the code implements the specification correctly... we have guards
>> that check the code... but who will check that the guards are enforcing the
>> correct things?
>>
>> I would argue that it is somewhat easier to verify code than verify a
>> specification... at least for a lot of the specifications you will see out
>> there.
>>
>> Some specifications are use-case based, with wooly descriptions of
>> behaviour. These tend to generate example based test-cases... which give
>> good confidence that the use-case(s) are met... but no confidence when you
>> walk off the tested path.
>>
>>
> Is this not the case whatever method you apply to test creation? Your idea
> of what you test for needs to come from somewhere. Regardless of the method
> of creating those test cases. Verifying what you actually need to test is
> always an issue. I find the questions in test creation or test modelling to
> produce one useful way to create some discussion on that.. Most of the time
> this is not unit testing though.
>
> I guess you mean generating tests from some use cases or natural language
> spec. I was thinking more about some more formalized version for a test
> generator. Which requires some effort of course, much as writing many
> tests..
>

All you are doing is then really writing the tests in a different
language... and the test generator meme will just pop pack up in that scope
(i.e. why don't we just generate the spec from what the code currently does)


>
> Teemu
>
>
>
> 
>