Re: Tool to create assertions from a running Java application

I figured I'd chime in, since I also was working for Agitar, which, as far
as I know, nearly completely turned over their engineering staff about 5
years ago.

Agitar had some very clever analysis software, based, as some here have
guessed, on Daikon, as well as other research packages.  One of the
clevernesses was how it would prioritize test-input generation to attempt
to produce not-yet-seen behaviors.  So, if in an initial test run, a method
always returned true, the engine would analyze the code to see if there was
a way to make it return false.  In the beginning, these observations were
encoded in a proprietary observations database, but later, they figured out
how to use those observations to generate Java code in the form of JUnit
tests.

In my opinion, those tests were never really a great idea to use, without
reflection, as _tests_ that should be treated identically to
human-generated tests.  But they still had use:

1) They were a map of the behavior of your program:

One of the more interesting bugs found was that Agitar found a method that
was only ever called with positive floats; the intended input had been for
positive and negative floats, and it turned out there was a bug in the
calling method.

2) They could indicate the scope of change:

Even if you never looked at the content of the generated tests, you could
generate tests against all the classes on V1 of a package, and then run
those tests on V2, and look at what failed (assuming, usually correctly,
that even intended behavior changes would cause test failures).  If the
failures were all in tests against classes whose behavior you expected to
change, great.  If there were changes in other classes, that might be a
problem.

3) They could be used with Theories to automatically find actual bugs.

If you could specify a desired feature of your code as a Theory (a function
asserting an invariant property over a practically infinite set of possible
inputs), then you could use Agitar's corner-case-seeking engine to try to
find an input that violated that invariant:

http://dspace.mit.edu/bitstream/handle/1721.1/40090/MIT-CSAIL-TR-2008-002.pdf

I think there's still room for a lot of progress to be made on these ideas,
but my experience with Agitar makes me think that it's more likely to come
from open source efforts, or internal tool building at a large software
company, than as a profit-making sellable tool.

   David Saff


On Sat, Feb 1, 2014 at 5:00 PM, Malte Finsterwalder <
[email protected]> wrote:

>
>
> Maybe you can check out agitar. They provide a system that does exactly
> this: Generate tests for existing code.
> They made a big buzz a few years ago. I look into it very briefly and the
> generated tests where not that bad.
> The biggest problem was: They were far too numerous and too detailed.
> Writing good tests puts emphasis on the important parts that really needs
> to be tested and ignores exidential details.
> I never quite understood how a tool like that should be used.
>
> I know that Kent Beck was involved into the undertaking, but I think he is
> no longer. Probably because it didn't quite work out in practice.
>
> Greetings,
>    Malte
>
>  
>