Re: Long running test suite

Hi John, all,

> I test for threading nasties by structuring things so I can explicitly
trigger _every_
> possible event sequence order and verify it works.

I've seen several test suites where people force execution order on
multithreaded SUTs (using test-only synchronization points), effectively
creating a programmatic single-step debugger. It's not pretty.

But it sounds like you're talking more about decoupling testable logic from
the threads that drive it, and only test the logic? I like that better.

> I have some subsystem tests, which are _not_ part of the unit test suite
which I
> leave running for as long as I want to hunt for cases I may have
missed.....

The Google LLVM crowd are doing lots of interesting stuff for C++ here.

Assuming you have a high-level test suite driving the whole program, they
provide compiler-supported dynamic analysis to expose threading bugs at
runtime, e.g.:

http://clang.llvm.org/docs/ThreadSanitizer.html
http://clang.llvm.org/docs/AddressSanitizer.html
http://jefftrull.github.io/c++/clang/llvm/fuzzing/sanitizers/2015/11/27/fuzzing-with-sanitizers.html

So the combination of low-level 100% stable unit tests, and driving the
(instrumented) application at a high-leve, maybe with fuzzed input to
expose flaky behavior is really compelling.

It's fun to see that C++ has such elaborate infrastructure, I don't know if
the same kind of machinery is readily available for other
languages/environments. Then again, maybe those languages/environments
don't encourage those kinds of bugs :-)

- Kim

On Fri, Jun 10, 2016 at 3:20 AM, John Carter [email protected]
[testdrivendevelopment] <[email protected]> wrote:

>
>
> > In fact, it ought to be possible to surrender all control of ordering
> and parallelism to the runner.
>
> Part of me says, "Any code that depends on the exact performance of the
> scheduler, the load the number of CPU's, .... is flaky, I want my tests to
> break and tell me so."
>
> "Any suit of tests that depend on order of execution to work has destroyed
> the value of defect localization (ie. Which test breaks tells you which
> part of the code is broken)"
>
> So part of me says you are right.
>
> However watching the behaviour of the herd of cats I work with....flaky
> sporadic failing tests are a human process level disaster.
>
> Tests that fail for any other reason other than the last checkin,  results
> in dilution of responsibility and ever growing flakiness.
>
> ie. Unit tests suit must be 100% deterministic and reproducible, and we
> must use other techniques to hunt threading / ordering issues.
>
> Processes are very nice in the sense that they completely clean up at the
> OS level. I use valgrind to enforce that....
>
> 1) Every Unit Under Test initialises everything it should.
> 2) Doesn't leak any resources (all _must_ be collected by test process
> exit point)
>
> I test for threading nasties by structuring things so I can explicitly
> trigger _every_ possible event sequence order and verify it works.
>
> I have some subsystem tests, which are _not_ part of the unit test suite
> which I leave running for as long as I want to hunt for cases I may have
> missed.....
>
> If I find one I encode it as an explicit ordering in a unit test.
>
>
>
> On Tue, Jun 7, 2016 at 2:27 PM, Charlie Poole [email protected]
> [testdrivendevelopment] <[email protected]> wrote:
>
>>
>>
>> Hi John,
>>
>> I like your list of ideas, but I take a small exception with your comment
>> about running tests on multiple threads.
>>
>> I agree that it sometimes introduces non-deterministic results, but
>> that's generally due to faults in the tests. Ideally, it should be possible
>> to run any set of unit tests (i.e. microtests) in random order on parallel
>> threads. In fact, it ought to be possible to surrender all control of
>> ordering and parallelism to the runner. I grant that this is a hard goal to
>> achieve.
>>
>> Charlie
>>
>> On Mon, Jun 6, 2016 at 6:38 PM, John Carter [email protected]
>> [testdrivendevelopment] <[email protected]> wrote:
>>
>>>
>>>
>>> Things I do to handle this....
>>>
>>> * Benchmark. Every build spits out an order list of how long each test
>>> took. Attack the long ones first.
>>> * Parallelism via Processes not threads. Multithreaded unit testing is a
>>> recipe for non-determinism and flakiness.
>>> * No sleep rule. Find every test with a "sleep" in it and fix it.
>>> Sleeps, delays, timeouts are test smells of the first order.
>>> * Unreal time. Wrap all time services and create a test interface that
>>> allows you to "roll time forward" and trigger all events in order.
>>> * Some tests want to be able to run for days to get better state space
>>> coverage. Make that a "developer only" option. Run a tiny / fast subset for
>>> unit test checkin coverage and let developer run exhaustive test himself
>>> when needed.
>>>
>>>
>>> On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno [email protected]
>>> [testdrivendevelopment] <[email protected]> wrote:
>>>
>>>>
>>>>
>>>> Hello,
>>>>
>>>> To those who have experienced test suites that run long, say ~25
>>>> minutes, what are some techniques you have used to mitigate it?  We still
>>>> would like to be able to check in often and run the test suites prior to
>>>> checking our code in to our repository but these long running tests makes
>>>> us stretch what we mean with "often".
>>>>
>>>> Joen
>>>>
>>>> __._,
>>>>
>>>>
>>>
>>>
>>> --
>>> John Carter                             Phone : (64)(3) 358 6639
>>> Tait Electronics                        Fax   : (64)(3) 359 4632
>>> PO Box 1645 Christchurch                Email :
>>> [email protected]
>>> New Zealand
>>>
>>>
>>> ------------------------------
>>> This Communication is Confidential. We only send and receive email on the
>>> basis of the terms set out at www.taitradio.com/email_disclaimer
>>> ------------------------------
>>>
>>>
>>
>
>
> --
> John Carter                             Phone : (64)(3) 358 6639
> Tait Electronics                        Fax   : (64)(3) 359 4632
> PO Box 1645 Christchurch                Email : [email protected]
> New Zealand
>
>
> ------------------------------
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.taitradio.com/email_disclaimer
> ------------------------------
>
>
> 
>