Re: Long running test suite

> In fact, it ought to be possible to surrender all control of ordering and
parallelism to the runner.

Part of me says, "Any code that depends on the exact performance of the
scheduler, the load the number of CPU's, .... is flaky, I want my tests to
break and tell me so."

"Any suit of tests that depend on order of execution to work has destroyed
the value of defect localization (ie. Which test breaks tells you which
part of the code is broken)"

So part of me says you are right.

However watching the behaviour of the herd of cats I work with....flaky
sporadic failing tests are a human process level disaster.

Tests that fail for any other reason other than the last checkin,  results
in dilution of responsibility and ever growing flakiness.

ie. Unit tests suit must be 100% deterministic and reproducible, and we
must use other techniques to hunt threading / ordering issues.

Processes are very nice in the sense that they completely clean up at the
OS level. I use valgrind to enforce that....

1) Every Unit Under Test initialises everything it should.
2) Doesn't leak any resources (all _must_ be collected by test process exit
point)

I test for threading nasties by structuring things so I can explicitly
trigger _every_ possible event sequence order and verify it works.

I have some subsystem tests, which are _not_ part of the unit test suite
which I leave running for as long as I want to hunt for cases I may have
missed.....

If I find one I encode it as an explicit ordering in a unit test.


On Tue, Jun 7, 2016 at 2:27 PM, Charlie Poole [email protected]
[testdrivendevelopment] <[email protected]> wrote:

>
>
> Hi John,
>
> I like your list of ideas, but I take a small exception with your comment
> about running tests on multiple threads.
>
> I agree that it sometimes introduces non-deterministic results, but that's
> generally due to faults in the tests. Ideally, it should be possible to run
> any set of unit tests (i.e. microtests) in random order on parallel
> threads. In fact, it ought to be possible to surrender all control of
> ordering and parallelism to the runner. I grant that this is a hard goal to
> achieve.
>
> Charlie
>
> On Mon, Jun 6, 2016 at 6:38 PM, John Carter [email protected]
> [testdrivendevelopment] <[email protected]> wrote:
>
>>
>>
>> Things I do to handle this....
>>
>> * Benchmark. Every build spits out an order list of how long each test
>> took. Attack the long ones first.
>> * Parallelism via Processes not threads. Multithreaded unit testing is a
>> recipe for non-determinism and flakiness.
>> * No sleep rule. Find every test with a "sleep" in it and fix it. Sleeps,
>> delays, timeouts are test smells of the first order.
>> * Unreal time. Wrap all time services and create a test interface that
>> allows you to "roll time forward" and trigger all events in order.
>> * Some tests want to be able to run for days to get better state space
>> coverage. Make that a "developer only" option. Run a tiny / fast subset for
>> unit test checkin coverage and let developer run exhaustive test himself
>> when needed.
>>
>>
>> On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno [email protected]
>> [testdrivendevelopment] <[email protected]> wrote:
>>
>>>
>>>
>>> Hello,
>>>
>>> To those who have experienced test suites that run long, say ~25
>>> minutes, what are some techniques you have used to mitigate it?  We still
>>> would like to be able to check in often and run the test suites prior to
>>> checking our code in to our repository but these long running tests makes
>>> us stretch what we mean with "often".
>>>
>>> Joen
>>>
>>> __._,
>>>
>>>
>>
>>
>> --
>> John Carter                             Phone : (64)(3) 358 6639
>> Tait Electronics                        Fax   : (64)(3) 359 4632
>> PO Box 1645 Christchurch                Email : [email protected]
>> New Zealand
>>
>>
>> ------------------------------
>> This Communication is Confidential. We only send and receive email on the
>> basis of the terms set out at www.taitradio.com/email_disclaimer
>> ------------------------------
>>
>>
> 
>



-- 
John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : [email protected]
New Zealand

-- 

------------------------------
This Communication is Confidential. We only send and receive email on the
basis of the terms set out at www.taitradio.com/email_disclaimer
------------------------------