Re: Long running test suite

Hi John,

I like your list of ideas, but I take a small exception with your comment
about running tests on multiple threads.

I agree that it sometimes introduces non-deterministic results, but that's
generally due to faults in the tests. Ideally, it should be possible to run
any set of unit tests (i.e. microtests) in random order on parallel
threads. In fact, it ought to be possible to surrender all control of
ordering and parallelism to the runner. I grant that this is a hard goal to
achieve.

Charlie

On Mon, Jun 6, 2016 at 6:38 PM, John Carter [email protected]
[testdrivendevelopment] <[email protected]> wrote:

>
>
> Things I do to handle this....
>
> * Benchmark. Every build spits out an order list of how long each test
> took. Attack the long ones first.
> * Parallelism via Processes not threads. Multithreaded unit testing is a
> recipe for non-determinism and flakiness.
> * No sleep rule. Find every test with a "sleep" in it and fix it. Sleeps,
> delays, timeouts are test smells of the first order.
> * Unreal time. Wrap all time services and create a test interface that
> allows you to "roll time forward" and trigger all events in order.
> * Some tests want to be able to run for days to get better state space
> coverage. Make that a "developer only" option. Run a tiny / fast subset for
> unit test checkin coverage and let developer run exhaustive test himself
> when needed.
>
>
> On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno [email protected]
> [testdrivendevelopment] <[email protected]> wrote:
>
>>
>>
>> Hello,
>>
>> To those who have experienced test suites that run long, say ~25 minutes,
>> what are some techniques you have used to mitigate it?  We still would like
>> to be able to check in often and run the test suites prior to checking our
>> code in to our repository but these long running tests makes us stretch
>> what we mean with "often".
>>
>> Joen
>>
>> __._,
>>
>>
>
>
> --
> John Carter                             Phone : (64)(3) 358 6639
> Tait Electronics                        Fax   : (64)(3) 359 4632
> PO Box 1645 Christchurch                Email : [email protected]
> New Zealand
>
>
> ------------------------------
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.taitradio.com/email_disclaimer
> ------------------------------
>
> 
>