Re: Long running test suite

The language can have a big impact though. I’m working in Go right now.
Message passing is included in the language as channels, and there are just
a few idioms you have to know to do it pretty well. Still a lot harder than
a single thread, but it’s the least confusing way to do it I have yet
encountered.

On Mon, Jun 13, 2016 at 2:50 PM John Carter [email protected]
[testdrivendevelopment] <[email protected]> wrote:

> On Fri, Jun 10, 2016 at 6:00 PM, Kim Gräsman [email protected]
> [testdrivendevelopment] <[email protected]> wrote:
>
>>
>>
>>
>> > I test for threading nasties by structuring things so I can explicitly
>> trigger _every_
>> > possible event sequence order and verify it works.
>>
>> I've seen several test suites where people force execution order on
>> multithreaded SUTs (using test-only synchronization points), effectively
>> creating a programmatic single-step debugger. It's not pretty.
>>
>
> I have a colleague who tried to go that route, I agree, not pretty.
>
> But it sounds like you're talking more about decoupling testable logic
>> from the threads that drive it, and only test the logic? I like that better.
>>
>
> My approach does assume that you have identified all shared state and
> appropriately serialized access to it. So it is now merely a matter of
> verify that your event handlers can handle any sequence of events the
> outside world may throw at them.
>
> This is made a _lot_ easier by only communicating between threads via
> message passing. (A matter that requires discipline).
>
> Did I ever mention I hate multithreading? I know I am not smart enough to
> do it reliably, and all evidence I have seen leads me to have severe doubts
> about the rest of humanity.
>
> Yes, it is trivial to get a trivial threading program to work, it's pretty
> easy to get a non-trivial one working 90% of the time.
>
> But as you add 9's there seems to always be one more bug no matter how
> smart the programmers.
>
> Carter's Corollary to the Dunning-Kruger effect : Those people who are
> indeed smart and are accurately aware of how smart they are, still aren't
> smart enough to do multi-threading. They produce pretty much the same
> number of bugs, the bugs are just further down the long tail of probability.
>
> > I have some subsystem tests, which are _not_ part of the unit test suite
>> which I
>> > leave running for as long as I want to hunt for cases I may have
>> missed.....
>>
>> The Google LLVM crowd are doing lots of interesting stuff for C++ here.
>>
>> Assuming you have a high-level test suite driving the whole program, they
>> provide compiler-supported dynamic analysis to expose threading bugs at
>> runtime, e.g.:
>>
>> http://clang.llvm.org/docs/ThreadSanitizer.html
>> http://clang.llvm.org/docs/AddressSanitizer.html
>>
>> http://jefftrull.github.io/c++/clang/llvm/fuzzing/sanitizers/2015/11/27/fuzzing-with-sanitizers.html
>>
>> So the combination of low-level 100% stable unit tests, and driving the
>> (instrumented) application at a high-leve, maybe with fuzzed input to
>> expose flaky behavior is really compelling.
>>
>
> I've devised a "build for checkin" and a "background build" split. It
> would be nice if we could run every test fast enough to give instant feed
> back... but as we add products.... we can't.
>
> So the "build for checkin" guarantees that all developers on currently
> active projects will be able to checkout and continue working.
>
> The background build wakes up a farm of servers and runs the tests under
> valgrind and address sanitizer and for every product.
>
> I haven't got as far as adding fuzzers yet.
>
>
>
>>
>> It's fun to see that C++ has such elaborate infrastructure, I don't know
>> if the same kind of machinery is readily available for other
>> languages/environments. Then again, maybe those languages/environments
>> don't encourage those kinds of bugs :-)
>>
>> Most of what valgrind / address sanitizer does isn't needed in a saner
> language like ruby (or D).
>
>
> And given a chance I would hop our developement to D to keep largish
> classes of bugs from being writen in the first place.
>
>
>
> --
> John Carter                             Phone : (64)(3) 358 6639
> Tait Electronics                        Fax   : (64)(3) 359 4632
> PO Box 1645 Christchurch                Email : [email protected]
> New Zealand
>
>
> ------------------------------
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.taitradio.com/email_disclaimer
> ------------------------------
>
> 
>