Testing

While it's on my mind... this came up in a github conversation, but Brian was busy so we were going to come back to it. (Brian might still be busy... no rush... I don't expect an immediate answer.)

I had expressed some doubts about the current (loose?) policy of requiring a unit test for every change. Writing unit tests (especially for server behavior) is time-consuming, meaning, to be worthwhile, it has to pass a fairly high bar for usefulness.

Tests are more useful when they are broad. While I agree that the current approach gives us *some* unit tests, and some tests are better than none, it also encourages extremely narrowly drawn tests and diverts attention away from a broader-based testing strategy.

Say you notice an odd behavior, fix it, and add a unit test designed to catch the behavior that you just fixed. Now the test is useful for essentially one and only one thing: to detect a regression back to the behavior that you just fixed. In most cases, this sort of regression is unlikely. So the benefit of the test is largely illusory -- you *think* you're testing, but really you're spot-checking.

It's as if one plane on one flight had a problem with the window seal in row 32, so safety inspectors from that point forward check the window seals in row 32 and don't bother with the other rows. Well, I feel safer already...

My suggestion is:

1. Specs. I know I keep saying it, and I'm going to keep saying it until we actually do it. It's very hard to design a comprehensive test when you don't have a comprehensive description of the correct behavior. We have poorly-thought-through tests because we have poorly-thought-through specs (or *no* specs).

2. Based on specs, identify key areas that would benefit from broad-scope testing.

3. Implement tests designed to catch places where the behavior deviates from the spec -- including places we don't know about yet.

I was thinking about this today because of a newly discovered bug in Pkey:embedInStream -- checking the PR, I see that there is a unit test, but it doesn't test the changed behavior! And I worry that if it did test it, it might test asStream only and not embedInStream -- a spec for the `repeats` argument in patterns would remind us to test both cases.

Again, I don't think it's necessarily a bad thing to push developers to think about unit testing. But I think we could do it better.

hjh


_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/