Re: Testing

I don't have a strong opinion about how mandatory the tests should be: I'm sure as a dev community we can find a happy medium together. A couple thoughts on the other parts, though:

  - I've found the "window-seals-in-seat-32" type of testing you're describing to be useful in large codebases which don't have the test coverage the authors wish they did. It's a cheap way to start ingraining "testing culture" and they really do sometimes catch bugs.
    Maybe the analogy is that if window 32 broke before, maybe there's a design issue with the plane that causes that particular window to fail more frequently. I've written this kind of regression test at companies whose codebases didn't have enough test coverage and it really was often the case that there are tricky areas that bugs frequently surfaced in, and that tests caught. It's limited, though, which leads to:

  - I do agree specs are what we really need long-term, and Property Tests (i.e. random fuzzing against a spec) to check reality against the spec!

tl;dr: Specs: Yes! Property testing: Please! Unit testing as a stopgap: seems valuable to me. Mandatory unit testing: no comment!

Tom


> El 13 may 2018, a las 20:57, [email protected] escribió:
> 
> While it's on my mind... this came up in a github conversation, but Brian was busy so we were going to come back to it. (Brian might still be busy... no rush... I don't expect an immediate answer.)
> 
> I had expressed some doubts about the current (loose?) policy of requiring a unit test for every change. Writing unit tests (especially for server behavior) is time-consuming, meaning, to be worthwhile, it has to pass a fairly high bar for usefulness.
> 
> Tests are more useful when they are broad. While I agree that the current approach gives us *some* unit tests, and some tests are better than none, it also encourages extremely narrowly drawn tests and diverts attention away from a broader-based testing strategy.
> 
> Say you notice an odd behavior, fix it, and add a unit test designed to catch the behavior that you just fixed. Now the test is useful for essentially one and only one thing: to detect a regression back to the behavior that you just fixed. In most cases, this sort of regression is unlikely. So the benefit of the test is largely illusory -- you *think* you're testing, but really you're spot-checking.
> 
> It's as if one plane on one flight had a problem with the window seal in row 32, so safety inspectors from that point forward check the window seals in row 32 and don't bother with the other rows. Well, I feel safer already...
> 
> My suggestion is:
> 
> 1. Specs. I know I keep saying it, and I'm going to keep saying it until we actually do it. It's very hard to design a comprehensive test when you don't have a comprehensive description of the correct behavior. We have poorly-thought-through tests because we have poorly-thought-through specs (or *no* specs).
> 
> 2. Based on specs, identify key areas that would benefit from broad-scope testing.
> 
> 3. Implement tests designed to catch places where the behavior deviates from the spec -- including places we don't know about yet.
> 
> I was thinking about this today because of a newly discovered bug in Pkey:embedInStream -- checking the PR, I see that there is a unit test, but it doesn't test the changed behavior! And I worry that if it did test it, it might test asStream only and not embedInStream -- a spec for the `repeats` argument in patterns would remind us to test both cases.
> 
> Again, I don't think it's necessarily a bad thing to push developers to think about unit testing. But I think we could do it better.
> 
> hjh
> 
> 
> _______________________________________________
> sc-dev mailing list
> 
> info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
> archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
> search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/

_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/