Re: Testing

On Sun, May 13, 2018 at 11:14 PM, <[email protected]> wrote:

>   - I do agree specs are what we really need long-term, and Property Tests
> (i.e. random fuzzing against a spec) to check reality against the spec!
>

For the purpose of community discussion, can someone please define or give
examples of "specs"? This is a nebulous term to me and since I'm not the
one using/asking for it, I can't really talk about the suggestion until
there's some clarification. My biggest question is - is it different than
documentation? How/why? I can't really think of anything that is a
specification that shouldn't also be documentation, unless you're also
considering implementation details.


> > El 13 may 2018, a las 20:57, [email protected] escribió:
> >
> > While it's on my mind... this came up in a github conversation, but
> Brian was busy so we were going to come back to it. (Brian might still be
> busy... no rush... I don't expect an immediate answer.)
> >
>

I just finished up coursework, so I am free now. Thanks for bringing it up
again.

>
>
>> I had expressed some doubts about the current (loose?) policy of
>> requiring a unit test for every change. Writing unit tests (especially for
>> server behavior) is time-consuming, meaning, to be worthwhile, it has to
>> pass a fairly high bar for usefulness.
>>
>
I think I may have said this already: it's going to take time, especially
at the beginning when we have no good model for how to write these tests.
That's just unavoidable. If a test takes too long to write, it could mean
one of a few things:

1. this is the first time anyone has had to think about writing this test,
so there's no example to draw on
2. the interface that the test relies on is bad, and should be refactored
so that it's more testable (testable code is generally better code, because
it's easier to predict and can typically be used by clients more easily)
3. the unit testing framework doesn't have the necessary features to handle
this case

But in general, I would rather have 1 well-tested submission than 10
under-tested or non-tested, because I know the well-tested change does what
it says, is not going to regress in the future (assuming the tests are
run), and can easily be used by a new developer to confirm that they're not
breaking anything.


> In this case, with the server (excepting inter-process communication), we
> should also get into the habit of writing C++-level tests. Nathan set up a
> stub for us there, and we should be making use of it. Running tests on
> scsynth through sclang is a backwards hack and also means that the
> components are not cleanly separated. One should be able to run tests on
> scsynth without having built sclang.
>
>
>> Tests are more useful when they are broad. While I agree that the current
>> approach gives us *some* unit tests, and some tests are better than none,
>> it also encourages extremely narrowly drawn tests and diverts attention
>> away from a broader-based testing strategy.
>>
>
I feel that the most efficient way we can test – efficient in the sense
that we get the most value for the least effort – is to add tests for code
that is changed often, and therefore has a higher chance of being broken by
future changes. Requiring tests for code touched in PR achieves this
perfectly. If you want us to write more directed tests, I would suggest
that we start from the parts of the codebase we know are prone to bugs,
and/or the parts we want to rewrite. For example, I wrote the test suite
for the lexer and parser and compiler because I wanted to make a number of
improvements there. I also wrote a ton of tests for inter-process
communication between server and sclang because I knew there was a desire
to fix that in future work, and I didn't want the lack of tests or the
burden of manual testing to get in the way of that.

Also, I think we've kind of collectively acknowledged that we'd prefer to
work incrementally on this. Everyone has different schedules and levels of
interest in maintenance, and pay-as-you-go seems to work well in the year
I've been on the project. So I think one answer to this criticism is to
encourage more tests per PR, and give each other real feedback on when and
where testing is needed or unneeded. This has been happening already and
it's been hugely valuable.

Another thing that might be getting left out of this discussion is that
unit testing is still an experiment for us. We have lots of code of varying
levels of testability, we don't know exactly what unit testing in
SuperCollider looks like because nobody's done it before, and it takes time
to establish what we can realistically accomplish as a team. So, I don't
think anybody has the answers, but there is a certain amount of patience
that's required so that we can see the full impact of the policies we're
following.

Thanks for bringing these points up; it's a valuable discussion.

-Brian