Re: Testing

> If a test takes too long to write, it could mean one of a few things:
>
> 1. this is the first time anyone has had to think about writing this test, 
> so there's no example to draw on
> 2. the interface that the test relies on is bad, and should be refactored 
> so that it's more testable (testable code is generally better code, because 
> it's easier to predict and can typically be used by clients more easily)
> 3. the unit testing framework doesn't have the necessary features to handle 
> this case

The specific hurdles in that EnvGen test were:

- Need to test both scsynth and supernova. (Shouldn't all server tests run 
on both? Why do other server test authors get to skip that but I picked a 
d@@@ supernova-only issue to test...?) I made a lot of dumb mistakes with 
the logic.

- Success, in that test, is the *absence* of an early node-free. That's a 
missing feature -- how to detect when something *didn't* happen.

> I feel that the most efficient way we can test – efficient in the sense 
> that we get the most value for the least effort – is to add tests for code 
> that is changed often, and therefore has a higher chance of being broken by 
> future changes. Requiring tests for code touched in PR achieves this perfectly

"Perfectly," that's overselling it a bit, isn't it? I'm pretty sure that 
EnvGen code won't be touched for another 5 years.

> I would suggest that we start from the parts of the codebase we know are 
>prone to bugs, and/or the parts we want to rewrite. 

Yes. I'd suggest SynthDef building as the #1 place to start. But nobody 
knows where to start designing these tests because... get this... *there's 
no spec*. Nobody really knows where the corner cases are, or has any way to 
guess, because James McCartney is the only person who ever had to think it 
through from the beginning. So step 1 is describe the logic carefully and 
look for areas that are under-determined.

I'd also add GUI. I keep finding little goofs... which are minor, but quite 
bad when they're user facing and the group of users is a classroom full of 
students.

I agree this discussion is valuable, and I agree that testing is valuable. 
I got hit with a particularly bad case and it raised some questions for me, 
that's all.

hjh

Sent with AquaMail for Android
http://www.aqua-mail.com


On May 16, 2018 06:49:11 [email protected] wrote:

> On Sun, May 13, 2018 at 11:14 PM, <[email protected]> wrote:
>
>>   - I do agree specs are what we really need long-term, and Property Tests
>> (i.e. random fuzzing against a spec) to check reality against the spec!
>>
>
> For the purpose of community discussion, can someone please define or give
> examples of "specs"? This is a nebulous term to me and since I'm not the
> one using/asking for it, I can't really talk about the suggestion until
> there's some clarification. My biggest question is - is it different than
> documentation? How/why? I can't really think of anything that is a
> specification that shouldn't also be documentation, unless you're also
> considering implementation details.
>
>
>> > El 13 may 2018, a las 20:57, [email protected] escribió:
>> >
>> > While it's on my mind... this came up in a github conversation, but
>> Brian was busy so we were going to come back to it. (Brian might still be
>> busy... no rush... I don't expect an immediate answer.)
>> >
>>
>
> I just finished up coursework, so I am free now. Thanks for bringing it up
> again.
>
>>
>>
>>> I had expressed some doubts about the current (loose?) policy of
>>> requiring a unit test for every change. Writing unit tests (especially for
>>> server behavior) is time-consuming, meaning, to be worthwhile, it has to
>>> pass a fairly high bar for usefulness.
>>>
>>
> I think I may have said this already: it's going to take time, especially
> at the beginning when we have no good model for how to write these tests.
> That's just unavoidable. If a test takes too long to write, it could mean
> one of a few things:
>
> 1. this is the first time anyone has had to think about writing this test,
> so there's no example to draw on
> 2. the interface that the test relies on is bad, and should be refactored
> so that it's more testable (testable code is generally better code, because
> it's easier to predict and can typically be used by clients more easily)
> 3. the unit testing framework doesn't have the necessary features to handle
> this case
>
> But in general, I would rather have 1 well-tested submission than 10
> under-tested or non-tested, because I know the well-tested change does what
> it says, is not going to regress in the future (assuming the tests are
> run), and can easily be used by a new developer to confirm that they're not
> breaking anything.
>
>
>> In this case, with the server (excepting inter-process communication), we
>> should also get into the habit of writing C++-level tests. Nathan set up a
>> stub for us there, and we should be making use of it. Running tests on
>> scsynth through sclang is a backwards hack and also means that the
>> components are not cleanly separated. One should be able to run tests on
>> scsynth without having built sclang.
>>
>>
>>> Tests are more useful when they are broad. While I agree that the current
>>> approach gives us *some* unit tests, and some tests are better than none,
>>> it also encourages extremely narrowly drawn tests and diverts attention
>>> away from a broader-based testing strategy.
>>>
>>
> I feel that the most efficient way we can test – efficient in the sense
> that we get the most value for the least effort – is to add tests for code
> that is changed often, and therefore has a higher chance of being broken by
> future changes. Requiring tests for code touched in PR achieves this
> perfectly. If you want us to write more directed tests, I would suggest
> that we start from the parts of the codebase we know are prone to bugs,
> and/or the parts we want to rewrite. For example, I wrote the test suite
> for the lexer and parser and compiler because I wanted to make a number of
> improvements there. I also wrote a ton of tests for inter-process
> communication between server and sclang because I knew there was a desire
> to fix that in future work, and I didn't want the lack of tests or the
> burden of manual testing to get in the way of that.
>
> Also, I think we've kind of collectively acknowledged that we'd prefer to
> work incrementally on this. Everyone has different schedules and levels of
> interest in maintenance, and pay-as-you-go seems to work well in the year
> I've been on the project. So I think one answer to this criticism is to
> encourage more tests per PR, and give each other real feedback on when and
> where testing is needed or unneeded. This has been happening already and
> it's been hugely valuable.
>
> Another thing that might be getting left out of this discussion is that
> unit testing is still an experiment for us. We have lots of code of varying
> levels of testability, we don't know exactly what unit testing in
> SuperCollider looks like because nobody's done it before, and it takes time
> to establish what we can realistically accomplish as a team. So, I don't
> think anybody has the answers, but there is a certain amount of patience
> that's required so that we can see the full impact of the policies we're
> following.
>
> Thanks for bringing these points up; it's a valuable discussion.
>
> -Brian