Re: Testing

On Thu, May 17, 2018 at 7:48 PM, <[email protected]> wrote:

> > If a test takes too long to write, it could mean one of a few things:
> >
> > 1. this is the first time anyone has had to think about writing this
> test, so there's no example to draw on
> > 2. the interface that the test relies on is bad, and should be
> refactored so that it's more testable (testable code is generally better
> code, because it's easier to predict and can typically be used by clients
> more easily)
> > 3. the unit testing framework doesn't have the necessary features to
> handle this case
>
> The specific hurdles in that EnvGen test were:
>
> - Need to test both scsynth and supernova. (Shouldn't all server tests run
> on both? Why do other server test authors get to skip that but I picked a d@@@
> supernova-only issue to test...?) I made a lot of dumb mistakes with the
> logic.
>
This is not something we've discussed before; AFAIK no test is specifically
designed to run on either server. This is exactly case #1 above. We may
need a spec here. :P

- Success, in that test, is the *absence* of an early node-free. That's a
> missing feature -- how to detect when something *didn't* happen.
>
Great, so this is something that should go into a new ticket or PR, if you
can describe it precisely. If this is something a test needs to know then
it's probably something a user will need to know as well.


> > I feel that the most efficient way we can test – efficient in the sense
> that we get the most value for the least effort – is to add tests for code
> that is changed often, and therefore has a higher chance of being broken by
> future changes. Requiring tests for code touched in PR achieves this
> perfectly
>
> "Perfectly," that's overselling it a bit, isn't it? I'm pretty sure that
> EnvGen code won't be touched for another 5 years.
>
Only time will tell. Sure, there are programs that can show you the most
commonly touched parts of your codebase. If you really want to be rigorous
about it you can find and use one of those.


> > I would suggest that we start from the parts of the codebase we know are
> prone to bugs, and/or the parts we want to rewrite.
>
> Yes. I'd suggest SynthDef building as the #1 place to start. But nobody
> knows where to start designing these tests because... get this... *there's
> no spec*. Nobody really knows where the corner cases are, or has any way to
> guess, because James McCartney is the only person who ever had to think it
> through from the beginning. So step 1 is describe the logic carefully and
> look for areas that are under-determined.
>
Agree. This would serve as a great case study.


> I'd also add GUI. I keep finding little goofs... which are minor, but
> quite bad when they're user facing and the group of users is a classroom
> full of students.
>
Such as?

This would probably reveal a lot of missing features. Since so much of it
comes from Qt, we should also be careful we're not testing other projects'
code.