Re: Testing

On Thu, May 17, 2018 at 9:18 PM, <[email protected]> wrote:

> ---- On Fri, 18 May 2018 08:31:03 +0800 <[email protected]> wrote ----
> > On Thu, May 17, 2018 at 7:48 PM,  <[email protected]> wrote:
> > >  The specific hurdles in that EnvGen test were:
> > >  - Need to test both scsynth and supernova. (Shouldn't all server
> tests run on both? Why do other server test authors get to skip that but I
> picked a d@@@ supernova-only issue to test...?) I made a lot of dumb
> mistakes with the logic.
> > This is not something we've discussed before; AFAIK no test is
> specifically designed to run on either server. This is exactly case #1
> above. We may need a spec here. :P
>
> This was a case where a UGen exhibited different behavior in scsynth and
> supernova, despite being built from the same source code. If we haven't
> discussed testing both servers, it suggests that we assume behavior will be
> the same in both places -- but we have evidence that this isn't always the
> case. We might have reason, then, to make it mandatory to test both.
>

I agree that both should be tested.


> For efficiency, probably we should move the scsynth/supernova logic up to
> the top level: the runTest (or whichever) method would collect a list of
> the tests to be run (because you might be running just a subset), and pass
> it to a method that would:
>
> 1. Boot scsynth.
> 2. Run the chosen tests.
> 3. Quit scsynth.
> 4. Boot supernova.
> 5. Run them again.
> 6. Quit supernova.
>

I'm not sure if you're glossing over this detail, but the server should be
booted/quit for each test, so that test results are completely independent
and reproducible. Again, anything that is not an integration test between
sclang and a server process should be done via C++ tests to avoid unwanted
dependencies, slowdown, and additional sources of failure. Whatever happens
in the meanwhile is a stopgap measure; i.e., technical debt.

At the very minimum it sounds like what needs to happen is that UnitTest's
runTest method should take an argument specifying which server program to
use. Then, it switches to that program for the duration of the test (and
switches back to whatever was being used before the call to runTest).
Everything else could be implemented in terms of that.

I have an idea about annotations that I will put in another thread.


> In my test, I consolidated the results: the test passes if Pass(scsynth)
> && Pass(supernova). But that's not really necessary. You could just run
> them separately and report the results -- any failures would still be noted
> as failures.
>
> >  - Success, in that test, is the *absence* of an early node-free. That's
> a missing feature -- how to detect when something *didn't* happen.
> > Great, so this is something that should go into a new ticket or PR, if
> you can describe it precisely. If this is something a test needs to know
> then it's probably something a user will need to know as well.
>
> OK.
>
> There's probably no way around this, but another time sink for writing
> unit tests is that you can't be sure the test distinguishes between success
> and failure without testing the test against two SC versions. I couldn't be
> certain that the logic was correct unless I ran it against an unfixed
> supernova (and got failure) *and*, with identical testing code, ran it
> against a fixed supernova and got success. That got to be really time
> consuming and irritating in that case. Maybe I'm just uniquely dense about
> it, or maybe just had a bad day, or maybe people are assuming tests are
> easy and quick to write because they aren't testing the tests properly. I
> don't know... I felt I shouldn't submit a unit test unless I was sure that
> it would tell the two cases apart, and it took a flipping lot of time to be
> sure of that.
>

Write the tests first, and then implement the fix. This is called red-green
testing, because you first get a "red" error result and have to work to
turn it "green" passing, and it's a very efficient way to work.


>
> >  I'd also add GUI. I keep finding little goofs... which are minor, but
> quite bad when they're user facing and the group of users is a classroom
> full of students.
> > Such as?
>
> - The scrollbar nightmare -- which we're still not fixing, just arguing
> about. (BTW, once the remaining Unicode string cases are fixed, then I'll
> have one Windows build where the students can use scrollbars reliably, but
> not file paths with Unicode characters, and another Windows build where the
> students can use Unicode file paths, but not scrollbars...)


> - Slider2D background transparency is not as expected https://github.com/
> supercollider/supercollider/issues/3704 -- I found that when I was trying
> to work around "backgroundImage_ doesn't work for all views"
> https://github.com/supercollider/supercollider/issues/2575
>
> - RangeSlider doesn't obey "stretch" in line layouts https://github.com/
> supercollider/supercollider/issues/3696
>
> - Slider orientation may be wrong in layouts https://github.com/
> supercollider/supercollider/issues/2824
>
> - Window's endFrontAction called inappropriately when user clicks on title
> bar https://github.com/supercollider/supercollider/issues/1664
>
> - Enhancement, not bug, but it's something that's useful but currently
> impossible in layouts: "Expand to fit" size setting for views in layouts
> https://github.com/supercollider/supercollider/issues/3695


Thanks! I don't use GUI features much so this is less present on my radar.