Re: Testing

---- On Fri, 18 May 2018 08:31:03 +0800 <[email protected]> wrote ---- 
> On Thu, May 17, 2018 at 7:48 PM,  <[email protected]> wrote:
> >  The specific hurdles in that EnvGen test were:
> >  - Need to test both scsynth and supernova. (Shouldn't all server tests run on both? Why do other server test authors get to skip that but I picked a d@@@ supernova-only issue to test...?) I made a lot of dumb mistakes with the logic.
> This is not something we've discussed before; AFAIK no test is specifically designed to run on either server. This is exactly case #1 above. We may need a spec here. :P

This was a case where a UGen exhibited different behavior in scsynth and supernova, despite being built from the same source code. If we haven't discussed testing both servers, it suggests that we assume behavior will be the same in both places -- but we have evidence that this isn't always the case. We might have reason, then, to make it mandatory to test both.

For efficiency, probably we should move the scsynth/supernova logic up to the top level: the runTest (or whichever) method would collect a list of the tests to be run (because you might be running just a subset), and pass it to a method that would:

1. Boot scsynth.
2. Run the chosen tests.
3. Quit scsynth.
4. Boot supernova.
5. Run them again.
6. Quit supernova.

In my test, I consolidated the results: the test passes if Pass(scsynth) && Pass(supernova). But that's not really necessary. You could just run them separately and report the results -- any failures would still be noted as failures.

>  - Success, in that test, is the *absence* of an early node-free. That's a missing feature -- how to detect when something *didn't* happen.
> Great, so this is something that should go into a new ticket or PR, if you can describe it precisely. If this is something a test needs to know then it's probably something a user will need to know as well.

OK.

There's probably no way around this, but another time sink for writing unit tests is that you can't be sure the test distinguishes between success and failure without testing the test against two SC versions. I couldn't be certain that the logic was correct unless I ran it against an unfixed supernova (and got failure) *and*, with identical testing code, ran it against a fixed supernova and got success. That got to be really time consuming and irritating in that case. Maybe I'm just uniquely dense about it, or maybe just had a bad day, or maybe people are assuming tests are easy and quick to write because they aren't testing the tests properly. I don't know... I felt I shouldn't submit a unit test unless I was sure that it would tell the two cases apart, and it took a flipping lot of time 
 to be sure of that.

>  I'd also add GUI. I keep finding little goofs... which are minor, but quite bad when they're user facing and the group of users is a classroom full of students.
> Such as?

- The scrollbar nightmare -- which we're still not fixing, just arguing about. (BTW, once the remaining Unicode string cases are fixed, then I'll have one Windows build where the students can use scrollbars reliably, but not file paths with Unicode characters, and another Windows build where the students can use Unicode file paths, but not scrollbars...)

- Slider2D background transparency is not as expected https://github.com/supercollider/supercollider/issues/3704 -- I found that when I was trying to work around "backgroundImage_ doesn't work for all views" https://github.com/supercollider/supercollider/issues/2575

- RangeSlider doesn't obey "stretch" in line layouts https://github.com/supercollider/supercollider/issues/3696

- Slider orientation may be wrong in layouts https://github.com/supercollider/supercollider/issues/2824

- Window's endFrontAction called inappropriately when user clicks on title bar https://github.com/supercollider/supercollider/issues/1664

- Enhancement, not bug, but it's something that's useful but currently impossible in layouts: "Expand to fit" size setting for views in layouts https://github.com/supercollider/supercollider/issues/3695

... just the ones that are still open.

> This would probably reveal a lot of missing features. Since so much of it comes from Qt, we should also be careful we're not testing other projects' code.

In the background image case at least, the documentation promises functionality ("you can assign a background image to any view") that the software doesn't deliver. Even if that's a Qt bug, we need to know about the limitation and document correctly.

hjh


_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/