Re: More strict testing requirements?

Am 2018-11-17 13:56, schrieb David Edmundson:
> On Fri, Nov 16, 2018 at 5:29 PM Martin Flöser <[email protected]>
> wrote:
> 
>> Hi all,
> 
>> The root problem was obviously the CI system no longer
>> installing KWin and running the tests from the build directory.
> 
> Interestingly KDE applications on CI install before running tests.
> 
> https://build.kde.org/job/Applications/job/konsole/job/kf5-qt5%20SUSEQt5.9/
> 
> I know frameworks deliberately changed, David F gave an Akademy
> presentation about why this system is so much better - though I'm not
> very convinced.
> 
> I'm not sure what the story with Plasma is.

According to Ben in https://phabricator.kde.org/T9988 the tests are run 
prior to installation. If that is not a general rule for build.kde.org 
it should be changed for Plasma to run tests after installation IMHO. 
Plasma heavily depends on KPackage which does not support this method.

> 
>> Given the experience I would like to suggest that we setup rules for
>> 
>> handling failures.
>> 
>> 1. Prior to push ctest should be run locally
>> 2. Whenever a test fails on build.kde.org [1] no change is allowed
>> to be
>> pushed which does not fix the failure.
>> 3. If a failure on build.kde.org [1] is not fixed after 3 days the
>> change
>> which introduced the regression gets reverted
> 
> I'm fine with the sentiment, but I don't like blind rules and
> deadlines. It would have been stupid last week to go through and
> revert all the xwayland clipboard patches just because some test was
> failing on CI.
> 
> I think we should be able to handle doing things on a case-by-case
> basis.

The third rule is basically meant as the exit rule to rule number 2. The 
first two are for me a must. If we allow changes to go in after a 
regression we can just remove the test suite. We had several additional 
breakages the last two weeks and that is clearly related to the test 
suite having been in a bad state.

Now the third rule is meant for cases that a) a developer thinks he 
doesn't need testing and b) for the case a developer pushes a breaking 
change just before going on vacations. If we want to enforce rule 2 it 
would mean the community either has to fix the work of others (which 
sucks) or has to wait till the dev gets back from vacations (which sucks 
as well). By allowing the community to revert such a change no dev can 
go "you are evil!" and at the same time we have an exit strategy for 
rule 2. Of course that should always be a case by case decision which 
doesn't mean we shouldn't write down rules.

Cheers
Martin