Re: More strict testing requirements?

On Fri, Nov 16, 2018 at 5:29 PM Martin Flöser <[email protected]> wrote:

> Hi all,
>
> The root problem was obviously the CI system no longer
> installing KWin and running the tests from the build directory.


Interestingly KDE applications on CI install before running tests.

https://build.kde.org/job/Applications/job/konsole/job/kf5-qt5%20SUSEQt5.9/

I know frameworks deliberately changed, David F gave an Akademy
presentation about why this system is so much better - though I'm not very
convinced.

I'm not sure what the story with Plasma is.


> Given the experience I would like to suggest that we setup rules for
> handling failures.
>
> 1. Prior to push ctest should be run locally
> 2. Whenever a test fails on build.kde.org no change is allowed to be
> pushed which does not fix the failure.
> 3. If a failure on build.kde.org is not fixed after 3 days the change
> which introduced the regression gets reverted
>
>
I'm fine with the sentiment, but I don't like blind rules and deadlines. It
would have been stupid last week to go through and revert all the xwayland
clipboard patches just because some test was failing on CI.

I think we should be able to handle doing things on a case-by-case basis.

David