More strict testing requirements?

Hi all,

with build #195 KWin is green again. Thanks everyone for helping making 
the tests pass again.

Personally I think it was a bad experience that we had failing tests. 
Due to having failing tests, we immediately got the problems that code 
changes introduced further failing tests. I don't want to blame anyone 
here, that's just the normal thing to happen if tests fail and also I 
did submit a change which started to fail as soon as the other failures 
got fixed. The root problem was obviously the CI system no longer 
installing KWin and running the tests from the build directory. That's a 
requirement KWin never in its history fulfilled and also cannot fulfill 
as long as dependencies do not support it. Also it's really difficult to 
test locally when one has a system install, but that's another story.

Given the experience I would like to suggest that we setup rules for 
handling failures.

1. Prior to push ctest should be run locally
2. Whenever a test fails on build.kde.org no change is allowed to be 
pushed which does not fix the failure.
3. If a failure on build.kde.org is not fixed after 3 days the change 
which introduced the regression gets reverted

I think point 1 is the most important as that should prevent most 
issues. To support this I added the testing instructions and fixed the 
activities test when running in ctest. Overall it's just a few minutes 
to wait and one knows whether the change is good or bad.

Unfortunately we still have a few "flaky" tests as we can see in build 
#193 and #194. There's nothing wrong with the tests, the problem is with 
build.kde.org. E.g. build #194 failed in Test::setupWaylandConnection - 
most likely because signal spies timed out. We just don't get the CPU 
time. This has been a long standing issue on build.kde.org. I already 
noticed in the past that our tests fail sporadically when building krita 
at the same time. Which was btw. the case during #194. I'll contact 
sysadmins about it. Hopefully there's a solution to it like just rerun 
the test.

So any opinion on the idea of setting up the rules?

Cheers
Martin