Broken build since four days

Hi Jens,

I spent the entire afternoon yesterday to fix the Plone 5 build that has 
been broken since four days. The Plone 5 AT job is still broken.

The idea of a CI system is that people are notified immediately of 
regressions, so those regressions are easy to fix, because they are 
fixed by the person who introduced the bug in the first place and the 
person usually worked on the code recently anyways.

If I (or anybody else) fixes those regressions for you, it takes me a 
lot longer than you or anybody who worked with the code recently.

I know our CI system is not perfect and it's sometimes hard to see what 
went wrong and we sometimes get false positives. Though, I'm constantly 
working on improving that situation and I always monitor the build, 
notify people manually and fix or restart the build if things go wrong.

In order to keep a green build, I usually just revert commits that break 
the build and notify people so they can just fix the regression later. 
This only works though if people do small commits that are easily 
revertable and people do not commit on an already broken build.

It is just not acceptable to keep the build broken for more than a 
couple of hours. A red build keeps me up at night, because this is an 
invitation for more regressions that are introduced without notice, 
leading to a build that is completely broken and a situation when nobody 
feels responsible because the CI system can't tell people any longer if 
they break things. See this recent discussion to illustrate what I mean:

https://github.com/plone/plone.app.jquerytools/commit/4fb9eae8b7b3a5a16a9992fc069ed560d7afdfe8#commitcomment-6734229

We had this situation before and usually David, Eric, me and others have 
to fix lots of failing tests before being able to make a release. This 
is not only highly frustrating, but it also means that new Plone 
releases take a lot more time than necessary.

-

About the Plone 5 AT job, it's still broken since this build:

http://jenkins.plone.org/job/plone-5.0-python-2.7-at/1481/

The Plone 5 build that triggered that build was:

http://jenkins.plone.org/job/plone-5.0-python-2.7/2589/changes#detail1

Those were the two commits that are responsible for the two tests that 
currently fail on the Plone 5 AT job:

https://github.com/plone/buildout.coredev/commit/29c9a070d6367d025c7e671fc2eff9c27c02be05

https://github.com/plone/buildout.coredev/commit/5dd14b8e7b51234d8ef62068b50408ff04342e6d

Please have a look and fix those test failures as soon as possible. As 
said before, I can't just revert commits any longer because things have 
become to complicated already.

We have lots of new Jenkins slave machines ready to put into production 
and this broken build is keeping us from being able to use them. I can't 
fully test the new machines if not all builds are green.

-

Another problem that was introduced recently is that it seems the Plone 
5 build runs grok tests now. Usually this is an easy fix. Though, since 
the build was broken for four days we have a lot of potential commits 
that might be responsible for that.

I did a temporary fix to get a green build:

https://github.com/plone/buildout.coredev/commit/6e53b212147927b4adad10b3788b48ad21e40645

Though, we have to figure out what went wrong. The package dependencies 
job does not show any grok dependencies:

http://jenkins.plone.org/view/Dependencies/job/plone-5.0-package-dependencies/ws/package-dependencies.txt

Do you have any idea what could have introduced those dependencies?

-

Please don't get me wrong, I highly appreciate your work on the PLIP and 
also on the open pull requests. It is perfectly ok to break the build 
every now and then, that is a normal part of the development process. 
Though, ignoring our CI rules and keeping the build broken for a couple 
of days makes all our jobs incredibly hard and we all have better things 
to do.

Cheers,
Timo

------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems