Re: moving Mozilla1.8 tinderboxes to Buildbot - perf impact

On Jan 2, 12:51 pm, Schrep <[email protected]> wrote:
> On Dec 28 2007, 6:43 pm, Robert Helmer <[email protected]> wrote:
>
>
>
> > Cross-posting to m.d.performance this time, as people who care about
> > this likely do not watch m.d.builds only.
>
> > I've been working on moving the Mozilla1.8 branch tinderboxes to be
> > under Buildbot control, by merging nightly builders into the release
> > automation projecthttp://wiki.mozilla.org/Build:Release_Automation
>
> > You can see the results of the staging machine:http://tinderbox.mozilla.org/Mozilla1.8-Staging/
>
> > Currently there are 4 automatically forced builds per day (to keep the
> > boxes from falling off of the Tinderbox waterfall, and make sure we
> > get a timely nightly), and besides that it only builds on checkin. If
> > we can teach Tinderbox server not to drop these builders, then I think
> > the ideal would be once forced nightly release and besides that only
> > build on checkin.
>
> > This is all well and good, but the perf impact I mention in the title
> > is that the current build machines build as often as possible, and
> > perf machines pick up the results as soon as each build come out.
> > Builds generally outrun the perf results, but we get a lot of perf
> > runs as a side-effect of the constant build cycles.
>
> > I think only building on checkin plus one nightly release is the right
> > thing to do, as it reduces cycle time and makes machines available for
> > other tasks (releases, verification, etc). However, if we do this then
> > the perf boxes will sit idle until a checkin, and we get fewer perf
> > runs as a result.
>
> > If this is a problem, one idea I had was to just do as many perf runs
> > as possible on the same build, until a new one is available. However
> > the perf machines currently use the start time of the build they are
> > testing, this has the nice effect of matching up with checkins. The
> > problem we'll run into is overlap between the time a perf run started
> > and the time a build for it finished, we won't easily be able to
> > correlate checkins with perf runs.
>
> > if we want to do multiple perf run results per build, this is
> > something we'd need to teach tinderbox server and the graph server
> > about, as they currently don't support this. I think what would make
> > more sense is being able to declare a revision (branch/timestamp combo
> > is the equivalent for CVS) and also the time that the perf run
> > happened, right now there is no difference.
>
> > Any thoughts? If 4 times per day plus on checkin is not enough to get
> > reasonable perf numbers, we can go live with continuous build cycling
> > instead (this is the only issue I am blocked on!), just seems a shame
> > to do a new build when all we really need is a new perf run.
>
> Are you asking specifically for the 1.8 tree or assuming the same
> infra will be used for 1.9 and Moz2?

Same infrastructure, just rolling out to 1.8 branch first.

> I'd say 4 builds per day as a
> starter is just fine.  It would be great to get the perf machines/
> graph server to be able to submit multiple runs per build (to reduce
> jitter amongst other things) but I wouldn't block on this.  Assuming
> we get a perf run after every checkin (as long as one isn't already
> running) or a min of 4 per day sounds fine IMHO.

It would probably be ok, but I am starting to feel like we should get
the infra up and make sure it's stable before making visible changes
like this. We do not necessarily have to wait for the ability to
submit multiple runs per build, but I'd like a smooth transition
without worrying about side-effects from changing things like this.

A small number of people have expressed concerns to me directly about
this the performance aspect which is why I started the thread, and
I've told them that they need to weigh in here publicly, I'm not going
to make the perf or "our build system sometimes falls over" arguments
for them :)

I think we're probably be better off getting everything onto the new
infrastructure before we try to make any changes like this, if only to
separate the concern about this feature from the actual rollout (I
wouldn't want to have people calling for us to put the old system
back, for example). Once we're on buildbot, it's a simple switch in
the config file to turn BonsaiPoller on or off, so we can make that as
an isolated change, and be able to fall back to a system that we know
works.