Re: moving Mozilla1.8 tinderboxes to Buildbot - perf impact

On Dec 28 2007, 6:43 pm, Robert Helmer <[email protected]> wrote:
> Cross-posting to m.d.performance this time, as people who care about
> this likely do not watch m.d.builds only.
>
> I've been working on moving the Mozilla1.8 branch tinderboxes to be
> under Buildbot control, by merging nightly builders into the release
> automation projecthttp://wiki.mozilla.org/Build:Release_Automation
>
> You can see the results of the staging machine:http://tinderbox.mozilla.org/Mozilla1.8-Staging/
>
> Currently there are 4 automatically forced builds per day (to keep the
> boxes from falling off of the Tinderbox waterfall, and make sure we
> get a timely nightly), and besides that it only builds on checkin. If
> we can teach Tinderbox server not to drop these builders, then I think
> the ideal would be once forced nightly release and besides that only
> build on checkin.
>
> This is all well and good, but the perf impact I mention in the title
> is that the current build machines build as often as possible, and
> perf machines pick up the results as soon as each build come out.
> Builds generally outrun the perf results, but we get a lot of perf
> runs as a side-effect of the constant build cycles.
>
> I think only building on checkin plus one nightly release is the right
> thing to do, as it reduces cycle time and makes machines available for
> other tasks (releases, verification, etc). However, if we do this then
> the perf boxes will sit idle until a checkin, and we get fewer perf
> runs as a result.
>
> If this is a problem, one idea I had was to just do as many perf runs
> as possible on the same build, until a new one is available. However
> the perf machines currently use the start time of the build they are
> testing, this has the nice effect of matching up with checkins. The
> problem we'll run into is overlap between the time a perf run started
> and the time a build for it finished, we won't easily be able to
> correlate checkins with perf runs.
>
> if we want to do multiple perf run results per build, this is
> something we'd need to teach tinderbox server and the graph server
> about, as they currently don't support this. I think what would make
> more sense is being able to declare a revision (branch/timestamp combo
> is the equivalent for CVS) and also the time that the perf run
> happened, right now there is no difference.
>
> Any thoughts? If 4 times per day plus on checkin is not enough to get
> reasonable perf numbers, we can go live with continuous build cycling
> instead (this is the only issue I am blocked on!), just seems a shame
> to do a new build when all we really need is a new perf run.

Are you asking specifically for the 1.8 tree or assuming the same
infra will be used for 1.9 and Moz2?   I'd say 4 builds per day as a
starter is just fine.  It would be great to get the perf machines/
graph server to be able to submit multiple runs per build (to reduce
jitter amongst other things) but I wouldn't block on this.  Assuming
we get a perf run after every checkin (as long as one isn't already
running) or a min of 4 per day sounds fine IMHO.