moving Mozilla1.8 tinderboxes to Buildbot - perf impact

Cross-posting to m.d.performance this time, as people who care about
this likely do not watch m.d.builds only.

I've been working on moving the Mozilla1.8 branch tinderboxes to be
under Buildbot control, by merging nightly builders into the release
automation project http://wiki.mozilla.org/Build:Release_Automation

You can see the results of the staging machine:
http://tinderbox.mozilla.org/Mozilla1.8-Staging/

Currently there are 4 automatically forced builds per day (to keep the
boxes from falling off of the Tinderbox waterfall, and make sure we
get a timely nightly), and besides that it only builds on checkin. If
we can teach Tinderbox server not to drop these builders, then I think
the ideal would be once forced nightly release and besides that only
build on checkin.

This is all well and good, but the perf impact I mention in the title
is that the current build machines build as often as possible, and
perf machines pick up the results as soon as each build come out.
Builds generally outrun the perf results, but we get a lot of perf
runs as a side-effect of the constant build cycles.

I think only building on checkin plus one nightly release is the right
thing to do, as it reduces cycle time and makes machines available for
other tasks (releases, verification, etc). However, if we do this then
the perf boxes will sit idle until a checkin, and we get fewer perf
runs as a result.

If this is a problem, one idea I had was to just do as many perf runs
as possible on the same build, until a new one is available. However
the perf machines currently use the start time of the build they are
testing, this has the nice effect of matching up with checkins. The
problem we'll run into is overlap between the time a perf run started
and the time a build for it finished, we won't easily be able to
correlate checkins with perf runs.

if we want to do multiple perf run results per build, this is
something we'd need to teach tinderbox server and the graph server
about, as they currently don't support this. I think what would make
more sense is being able to declare a revision (branch/timestamp combo
is the equivalent for CVS) and also the time that the perf run
happened, right now there is no difference.

Any thoughts? If 4 times per day plus on checkin is not enough to get
reasonable perf numbers, we can go live with continuous build cycling
instead (this is the only issue I am blocked on!), just seems a shame
to do a new build when all we really need is a new perf run.