Linux Tp numbers on mozilla-central varying too much to be useful

So, as today's sheriff, I started looking at performance numbers on
tinderbox.  I looked at the results of the pageload test on Linux.

We're running this test on 5 machines.  These machines each have a
number of spikes in their graphs, but the spikes don't seem
correlated with each other:
http://graphs.mozilla.org/graph.html#show=911694,395125,395135,395166,1431032

This makes it seem like these numbers aren't useful at all, since
it's hard to know which ones are trustworthy and which are not.  Do
we have any idea what's going on here?

I'm particularly worried about two things:

(1) The three original machines have been much less correlated with
each other since around the beginning of July.

(2) Machines have been spiking independently of each other since
around the 21st of August.

-David

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/