Re: Benchmarking of different NetBSD versions

On Sat, Dec 17, 2005 at 02:34:53PM +0000, Martin J. Laubach wrote:
>   I'd like to redo fefe's benchmarks, so that we have a stable
> baseline from which we can re-run the benchmarks after major
> changes have gone in.
> 
>   My question is, is it worthwhile to do that under Qemu, or
> will I just be measuring emulator overhead and the real results
> will be lost in the noise?

Qemu is a bad idea, and so is Xen.  Both introduce multiple confounds
into the benchmarking process.

The best thing to do is to go out and get a few identical, cheap
hard drives, and simply tuck them away with each OS and revision to
be benchmarked installed in the test configuration.  Then you can
rerun your old benchmarks any time.  Of course, you can use ghost
to do something similar to this without the physical disks -- but
they are cheap, and convenient, so I just use the disks.

-- 
 Thor Lancelot Simon	                                      [email protected]

"The inconsistency is startling, though admittedly, if consistency is to be
 abandoned or transcended, there is no problem."		- Noam Chomsky