Re: how to write snappy benchmark?

On Fri, 4 Jul 2003, David Jeske wrote:

> I'm interested in soliciting ideas about how to write this "snappy"
> benchmark that's been referred to a few times. While I don't have time
> to write the benchmarks, if I can get a clear picture of what will
> help, I may be able to project manage and pay to have them
> written. Here is how I would state the objective:
> 
>  To generate a set of numeric performance results for application and
>  widget level operations which help predict end-user perception of
>  responsiveness and speed. The operations used should provide a direct
>  comparison across across multiple platforms and setups, including
>  mutiple X11 setups, Windows, and MacOS.
> 
> I think achieving this objective is tricky, because it's hard to
> figure out exactly what the user percieves. Is it the redraw time? Is
> it the time from processing one event to another, independent of each
> redraw? Is it something else entirely? Here are some brainstorm ideas
> for operations to test, comments and additional suggestions welcome. I
> think the goal is to come up with a set of useful tests, and then
> simplify them into a small set of "tester applications" which can be
> written.
> 

   The point of most benchmarks is to control the test in a way
that gives reliable results.  You are trying to benchmark a complex
system in a way that probably isn't possible.  Most of what you
are suggesting will either be totally unreliable or will degenerate
back into testing how fast X can draw lines and mostly irrelevant
stuff like that.  And what types of apps are you emulating?  Dumb
ones?  One can write optimal versions of the test that work well
for any system.  Basically, it's an apples to oranges comparison.


			Mark.