Re: how to write snappy benchmark?

Mark, please re-read your post and recognize that there isn't a single
piece of helpful or constructive information in it. Why do you write a
message like that? Do you think that building an objective test of
responsiveness is such a dangerous goal that you have to discourage
someone from talking about it? Other people here seem to think it's a
good idea.

On Fri, Jul 04, 2003 at 03:35:32PM -0700, Mark Vojkovich wrote:
>    The point of most benchmarks is to control the test in a way
> that gives reliable results.  You are trying to benchmark a complex
> system in a way that probably isn't possible. 

If winbench can do application level tests of the number of times the
system can perform an operation in MSWord and produce reliable results
about which machines are faster than others, we can certainly do it
for some simple GTK widget response operations.

> Most of what you are suggesting will either be totally unreliable or
> will degenerate back into testing how fast X can draw lines and
> mostly irrelevant stuff like that.

If you know enough about X, GTK, KDE, Xaw, and win32 to say this, why
don't you look at what I suggested and propose something better? We'd
all love to hear your ideas. Testing responsiveness is really hard,
but really important.

> And what types of apps are you emulating? 

A version of the test would be written for a major widgetset and then
tested. For example, a version could be written using GTK+ which uses
the existing widgets for buttons, canvas, etc. Likewise for libXaw,
likewise for win32. It would take some serious debugging to know where
the problems came from, but the hope was to focus enough on the
application-level events to do comparisons across these different
implementations.

> One can write optimal versions of the test that work well for any
> system.  Basically, it's an apples to oranges comparison.

Of course you could write optimal versions of the test, but they would
be apples-to-apples comparisons. If it takes 3x longer to popup an
application menu on Xaw/X11 than on MFC/win32 on the same machine,
that would be really good information to know. If that were the case,
the only way for you to fix the Xaw version would be to fix Xaw.

However, if you have a feeling that the tests I proposed were not
good, do you have any _better_ ideas? Show me a better way to
benchmark responsiveness. Talk about some of the detailed reasons that
some of my proposed tests wouldn't get at responsiveness, so we can
understand how to do it better.

As an extreme example, it would be easy to simulate the user
requesting the copy and paste of 100,000 cells in a spreadsheet. By
doing this, I could certainly compare the performance of OpenOffice,
KSpread, Gnumeric, and Excel. I fail to see how one could write
"optimal" versions of these tests that would work well for any system
without rewriting the spreadsheet application.

I want to do the same thing at the widget level. Test how fast a GTK+
can respond to things. If you can really write an optimal version of
the test which will give GTK+ on linux the same performance as win32,
then either (a) they are equally responsive and you have fixed our
responsiveness problems, or (b) they are not equally responsive and we
need to come up with better tests of responsiveness.

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + [email protected]