Re: how to write snappy benchmark? idea

Hi,

David Jeske wrote:
> Mark, please re-read your post and recognize that there isn't a single
> piece of helpful or constructive information in it. Why do you write a
> message like that? Do you think that building an objective test of
> responsiveness is such a dangerous goal that you have to discourage
> someone from talking about it? Other people here seem to think it's a
> good idea.
> 
> On Fri, Jul 04, 2003 at 03:35:32PM -0700, Mark Vojkovich wrote:
> 
>>   The point of most benchmarks is to control the test in a way
>>that gives reliable results.  You are trying to benchmark a complex
>>system in a way that probably isn't possible. 
> 
> 
> If winbench can do application level tests of the number of times the
> system can perform an operation in MSWord and produce reliable results
> about which machines are faster than others, we can certainly do it
> for some simple GTK widget response operations.
> 
> 
>>Most of what you are suggesting will either be totally unreliable or
>>will degenerate back into testing how fast X can draw lines and
>>mostly irrelevant stuff like that.
> 
> 
> If you know enough about X, GTK, KDE, Xaw, and win32 to say this, why
> don't you look at what I suggested and propose something better? We'd
> all love to hear your ideas. Testing responsiveness is really hard,
> but really important.
> 
> 
>>And what types of apps are you emulating? 
> 
> 
> A version of the test would be written for a major widgetset and then
> tested. For example, a version could be written using GTK+ which uses
> the existing widgets for buttons, canvas, etc. Likewise for libXaw,
> likewise for win32. It would take some serious debugging to know where
> the problems came from, but the hope was to focus enough on the
> application-level events to do comparisons across these different
> implementations.
> 
> 
>>One can write optimal versions of the test that work well for any
>>system.  Basically, it's an apples to oranges comparison.
> 
> 
> Of course you could write optimal versions of the test, but they would
> be apples-to-apples comparisons. If it takes 3x longer to popup an
> application menu on Xaw/X11 than on MFC/win32 on the same machine,
> that would be really good information to know. If that were the case,
> the only way for you to fix the Xaw version would be to fix Xaw.
> 
> However, if you have a feeling that the tests I proposed were not
> good, do you have any _better_ ideas? Show me a better way to
> benchmark responsiveness. Talk about some of the detailed reasons that
> some of my proposed tests wouldn't get at responsiveness, so we can
> understand how to do it better.
> 
> As an extreme example, it would be easy to simulate the user
> requesting the copy and paste of 100,000 cells in a spreadsheet. By
> doing this, I could certainly compare the performance of OpenOffice,
> KSpread, Gnumeric, and Excel. I fail to see how one could write
> "optimal" versions of these tests that would work well for any system
> without rewriting the spreadsheet application.
> 
> I want to do the same thing at the widget level. Test how fast a GTK+
> can respond to things. If you can really write an optimal version of
> the test which will give GTK+ on linux the same performance as win32,
> then either (a) they are equally responsive and you have fixed our
> responsiveness problems, or (b) they are not equally responsive and we
> need to come up with better tests of responsiveness.
> 
  we could  count the number of lignes of code are reached when an event 
is received by gtk until one kind of popup or window is displayed .

only my two cents

Eric Streit