Lies, damned lies, and statistics or thoughts on report-time-results

Jens Thiele <[email protected]> Thu, 10 Nov 2022 12:44:04 +0100
Newsgroups gmane.lisp.scheme.gauche
Message-ID <[email protected]>
Hi,

I really like time-these/report and report-time-results, but for the
problem at hand it does report misleading results because it reports the
rate per cputime and not per realtime:
"
(define (cputime result)  (+ (usertime result) (systime result)))
(define (rate result)     (/. (~ result'count) (cputime result)))
"

Benchmark: ran suisse, germany, south-america, each for at least 3.0 cpu seconds.
         suisse: 4.121 real, 3.120 cpu (2.690 user + 0.430 sys)@ 801.28/s n=2500
        germany: 6.590 real, 3.200 cpu (1.890 user + 1.310 sys)@2131.25/s n=6820
  south-america: 3.055 real, 3.020 cpu (2.700 user + 0.320 sys)@ 774.17/s n=2338

                  Rate suisse germany south-america
         suisse  801/s     --   0.376         1.035
        germany 2131/s  2.660      --         2.753
  south-america  774/s  0.966   0.363            --

What's happening?

For the problem at hand quite some time is spent on disk i/o especially
in the germany case (it could also be a network socket or some sleep
...)

Maybe report-time-results optionally could use the realtime for rate
calculation?

greetings,
jens