Re: Help requested: Interpreting results from bench.pl

On Thu, Mar 12, 2026 at 04:40:35PM +0000, Paul "LeoNerd" Evans wrote:
> But beyond that I don't really have a feel for how to interpret all of
> this. For instance, the commits between +4 and +5 have made the Ir_m1
> score drop from 91.83 all the way down to 47.18 - what does that
> actually mean? If I want to look in more detail into the code and try
> to work out what specifically slowed it down and how to improve it so
> it isn't as bad, what should I be looking for?

As the author of bench.pl, here is my ha'penny-worth.

TL;DR: I ignore all the "cache missed" fields, you should too.

One of the unique selling point of bench.pl is that it can measure very
*small* things: stuff which would be just noise in standard benchmarks. So
for example it can measure just the cost of executing a single PADSV op.

This has two main uses. First: while working on performance improvements,
you can see whether your changes have the desired effect. For example if
your goal is to remove some of the branching in pp_padsv(), bench.pl can
inform you that you have succeeded, or more importantly that you have
failed to do so; you can then look more closely to work out why your
change didn't have the expected result.

Secondly, it alerts to you things that have got worse. For example, I
recently ran bench.pl on blead against a perl from a year ago, which
showed some slowdowns. Because bench.pl results are more or less
completely reproducible, its possible to use bench.pl in a bisect mode to
find the exact commit which caused the change. Which is what I did, and
was able to track down the problem to a specific commit which had, as an
unintended side-effect, stopped making utf8 string constants COW. There is
zero chance that could have been spotted with conventional benchmarks.

Of course, bench.pl is just measuring a surrogate outcome: it doesn't say
whether your program will run quicker, but instead just says, for example,
that pp_add(), when both args are simple SvIOK values, uses a lot less
instructions and branches than it used to, which may well make some
programs go faster (and maybe some slower).

Because the typical benchmarks used by bench.pl are for very small things,
the 'missed cache' values are often zero, or a very small number; and so
when viewed as percentages, changes can often be large, e.g. going from 1
to 2 cache misses gives a 200% result. I mostly ignore cache misses.

I tend to use bench.pl for regression spotting as follows:

First run it against the perls of interest, saving the results to a file
with -w. This then allows the same results to be viewed in various ways by
reading the results back with -r. Note that with -j parallelism, all the
1000 or so benchmarks in t/perf/benchmarks can be run in a few minutes.

Then use the standard (percentage) display format along with --sort to
see which things have got the most worst. Then display those results with
--raw, to see what the actual values are. For example a really small
instruction like pp_padsv() might show a relatively big percentage change,
but with --raw you can see that this is only a single extra instruction
read or whatever.