Re: Unexplained variance in run-time of simple program (part 2)

On 26/03/2026 16:24, Marc Gonzalez wrote:

> Past discussion:
> Large(ish) variance induced by SCHED_FIFO / Unexplained variance in run-time of trivial program
> https://lore.kernel.org/linux-rt-users/[email protected]/
>
> SYNOPSIS:
> I have a simple(*) program.
> I just want to know how long the program takes to run.

I probably need to start from the absolutely MOST simple program possible, then work from there.

spin:
	mov ecx, 1
	shl ecx, 10
	xor eax, eax
loop:
	times 60 inc eax
	dec ecx
	jnz loop
	ret

I.e. just a long dependency chain of 61440 increment instructions.

Experimental setup:

- Boot kernel 6.8 with nohz_full=3 rcu_nocbs=3 isolcpus=nohz,domain,managed_irq,3 irqaffinity=0-2 nosmt mitigations=off nosoftlockup tsc=reliable log_buf_len=16M single
- Prepare system with:
echo -1 > /proc/sys/kernel/sched_rt_runtime_us
for I in 0 1 2 3; do echo userspace > /sys/devices/system/cpu/cpu$I/cpufreq/scaling_governor; done
for I in 0 1 2 3; do echo   2000000 > /sys/devices/system/cpu/cpu$I/cpufreq/scaling_setspeed; done
- Call spin() 10M times, recording the following events every time:
HW_CPU_CYCLES, HW_INSTRUCTIONS, UOPS_EXECUTED, EXEC_STALLS

Cycle-count distribution:

63500: 470202
63625: 8747244
63750: 770281
63875: 12143
64000: 105
64125: 16
64250: 3
64375: 2
64500: 1
64875: 1
68375: 1
91250: 1

This looks good, as far as I can tell.

 4.70% within [63500, 63625[
87.47% within [63625, 63750[
 7.70% within [63750, 63875[
 0.12% within [63875, 64000[

Covers 99.99% of samples.

Therefore, I think I would get very stable results by simply:
- running 100 iterations of the code
- discarding the worst 10 (20? 50?) outliers (what about the best outliers?)
- taking the arithmetic mean (or the median?)

I note that the benchmark overhead itself seems to be ~3000 cycles
(ioctl to reset the event counters + read to copy the event counters to user space)
3000 cycles is a whopping 5% of what I'm trying to measure.
It might make sense to call spin() a few times (2? 4? 10?) to lower the overhead's impact...

As always, happy to read anyone's input / insight into the process :)

Regards