Re: Unexplained variance in run-time of simple program (part 2)

Hello John,

On 10/04/2026 19:16, Marc Gonzalez wrote: [snip]

I have come to the conclusion that it seems IMPOSSIBLE to get reproducible run-times,
even for simple programs, on "modern" systems (in quotes because Haswell, so 2013).

SETUP:
- System is idle, booted in single-user mode, with CPU3 isolated for benchmarks.
- CPU frequency is pinned at 2.8 GHz (nominal 3.3 GHz) to avoid thermal throttling & turbo boost.
- Benchmark is run as SCHED_FIFO 99, pinned to CPU3.
Linux version 6.8.0-110-generic
nohz_full=3 rcu_nocbs=3 isolcpus=nohz,domain,managed_irq,3 irqaffinity=0-2 nosmt mitigations=off nosoftlockup tsc=reliable log_buf_len=16M single


If I benchmark an absolutely trivial program (no memory access, 1 predictable loop)
then the run-time is extremely reproducible:

spin:
	xor eax, eax
	mov ecx, 1000
loop:
	times 60 inc eax ; 120 bytes
	dec ecx
	jnz loop ; 8-bit offset
	ret

This code is expected to run in 60,000 cycles
So the TSC pulsing at 3.3 GHz would count 70715 cycles + a few cycles for rdtscp & call overhead.

After measuring 12M calls to spin(), the run-time distribution is:

70725: 31
70730: 6235014
70735: 5142415
70740: 622531
70750: 1
70905: 3
70910: 4
103965: 1

(Not sure what's up with that 103965 outlier)

So the run-time is within 70730-70745 with 0,999996 certainty (11999960/12M)
A 15-cycle spread is 212 ppm. All is well & as expected for the trivial case.


Everything breaks down when I measure my actual (simple) program.
By simple, I mean:
- no system calls, no library calls, no I/O, just bit twiddling
- tiny code, small(ish) dataset

Here, I ran the code 15 MILLION TIMES, and took the MINIMUM run-time measured.
And I repeated this 20 times, expecting the various MINIMUMS to be quite similar:

66413
65707
66214
66277
67345
66640
66029
66693
67551
66771
66376
66536
66871
66177
65674
65435
66136
66223
66145
66305

The minimums of 15 MILLION runs fell somewhere between 65435 and 67551 :(
That's a spread of (67551-65435) / ((65435+67551) / 2) = 3.18% = 31823 ppm
150 times larger than for the trivial case.

Do I need to run the code 50M times? 100M times? to get consistent run-times? :(

Running out of ideas.

Maybe the memory traffic is generating a lot of "volatility" / non-determinism?
(But I don't think I'm even using all of L1$)
Maybe I need to disable the memory pre-fetchers?

I'm stumped...

Regards