Re: Unexplained variance in run-time of simple program (part 2)

On 08/04/2026 11:29, John D. McCalpin wrote:

> I am still not clear on exactly what you mean by “executing the benchmark 2^16 times”.
> Are you forking a new process 2^16 times, or is a single process executing a block of code 2^16 times.
> The difference is critical for determining where to look for sources of performance variability.
Based on your very insightful feedback (thanks again!) I currently use this wrapper:

#include <stdlib.h>
#include <stdio.h>

typedef unsigned int u32;

// Ignore id: process will be pinned to core 3
// Ignore hi: t1-t0 < 2^32 cycles (1.3 seconds)
static inline u32 rdtscp(void)
{
	u32 hi, lo, id;
	asm volatile("rdtscp" : "=d"(hi), "=a"(lo), "=c"(id));
	return lo;
}

extern void init(void);
extern void spin(void); // THE CODE UNDER BENCHMARK

static u32 v[4];

int main(int argc, char **argv)
{
	if (argc != 3) return 1;

	const u32 N = strtoul(argv[1], NULL, 10);
	const u32 S = strtoul(argv[2], NULL, 10);

	void *buf = aligned_alloc(4096, N * sizeof v);

	for (u32 n = 0; n < N; ++n)
	{
		for (u32 i = 0; i < 4; ++i)
		{
			u32 t0 = rdtscp();
			for (u32 s = 0; s < S; ++s) spin();
			u32 t1 = rdtscp();
			v[i] = t1 - t0;
		}
		__builtin_ia32_movntdq(buf + n * sizeof v, __builtin_ia32_movntdqa((void *)v));
	}

	for (int i = 0; i < N * 4; ++i) printf("%u\n", ((u32 *)buf)[i]);

	return 0;
}

N times: time the code 4 times, and transfer the 4 results to memory using a non-temporal store.
Once measuring is done, print all the results.
Would it be more efficient to write a full cache line with movntdq? (64 bytes vs 16 bytes)


The baseline code (to check for correct results) is:

spin:
	xor eax, eax
	mov ecx, 1000
loop:
	times 60 inc eax ; 120 bytes
	dec ecx
	jnz loop ; 8-bit offset
	ret


I ran the following experiments:
./run.sh ./baseline 200000 1
./run.sh ./baseline 200000 2
./run.sh ./baseline 200000 4


The results in histogram (in 50-cycle steps) :

histo.1
99000: 199999
99100: 1

histo.2
197950: 85290
198000: 114706
198050: 1
198200: 2
198250: 1

$ cat histo.4
395900: 110104
395950: 89891
396000: 1
396150: 2
396200: 1
396450: 1

1st col: cycles @ 3.3 GHz
2nd col: number of times result fell in that interval

I think this is looking pretty good!
I just need to test on my actual code that uses L1$.

Regards