Re: Unexplained variance in run-time of simple program (part 2)

Responses inline.


> On Apr 7, 2026, at 3:52 PM, Marc Gonzalez <[email protected]> wrote:
> 
> Hello Doctor Bandwidth,
> 
> I was secretly hoping you would chime in! :)
> 
> You've moved to Spain if I understand correctly?

I moved to Spain last year and have been working at the Barcelona Supercomputing Center for about 9 months.


> 
> On 07/04/2026 10:37, John D. McCalpin wrote:
> 
>> Going back to the example discussed at [in 2025-09], it looks like
>> the benchmark takes ~2500 ns and that you are executing the
>> benchmark 2^16 times.   Does that mean you are launching a new
>> process 2^16 times?  If so, exactly what mechanism are you using?
>> The variation you are reporting of 0.1 to 0.3 microseconds per
>> execution seems very small to me.  Are you sure that the variation
>> is in the "execution time", or could it be associated with the
>> launching of the execution?   If the benchmark code is being
>> executed 2^16 times in a loop in a single execution, then the
>> situation quite different and performance counters should be very
>> effective at determining the cause of the variability.
> 
> 
> I understand that process creation is a costly operation.
> I do run the benchmark several times from the same process.
> (I plan on revisiting your above remarks after I digest them.)

I am still not clear on exactly what you mean by “executing the benchmark 2^16 times”.
Are you forking a new process 2^16 times, or is a single process executing a block of code 2^16 times.
The difference is critical for determining where to look for sources of performance variability.


>> For the performance counter side of things, I would recommend
>> programming the performance counters externally to the benchmark
>> process and using inline RDPMC instructions to get the counter
>> values (inside the benchmark executable).
> 
> 
> Why do you suggest programming the PMCs from outside the benchmark process?
> (Is it perhaps because it's simpler than using the kernel API?)
> 
> It never occurred to me that I could read the PMCs from user-space!
> 
> This is what I've been doing until now:
> 
> int open_event(u64 type, u64 config, int fd)
> {
> struct perf_event_attr attr = {
> .type = type,
> .size = sizeof(attr),
> .config = config,
> .read_format = PERF_FORMAT_GROUP,
> };
> 
> return syscall(SYS_perf_event_open, &attr, 0, 3, fd, 0);
> }
> 
> int main_fd = open_event(PERF_TYPE_HARDWARE, PERF_COUNT_HW_CPU_CYCLES, -1);
> open_event(PERF_TYPE_HARDWARE, PERF_COUNT_HW_INSTRUCTIONS, main_fd);
> open_event(PERF_TYPE_RAW, UOPS_EXECUTED, main_fd);
> open_event(PERF_TYPE_RAW, EXEC_STALLS, main_fd);
> 
> for (int i = 0; i < 1000000; ++i)
> {
> ioctl(main_fd, PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP);
> for (int i = 0; i < N; ++i) spin(ctx);
> if (read(main_fd, v, sizeof v) < sizeof v) return 2;
> printf("%lu %lu %lu %lu\n", v[1]/N, v[2]/N, v[3]/N, v[4]/N);
> }
> 
> The problem with that technique is that the PMCs are "polluted" by
> the exit from ioctl & the entry into read.
> 
> Reading the PMCs from user-space with RDPMC solves this issue!

The perf_event subsystem is able to read the performance counters without requiring a kernel transition at every read, but internally the process is rather complex.  For fine-grain measurements on code that (1) can be bound to a single logical processor and (2) be read frequently enough that overflows can be unambiguously detected, there is no reason not to use the RDPMC instruction directly.  (Recent versions of the Linux kernel have started disabling the RDPMC instruction for user mode by clearing CR4.PCE by default, and only enabling it when the perf_event library wants to use it.  This irritates me more than I can easily express, but it can be overridden (in the kernels I have been using) by echoing a value of “2” to /sys/devices/cpu/rdpmc.)




> Found this interesting suggestion from 10 years ago:
> https://community.intel.com/t5/Software-Tuning-Performance/How-to-read-performance-counters-by-rdpmc-instruction/m-p/1009043
> I think you might be familiar with the responder ;)
> 
> 
>> I am not certain what the lowest overhead mechanism might be for
>> getting those performance counter values out of the program — I
>> would probably try attaching to a persistent System V shared memory
>> segment and simply storing the values in memory for later post-processing.
> 
> 
> Are you implying that simply using printf might disturb the caches
> from the write calls? (I redirect the output to a tmpfs.)

The degree of disturbance depends on the size of the “benchmark” being measured.  If the code executes millions of instructions between output points, then it is probably OK to just use “printf()”, but if you are reading performance counters around loops of a thousand cycles then you need to be more careful about how the data is output (especially if the code under test is measuring access to the caches).

A technique I commonly use is to allocate a small output buffer (much smaller than the L1 DCache) and zero it immediately before the first performance counter reads so that it will be resident in the L1 cache in a writable state.  I then run a block of iterations and save the performance counter values in the buffer.  Once the buffer is filled I copy the contents to a larger buffer in memory, re-zero the small output buffer, and start the next block of iterations.  If I want to be careful not to cause extra cache pollution, I copy the small output buffer to the large output buffer using streaming stores. 
This approach is useful for getting ensembles of performance counter results for repetitions of an identical code loop, or for collecting performance counter results for the same loop with different loop bounds, strides, or offsets.  

Changing the performance counter programming is always expensive (since the WRMSR instruction can only be executed in kernel mode), so I either change those externally to my program or internally but only around very large (millions of cycles) blocks of tests.   If the counters need to be changed at finer granularity the benchmark can be embedded in a loadable kernel module, but the execution time of the required process binding and WRMSR instructions is still substantial. 


> 
>> On Intel SKX processors I measured the overhead of an RDPMC
>> instruction at as low as ~25 cycles, with RDTSCP instructions taking
>> a little bit longer (~40 cycles), but most importantly inline RDPMC
>> does not involve any uncontrolled code paths or any hardware
>> accesses outside the core.  RDTSCP is similar, but might be
>> accessing off-core resources — the timing depends on both the core
>> and uncore clock frequencies.
> 
> Thanks for the great suggestion. I'll be reading up on RDPMC.
> 
> Why do you think I would need RDTSCP?
> At the moment, I just pin all cores at 2 GHz & count cycles using PMCs.

RDTSCP is useful to have in the toolbox because (1) it is almost never disabled (via CR4.TSD), (2) it has lower overhead than other timers, (3) it has some ordering restrictions on execution, (4) the instruction also returns the contents of MSR IA32_TSC_AUX in the %ecx register.  The interfaces that I use are available at https://github.com/jdmccalpin/low-overhead-timers.

There is one exception to (2) and that is using one of the CYCLES events in the programmable core performance counters.  On SKX processors, for example, the programmable event CPU_CLK_UNHALTED can be read in about 24 cycles, while RDSTCP takes about 40 cycles and RDTSC is in the middle somewhere.  CPU_CLK_UNHALTED is quite useful for interval measurements inside a benchmark because the processor is almost always going to be active all the time during the benchmark execution, and “core cycles” is often the most convenient unit of measurement.  (I often set the core frequency to match the TSC frequency to minimize the probability of unwanted frequency transitions and to make the arithmetic easier.  You still need to watch out for unexpected stalls and frequency transitions if you are using 256-bit or 512-bit SIMD instructions.)

RDTSCP is “half-ordered” with respect to other instructions.  It will not execute until all prior instructions have executed, but it does not prevent future instructions from being started earlier than the RDTSCP executes.  The CPUID instruction in the reference below is *very expensive* and not recommended for fine-grain timing.  If you want to ensure that no instructions have started execution before you read the timer, then use the sequence:
	mfence
	lfence
	rdtscp (or rdtsc in this particular case)
before the code being monitored, and the sequence:
	rdtscp
	lfence
after the code being monitored.   
I very rarely find a need for the lfence instructions.  It prevents later instructions from starting early (which could add cycles before the RDTSCP instruction grabs the TSC), but I have never found a case where this biased the results by an amount that was visible above the background noise.  (RDTSC and RDTSCP are microcoded instructions that have variable execution times because they need to communicate with off-core resources to obtain a TSC value that is independent of the CPU core frequency and the history of CPU active and halted cycles.)

On Linux systems, IA32_TSC_AUX is programmed to contain the socket number and logical processor number in concatenated 12-bit fields, and the RDTSCP instruction guarantees that the TSC value (returned as two 32-bit values in registers %eax and %ebx) and the IA32_TSC_AUX value were collected atomically on the same logical processor.  So RDTSCP provides an extremely low-overhead mechanism to determine the logical processor number where the process is currently executing.  As a simple hardware instruction, it won’t do anything that might encourage the OS to consider re-scheduling the process (while calling a system library routine like sched_get_affinity() might).  This can be very helpful when trying to debug process affinity problems, e.g., in software environments with multiple agents able to make affinity requests, e.g., SLURM + OpenMP runtime + MPI runtime.



> Reference for myself:
> https://www.codestudy.net/blog/difference-between-rdtscp-rdtsc-memory-and-cpuid-rdtsc/
> 
> Regards
> 
>