Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

On 2026-08-23 15:00:43 [+0000], Nikita Taranov wrote:
> Hi,
Hi,

> The commit message anticipates a slow transition, but as latency:
> 
>   "The side effects would be that on auto scaling the new hash is used
>    later and the SET_SLOTS prctl() will block longer."
> 
> What I am reporting is that under load it may not complete at all for
> tens of seconds.

Hmm. There are two things that can an effect how quick the transition
is:
- The completion of an RCU grace period to ensure all futex users moved
  from per-CPU reference counting to atomic
- The possibility of all FUTEX participants (all threads of the task) to
  not use futex for a while.

The latter means, if you multiple FUTEX users all the time (as seen in
your example) then the reference counter never drops to 0. As such, the
transition to the new private hash bucket can not happen.

> The trigger needs futex activity to overlap a hash growth, which is just
> an ordinary thread-pool ramp: the growth is requested while the earlier
> threads are already working.  That is not specific to startup -- a pool
> that grows later under load is hit at least as hard, see below.  If the
> process goes quiet for a moment afterwards the count drains and everything
> is fine, which is why this is easy to miss and why the symptom is
> bimodal.

Right. As long as you allow it to settle then everything will be fine.
Once the reference counter was allowed to drop to 0, the first user will
install the new private-hash and all other threads wait until it is
done.

…
> 96 threads, each issuing FUTEX_WAKE_PRIVATE on a private futex with no
> waiters, so the syscall does little besides take and drop a hash
> reference.  Threads start work as they are created -- deliberately no
> start barrier, since an idle window after thread creation lets the pivot
> complete and hides the problem.
> 
> Build with
> 
>   gcc -O2 -pthread futex_hash_repro.c -o futex_hash_repro
> 
>   # let the hash auto-scale (default)
>   $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 0
>   threads=96 slots(start=0 set=0 end=512) 5.01s   13.0 Mops/s
>   threads=96 slots(start=0 set=0 end=512) 5.01s    9.4 Mops/s
>   threads=96 slots(start=0 set=0 end=512) 5.01s   11.7 Mops/s
> 
>   # pre-size the hash before any thread exists
>   $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 512
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1049.0 Mops/s
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1050.2 Mops/s
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1049.8 Mops/s

You might have hit the sweet spot with your CPU. I have here a 144 CPU
box and 

| bigeasy@herakles:~$ ./futex_hash_repro 140 5 0
| threads=140 slots(start=0 set=0 end=1024) 5.00s 659.2 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 4096
| threads=140 slots(start=0 set=4096 end=4096) 5.00s 789.1 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 4096
| threads=140 slots(start=0 set=4096 end=4096) 5.00s 786.4 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 0
| threads=140 slots(start=0 set=0 end=1024) 5.00s 631.1 Mops/s
|
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 0
| threads=96 slots(start=0 set=0 end=512) 5.00s 704.4 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 512
| threads=96 slots(start=0 set=512 end=512) 5.00s 737.4 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 512
| threads=96 slots(start=0 set=512 end=512) 5.00s 733.1 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 0
| threads=96 slots(start=0 set=0 end=512) 5.00s 690.6 Mops/s

It is not as bad as you describe. Could you try v7.2 which has commit
a734d9fca84e1 ("futex: Optimize futex hash bucket access patterns")
which might help to settle the counter.

> The magnitude depends on how far mm->futex_atomic has to travel.  Across
> the three SNC clusters of this socket it is ~11x; confining the same 96
> threads to one cluster brings the run to ~3.

Not sure what we could do here. One idea might be to block further
futex syscalls so they don't acquire a new reference on the existing
hash and allow a transition to the new hash more quickly.

Sebastian