Re: [PATCH v3 3/7] KVM: pfncache: Use RCU for readers instead of a rwlock

On Fri, 2026-08-07 at 10:57 +0200, Woodhouse, David wrote:
> 
> We might even find that concurrent event channel delivery is *better*
> when it's serialised by a spinlock, than using rwlocks. Either way, I
> think we can *certainly* live with just flipping it to be a (raw)
> spinlock in the RT case.
> 
> I'll play.

Hm, initial results are that the spinlock version sucks for parallel
delivery, but rwlock isn't that great either. RCU scales nicely. I
built a test with multiple threads writing eventfds to delivery event
channel interrupts in parallel.

Slightly artificial as it doesn't really do any invalidations (although
that *is* the steady state for a running guest). I still need to do
some proper performance testing of a Xen guest with both emulated and
PCI passthrough network.

My friend says...
=================================

Deliveries/sec, spread layout (clean kernel, only kvm.ko swapped)

┌───────┬────────┬────────┬──────────────┐
│ vCPUs │ rwlock │  RCU   │ raw spinlock │
├───────┼────────┼────────┼──────────────┤
│ 1     │ 3.45M  │ 4.06M  │ 3.85M        │
├───────┼────────┼────────┼──────────────┤
│ 2     │ 5.72M  │ 7.37M  │ 5.70M        │
├───────┼────────┼────────┼──────────────┤
│ 4     │ 12.57M │ 16.60M │ 9.79M        │
├───────┼────────┼────────┼──────────────┤
│ 8     │ 16.25M │ 31.33M │ 6.70M        │
├───────┼────────┼────────┼──────────────┤
│ 16    │ 10.65M │ 50.48M │ 5.13M        │
└───────┴────────┴────────┴──────────────┘

RCU wins enormously — 4.7× the rwlock at 16 vCPUs, and it's the only
arm that keeps scaling (10.65M → 50.48M while rwlock regresses past 8
vCPUs). Mean latency stays flat at ~313ns where rwlock climbs to
1499ns.

The raw spinlock is a disaster under contention — 5.13M/s at 16 vCPUs,
half the rwlock, with 3108ns mean latency. It actively degrades from 4
vCPUs on.

So my queued_read_lock() reasoning was wrong. Yes it does an
atomic_add_return_acquire(), but readers still proceed concurrently
after that — they don't serialise the critical section the way a
spinlock does. The GPC read sections here (a test_and_set_bit() plus a
couple of writes) are long enough that mutual exclusion costs real
throughput. The line-bouncing on the lock word is not the dominant
term; serialisation of the critical section is.

That means the argument in your draft to Sean — that a naïve spinlock
is fine because single-vCPU users own the line anyway — holds for the
uncontended case (3.85M vs 3.45M at 1 vCPU, spinlock slightly ahead)
but fails badly for shinfo_cache, which is per-VM and genuinely
contended. Exactly the case you flagged as holding you back. The data
says your instinct to worry was right.