Re: [PATCH RFC 0/3] SUNRPC: a latency floor for interactive clients via sparse-flow dispatch

On 25 Jun 2026, at 14:59, Chuck Lever wrote:
> My worry was that a sustained interactive flow (a tree walk holding a
> couple of RPCs in flight) never reaches zero, spends its 64 once, and
> then rides the normal queue.
> ...
> Could you run the same probe under your aggressor + 10ms hook?
> ...
> The question that settles it: for the interactive xprt under load,
> does the depth==0 fraction hold up, or do you see depth>=1 runs longer
> than 64?  If it holds, the trigger is fine [...].  If it collapses,
> that's the argument for a low-watermark refill rather than exact-zero.

Hey Chuck - I didn't need to run any tests to be conviced that your re-arm
problem exists, it seems obvious to me.  That said - I did send a clanker
out to run it, here's the response:

"

I ran it, and it collapses -- but as a sharp step, not a gradual fade,
which I think is the useful part of the answer.

Setup: your probe (depth = xpt_nr_rqsts, filtered to committed enqueues,
keyed by xprt) plus xpt_hi_credit, offsets from BTF.  The victim is a
sustained flow holding a fixed window W of RPCs continuously in flight on
one connection -- a deliberately conservative stand-in for the tree walk:
a real walk has dependency gaps that return it to depth 0, mine refills
instantly and so idles as rarely as possible.  Aggressor is the backlogged
worker set from the series; 10ms service hook throughout.  (This is a
different workload from the burst victim in the cover letter, which drains
fully and idles 50ms between bursts -- that one re-arms every cycle and is
not what's at issue here.)

Re-arm rate, committed-enqueue depth==0 fraction, by victim window:

    W=1 (strictly serial)   depth0 = 100%    credit stays 63
    W=2                     depth0 =  0.4%   credit pinned at 0
    W=4                     depth0 =  0.1%   credit pinned at 0
    W=8                     depth0 =  0.2%   credit pinned at 0

So the whole thing hinges on the transport going *completely* empty.  A
strictly serial flow re-arms on every single enqueue and rides the high
queue forever.  The moment a flow keeps even two RPCs continuously in
flight, the exact-zero condition essentially stops occurring: it spends
its 64 within the first ~64 enqueues and then lives in the bulk queue --
depth>=1 runs are not "longer than 64", they're unbounded.

What that costs depends on how deep the bulk queue is.  Victim per-op p50
(ms), vs window W and aggressor connection count K (floor = victim alone;
aggressor p50 ~ the bulk rate: 17 / 27 / 52 ms for K = 4 / 8 / 16):

    W    floor   K=4    K=8    K=16
    1     14      14     16     17     <- serial: protected at every K
    2     26      25     37     26     <- marginal, right on the edge
    3     20      19     24     39
    4     17      17     28     52     <- == aggressor: fully demoted
    6     16      17     42     79
    8     16      19     57    106     <- worse than the aggressor

Under light contention (K=4) bulk is shallow, so even a non-re-arming flow
stays near its floor and the demotion is invisible.  It only bites once the
bulk queue is deep, and then the threshold window shrinks as K grows -- by
K=16 anything holding >=3 in flight is at or past the aggressor's latency.
(W=8 lands *worse* than the aggressor because the demoted victim's own
workers then serialize behind each other in the bulk queue.)

Reconciling with your uncontended histogram (depth 0: 25.6%, 1: 62.3%):
that cold walk idles to zero a quarter of the time, which is plenty -- it
sits up at the W=1 end and would stay protected.  So the exact-zero trigger
is fine for genuinely sparse or gappy flows.  Where it's brittle is the
continuously-pipelined flow with no gaps -- aggressive readahead, an app
doing async I/O, a v4.1 client that keeps its slot table full -- which
demotes wholesale the first time it sustains a couple RPCs under load.

So I read this as the argument for the low-watermark refill you floated:
re-arm when depth drops below a small threshold rather than at exactly 0.
A watermark of even 2-4 would move the W=1 column's behavior out to cover
the modestly-pipelined flows, while still excluding a real bulk client
(which sits far above any sane watermark).  It doesn't reintroduce identity
and it's a one-line change to the refill condition.

I could spin that as a 4th patch for v1 and re-run this sweep against it --
I'd expect the protected band to widen from "W<=1" to "W<=watermark".  Does
a fixed small watermark match your intuition, or were you thinking of
deriving it (thread count, or the slot-table hint you mentioned)?

The probe harness (kprobe sweep + the load generator) is standalone if you
want to reproduce any of this.

"

Ben