Re: [PATCH RFC 0/3] SUNRPC: a latency floor for interactive clients via sparse-flow dispatch

> an idle->active transport is granted a budget of 64 high-priority
> dispatches [...] refilled only when the transport next idles.
[...]
> Open questions for the list: is the budget (64 here) the right
> shape [...] And does the cooperative framing hold[?]

Before the budget shape, I wanted to pin down the re-arm trigger
itself, since the whole floor rests on it.  Traced through, the grant
is gated on a true zero-crossing: svc_xprt_enqueue() refills the
credit only when atomic_read(&xprt->xpt_nr_rqsts) == 0 at the instant
new data arrives -- i.e. every earlier request on that transport has
already passed svc_xprt_release().  Not a low-watermark; exact zero.

My worry was that a sustained interactive flow (a tree walk holding a
couple of RPCs in flight) never reaches zero, spends its 64 once, and
then rides the normal queue.  The measurements push back on that.  I
put a kprobe on the in-flight depth at each committed enqueue and ran
a cold v4.2 loopback walk of a 7500-file tree (~15.5k server RPCs):

    depth 0 : 25.6%   <- re-arms the 64-credit budget
    depth 1 : 62.3%
    depth 2 : 12.2%
    depth 3 : ~0

So zero recurs about every fourth enqueue.  ~3 credits spent between
refills against a budget of 64 -- the credit never approaches zero and
the flow stays high-priority for the whole walk.  Uncontended, the
"== 0" trigger is far from one-shot.

The catch: this can't reach the case the series exists for.  Loopback
nfsd keeps up, so depth collapses on its own.  The demotion risk is
under pool contention -- aggressor saturating the threads, the
interactive flow's replies delayed, depth staying elevated, the
zero-crossings that refill credits becoming rare.  I couldn't build
that window without your per-op service-time hook.

Could you run the same probe under your aggressor + 10ms hook?  Keyed
by transport it separates the interactive flow from the aggressor in
one histogram:

    T=/sys/kernel/tracing
    echo 'p:svcenq svc_xprt_enqueue depth=+76($arg1):s32 \
        flags=+56($arg1):x64 xprt=$arg1:x64' >> $T/dynamic_events
    echo '!(flags & 1)' > $T/events/kprobes/svcenq/filter
    echo 'hist:keys=xprt,depth:sort=xprt,depth' \
        > $T/events/kprobes/svcenq/trigger
    echo 1 > $T/events/kprobes/svcenq/enable

(+76 = offsetof(svc_xprt, xpt_nr_rqsts), +56 = xpt_flags, x86_64;
unchanged by the series since xpt_hi_credit lands after it.
depth=$arg1->xpt_nr_rqsts.counter:s32 might be a cleaner approach.
 filter !(flags & 1) keeps only committed enqueues -- XPT_BUSY
clear at entry.)

The question that settles it: for the interactive xprt under load,
does the depth==0 fraction hold up, or do you see depth>=1 runs longer
than 64?  If it holds, the trigger is fine and the uncovered tail is
just the N>64 knee you already document.  If it collapses, that's the
argument for a low-watermark refill rather than exact-zero -- and it
belongs in the cover letter, which currently asserts "idle" without
showing an interactive flow reaches it under contention.


-- 
Chuck Lever