Re: [RFC] knfsd: per-client fair scheduling to prevent single-client starvation

On 19 May 2026, at 20:29, Chuck Lever wrote:

> On Tue, May 19, 2026, at 6:02 PM, Benjamin Coddington wrote:
>> On 19 May 2026, at 14:44, Chuck Lever wrote:
>>
>>> On Tue, May 19, 2026, at 5:08 PM, Benjamin Coddington wrote:
>>>> Just to be clear - the issue I'm exploring isn't the same as when all the
>>>> kNFSD threads are slow due to their workload.  This is very much a
>>>> multi-client dynamic where one client (or a group of automated client
>>>> instances) are able to easily starve another simply because they create the
>>>> most connections.
>>>>
>>>> That's different from the other problem that we've discussed a bunch at
>>>> bakeathon and on the list previously.
>>>>
>>>> This is not so much a deadlock issue as it is an issue
>>>> of per-client fairness.  I think this problem is in a different class.
>>>
>>> Does dynamic svc thread creation have any impact?
>>
>> I haven't tested it - I think it would just pin to max-threads for the
>> workload in question.
>
> If the aggregate workload consumes all the threads, then that doesn’t
> sound like xprt scheduling is the bottleneck. But I should look at
> numbers instead of speculating.

Chuck,

Here are the numbers I promised. Short version: under saturation a client's
share of nfsd service tracks its transport count, not its identity -- a
one-connection client gets ~1/(K+1) of the pool when another client drives K
connections.

Method

To measure the dispatch path in isolation I used a small debug patch to nfsd:
a request bearing a "magic" filehandle is answered after a fixed,
caller-specified delay, at the top of fh_verify() before any export/dentry
work -- so the thread is held for a known time D and does zero filesystem
work. With T nfsd threads that fixes the pool ceiling at ~T/D ops/s and makes
nfsd thread-time the only contended resource: no I/O, no caching, no backing
filesystem, just inbound request scheduling.

A userspace load client (single host, loopback) then runs two clients at
once, both offering more than the server can serve:
  - greedy:      K connections
  - interactive: 1 connection
The v3 probe is GETATTR; the v4.1 probe is COMPOUND{SEQUENCE, PUTFH}, with
the greedy client's K connections bound to one session via
BIND_CONN_TO_SESSION. The magic fh carries D in both cases.

Findings

1. Share tracks connection count. NFSv3, 8 nfsd threads, D=10ms, greedy
   connections K swept, interactive fixed at 1:

     K    interactive ops/s   intr share   greedy share
     1         240.6            50.4%         49.6%
     2         216.0            33.8%         66.2%
     4         129.4            20.1%         79.9%
     8          72.2            11.2%         88.8%
    16          38.0             5.9%         94.1%

   interactive is ~1/(K+1), greedy ~K/(K+1). Total stays pinned at the ~648
   ops/s pool ceiling throughout, and every op completes in bounded time --
   nothing is stuck.

2. The interactive client is actively harmed, not just out-numbered. Alone it
   sustains 229 ops/s; with the K=16 greedy client present it gets 38 (6x
   less) and its median latency rises from 16ms to 105ms.

3. Same result for v4.1. With 8 greedy connections bound to a single session,
   the split is 89% / 11%, identical to v3. The session is a clear client
   identity, but the dispatch path ignores it -- the unfairness is in sunrpc,
   below the version.

4. It is structural, not a small-pool artifact. Sweeping the thread count,
   the ratio is unchanged while only the absolute ceiling moves:

     K    share@T=4   share@T=8   share@T=16
     1      50.0%       50.7%       50.5%
     4      20.1%       20.0%       20.2%
    16       5.9%        5.9%        5.9%

   (K=16 total throughput: 321 / 649 / 1299 ops/s at T = 4 / 8 / 16.)

On your earlier point -- "if the aggregate workload consumes all the threads,
that doesn't sound like xprt scheduling is the bottleneck": the pool is
saturated here, that's the precondition. The finding is that the saturated
capacity is divided per-transport, so each client's share is set by how many
sockets it opened. Saturation is the setup; the per-xprt split is the problem.

I kept this to the pure scheduling case -- fixed delay, no filesystem work --
on purpose, because it's the most conservative: a real workload with heavier
ops only saturates the pool sooner and hits the same split at lower connection
counts. pool_stats agrees, with threads-woken collapsing once the pool
saturates.

The harness (synthetic clients plus the debug latency patch) is standalone;
happy to share it or run other configurations -- different fairness units,
per-connection caps, and so on.

Ben