Re: [PATCH RFC 0/4] nfsd: per-client fair-queue dispatch

On Thu, Jun 4, 2026, at 5:11 AM, Daire Byrne wrote:
> I'm quite interested in this functionality - like I said we already
> set svc_rpc_per_connection_limit as a way to limit the damage greedy
> clients can do to wider service delivery (especially interactive
> desktops).
>
> Another thing we do is use the fq qdisc to limit the outbound
> bandwidth (maxrate) per stream (client/nconnect).
>
> And finally, we use some qdisc prio stuff to try and "prefer" things
> like workstations over the batch render farm (differentiated by
> subnet). We also mostly use NFSv3 (for a few different reasons).
>
> I guess anything that can give us better control over these kinds of
> "QOS" scenarios we'll happily take.
>
> So for example preferring a subnet of workstations over a subnet of
> batch render farm would be nice.
>
> Daire
>
>
> On Wed, 3 Jun 2026 at 23:52, NeilBrown <[email protected]> wrote:
>>
>> On Thu, 04 Jun 2026, Benjamin Coddington wrote:
>> > knfsd dispatches ready transports from a single per-pool FIFO, so a
>> > client's share of nfsd service scales with the number of connections it
>> > holds rather than being shared per client.  A client that opens many
>> > connections (nconnect, or a farm of data movers) starves other clients
>> > on the same server purely by out-numbering them in sockets.
>> >
>> > I measured this with a load generator that pins each request to a fixed
>> > service time and does no filesystem work, so that nfsd thread-time is the
>> > only scarce resource (8 threads, 10ms/op, ~648 ops/s pool ceiling).  A
>> > greedy client opens K connections alongside one single-connection
>> > interactive client.
>> >
>> > NFSv3, dispatch as it is today:
>> >
>> >   greedy K   greedy share   interactive ops/s
>> >      1           50%              241
>> >      4           80%              129
>> >      8           89%               72
>> >     16           94%               38
>> >
>> > The interactive client's share tracks 1/(K+1) and its throughput falls
>> > roughly 6x while it does nothing different.  NFSv4.1 behaves identically
>> > (89% greedy at K=8) even when the greedy connections are bound to a
>> > single session, because the dispatch decision is below the NFS version.
>> >
>> > The same NFSv4.1 workload with fair queueing enabled:
>> >
>> >   greedy K   greedy share   interactive ops/s
>> >      8           72%              182
>> >     16           73%              177
>> >     32           70%              193
>> >
>> > The greedy client's share no longer climbs with its connection count and
>> > the interactive client recovers (72 -> 182 ops/s at K=8).  Aggregate
>> > throughput is unchanged: the T/D pool ceiling is the same with fair
>> > queueing on and off.  The split does not reach 50/50 because a single
>> > interactive connection is bounded by its request window and by XPT_BUSY
>> > serialising one transport; with a deeper window it reaches ~59/41.
>> >
>> > The approach:
>> >
>> >   - sunrpc grows an opaque per-transport fairness key (patch 1), with a
>> >     default derived from the source address (the source port is excluded
>> >     so a client's several connections share one key), and an opt-in
>> >     per-pool scheduler that buckets ready transports by that key and
>> >     dispatches round-robin across keys (patch 2).  When it is disabled,
>> >     which is the default, the existing lockless FIFO path is unchanged.
>> >
>> >   - nfsd gains a "fairq" module parameter to turn it on (patch 3) and
>> >     stamps the NFSv4.1 clientid as the key when a connection binds to a
>> >     session (patch 4), so all of a client's connections share one key.
>> >     NFSv3 uses the source-address default.
>> >
>> > This is an RFC; a few questions for the list:
>> >
>> >   - Unit of fairness: clientid (used here) or session?  Earlier
>> >     discussion leaned toward exploring per-session.
>> >
>> >   - Mechanism: a fixed bucket hash under a per-pool spinlock taken only
>> >     on the opt-in path, versus a lockless or per-flow structure.
>>
>> I'm not keen on the hash bucket approach, though I can't clearly say
>> why.
>> I'm also not keen on the opt-in design.  I don't like asking the admin to
>> tune performance.  We should always provide optimal performance.
>>
>> I imagine creating an object which represents a client - using IP
>> address or possibly v4 client id.  These may well be located using a
>> hash table (rhashtable?), but that is peripheral to the design.
>> Each xprt has an associated client which can be changed at any time
>> (nfsd could change to a v4.1-client).
>>
>> Each client has a lwq of xprts and is part of an lwq of clients.
>>
>> To enqueue an xprt, we grab the client, enqueue to that, then if
>> necessary enqueue the client.
>>
>> To dequeue next, we dequeue the first client, dequeue the first xprt,
>> then optionally enqueue the client again.
>>
>> So this would be slightly more work than the current (2 dequeues instead
>> of 1) but I think that might be acceptable.
>>
>> clients would be refcounted by xprts and probably rcu-freed.
>>
>> >
>> >   - Would a per-client in-flight cap be preferable to proportional fair
>> >     queueing?
>>
>> A per-client cap would only be ok if the admin didn't have to tune it.
>> So the client would need to get some sort of feed-back from the server
>> so that it knows when it is pushing too hard.  If we were still using
>> UDP we could possibly use packet-loss for that feed-back, but we aren't
>> and don't want to.
>> With v4.1 we can of course use the slot based flow control and I think
>> we should if we can agree on a good design.  With v3 I don't think there
>> is any way to get the needed feed-back
>>
>> Thanks,
>> NeilBrown
>>
>>
>> >
>> > The measurement used a debug-only filehandle-latency hook that is not
>> > part of this series.

[ Daire, we're generally a "bottom post" list. My comments below
are in response to your notes quoted at the top as well as Neil's
and Ben's quoted inline ]

First some general responses to the shape of Ben's patches.

- Thanks for giving us something to start kicking around!

- I'm not a fan of adding administrative controls, and these days, a
  module parameter is outdated and way too global. I'd like to see
  that removed until we have a clear need (use cases) for tuning

- Having to rely on client identity is going to be difficult to get
  right. The scope of the identity, for example, is never going to
  cover all the use cases that we want. For example, fairness-by-
  client-address means all clients behind a NAT get reduced to a
  single fairness unit.

- The observability you proposed for measuring your solution is
  worth some attention. We should consider making it a part of
  the patch series rather than something that was useful only
  while it was being developed.

- Probably the largest challenge will be including NFSv3, NFSv4.0,
  and LOCALIO, none of which have the concept of a "session".

After some thought, IMO the problem as initially stated has some
areas that are going to be challenging and delay a real solution.

Instead, we could look at the problem as "preventing starvation of
any one connection" rather than the more difficult goal of "ensuring
complete scheduling fairness". This changes the problem class
fundamentally.

Strictly speaking, nothing in Ben's data is starved today: the FIFO
transport queue guarantees every backlogged transport one receive
per cycle, and every op gets completed. What degrades is the floor:
the latency and throughput a lightly loaded connection can count on.
It degrades linearly with the aggressor's connection count: with K+1
backlogged transports and pool service rate T/D, the interactive
connection waits ~(K+1) * D/T between services, which is unbounded
as K grows.

The reframed goal is precise: make the service floor of any one
connection independent of how many connections and slots everyone
else holds. That is a much weaker contract than proportional
fairness, but the best part is it dissolves the hardest challenges.

Equal shares require knowing who "everyone" is; a floor does not.
No client identity, no clientid-vs-session-vs-address debate, no
NAT collapse, no client-object lifetime, no per-pool share
fragmentation. And IMO it is what both Ben and Daire actually
asked for: Ben wrote "I would prefer the 'interactive' clients
be prioritized, but I don't control their nconnect." And Daire
described something similar above.

Looking at prior art, networking has solved exactly this reframed
problem (and Daire alludes to that above): fq_codel's success
comes less from its fair queueing than from its sparse-flow
optimization; that is, flows with nothing in flight jump the queue
and heavy flows share what remains.

TLDR; this reframe turns "schedule fairly" into "let request-
response traffic overtake streams," and NFSD already appears to
track the bit of state needed to tell them apart.


-- 
Chuck Lever