Re: [PATCH RFC 0/4] nfsd: per-client fair-queue dispatch

On Thu, 2026-06-04 at 08:52 +1000, NeilBrown wrote:
> On Thu, 04 Jun 2026, Benjamin Coddington wrote:
> > knfsd dispatches ready transports from a single per-pool FIFO, so a
> > client's share of nfsd service scales with the number of connections it
> > holds rather than being shared per client.  A client that opens many
> > connections (nconnect, or a farm of data movers) starves other clients
> > on the same server purely by out-numbering them in sockets.
> > 
> > I measured this with a load generator that pins each request to a fixed
> > service time and does no filesystem work, so that nfsd thread-time is the
> > only scarce resource (8 threads, 10ms/op, ~648 ops/s pool ceiling).  A
> > greedy client opens K connections alongside one single-connection
> > interactive client.
> > 
> > NFSv3, dispatch as it is today:
> > 
> >   greedy K   greedy share   interactive ops/s
> >      1           50%              241
> >      4           80%              129
> >      8           89%               72
> >     16           94%               38
> > 
> > The interactive client's share tracks 1/(K+1) and its throughput falls
> > roughly 6x while it does nothing different.  NFSv4.1 behaves identically
> > (89% greedy at K=8) even when the greedy connections are bound to a
> > single session, because the dispatch decision is below the NFS version.
> > 
> > The same NFSv4.1 workload with fair queueing enabled:
> > 
> >   greedy K   greedy share   interactive ops/s
> >      8           72%              182
> >     16           73%              177
> >     32           70%              193
> > 
> > The greedy client's share no longer climbs with its connection count and
> > the interactive client recovers (72 -> 182 ops/s at K=8).  Aggregate
> > throughput is unchanged: the T/D pool ceiling is the same with fair
> > queueing on and off.  The split does not reach 50/50 because a single
> > interactive connection is bounded by its request window and by XPT_BUSY
> > serialising one transport; with a deeper window it reaches ~59/41.
> > 
> > The approach:
> > 
> >   - sunrpc grows an opaque per-transport fairness key (patch 1), with a
> >     default derived from the source address (the source port is excluded
> >     so a client's several connections share one key), and an opt-in
> >     per-pool scheduler that buckets ready transports by that key and
> >     dispatches round-robin across keys (patch 2).  When it is disabled,
> >     which is the default, the existing lockless FIFO path is unchanged.
> > 
> >   - nfsd gains a "fairq" module parameter to turn it on (patch 3) and
> >     stamps the NFSv4.1 clientid as the key when a connection binds to a
> >     session (patch 4), so all of a client's connections share one key.
> >     NFSv3 uses the source-address default.
> > 
> > This is an RFC; a few questions for the list:
> > 
> >   - Unit of fairness: clientid (used here) or session?  Earlier
> >     discussion leaned toward exploring per-session.
> > 
> >   - Mechanism: a fixed bucket hash under a per-pool spinlock taken only
> >     on the opt-in path, versus a lockless or per-flow structure.
> 
> I'm not keen on the hash bucket approach, though I can't clearly say
> why.
> I'm also not keen on the opt-in design.  I don't like asking the admin to
> tune performance.  We should always provide optimal performance.
> 
> I imagine creating an object which represents a client - using IP
> address or possibly v4 client id.  These may well be located using a
> hash table (rhashtable?), but that is peripheral to the design.
> Each xprt has an associated client which can be changed at any time
> (nfsd could change to a v4.1-client).
> 
> Each client has a lwq of xprts and is part of an lwq of clients.
> 
> To enqueue an xprt, we grab the client, enqueue to that, then if
> necessary enqueue the client.
> 
> To dequeue next, we dequeue the first client, dequeue the first xprt,
> then optionally enqueue the client again.
> 
> So this would be slightly more work than the current (2 dequeues instead
> of 1) but I think that might be acceptable.
> 
> clients would be refcounted by xprts and probably rcu-freed.
> 
> > 
> >   - Would a per-client in-flight cap be preferable to proportional fair
> >     queueing?
> 
> A per-client cap would only be ok if the admin didn't have to tune it.
> So the client would need to get some sort of feed-back from the server
> so that it knows when it is pushing too hard.  If we were still using
> UDP we could possibly use packet-loss for that feed-back, but we aren't
> and don't want to.
> With v4.1 we can of course use the slot based flow control and I think
> we should if we can agree on a good design.  With v3 I don't think there
> is any way to get the needed feed-back
> 

Shrink the TCP window to 0 for a bit on the connection(s)? That's the
typical way to slow down a spammy client on a TCP socket. It's a bit
coarse grained but typically the best we can do with v3. IDK, could we
do something with the ECN flag?
-- 
Jeff Layton <[email protected]>