Re: [PATCH RFC 0/4] nfsd: per-client fair-queue dispatch

On Wed, 2026-06-03 at 11:09 -0400, Benjamin Coddington wrote:
> knfsd dispatches ready transports from a single per-pool FIFO, so a
> client's share of nfsd service scales with the number of connections it
> holds rather than being shared per client.  A client that opens many
> connections (nconnect, or a farm of data movers) starves other clients
> on the same server purely by out-numbering them in sockets.
> 
> I measured this with a load generator that pins each request to a fixed
> service time and does no filesystem work, so that nfsd thread-time is the
> only scarce resource (8 threads, 10ms/op, ~648 ops/s pool ceiling).  A
> greedy client opens K connections alongside one single-connection
> interactive client.
> 
> NFSv3, dispatch as it is today:
> 
>   greedy K   greedy share   interactive ops/s
>      1           50%              241
>      4           80%              129
>      8           89%               72
>     16           94%               38
> 
> The interactive client's share tracks 1/(K+1) and its throughput falls
> roughly 6x while it does nothing different.  NFSv4.1 behaves identically
> (89% greedy at K=8) even when the greedy connections are bound to a
> single session, because the dispatch decision is below the NFS version.
> 
> The same NFSv4.1 workload with fair queueing enabled:
> 
>   greedy K   greedy share   interactive ops/s
>      8           72%              182
>     16           73%              177
>     32           70%              193
> 

These seem like pretty clear improvements.

> The greedy client's share no longer climbs with its connection count and
> the interactive client recovers (72 -> 182 ops/s at K=8).  Aggregate
> throughput is unchanged: the T/D pool ceiling is the same with fair
> queueing on and off.  The split does not reach 50/50 because a single
> interactive connection is bounded by its request window and by XPT_BUSY
> serialising one transport; with a deeper window it reaches ~59/41.
> 
> The approach:
> 
>   - sunrpc grows an opaque per-transport fairness key (patch 1), with a
>     default derived from the source address (the source port is excluded
>     so a client's several connections share one key), and an opt-in
>     per-pool scheduler that buckets ready transports by that key and
>     dispatches round-robin across keys (patch 2).  When it is disabled,
>     which is the default, the existing lockless FIFO path is unchanged.
> 

That makes sense: you can make the IP address the key by default, and
then nfsd can override that later with the sessionid or whatever.

>   - nfsd gains a "fairq" module parameter to turn it on (patch 3) and
>     stamps the NFSv4.1 clientid as the key when a connection binds to a
>     session (patch 4), so all of a client's connections share one key.
>     NFSv3 uses the source-address default.
> 

Like Chuck, I greatly prefer no tunables, but while experimenting with
this, it may be helpful to add some under debugfs. The goal at the end
should be "no tunables" though.

> This is an RFC; a few questions for the list:
> 
>   - Unit of fairness: clientid (used here) or session?  Earlier
>     discussion leaned toward exploring per-session.
> 

I don't think we know. Per-session sounds like the right thing, but I'd
leave room for experimentation here. For v4.0, you might want to do
this by clientid. For v3, I think using the IP address without port
like you have is probably as good as can be done.

>   - Mechanism: a fixed bucket hash under a per-pool spinlock taken only
>     on the opt-in path, versus a lockless or per-flow structure.
> 

I don't have a real preference here. Whatever performs best, most
likely. This will be in a hot path.

>   - Would a per-client in-flight cap be preferable to proportional fair
>     queueing?
> 

I think probably not, but this too might require some experimentation.
-- 
Jeff Layton <[email protected]>