Re: [RFC] knfsd: per-client fair scheduling to prevent single-client starvation

On Wed, 03 Jun 2026, Chuck Lever wrote:
> 
> On Tue, Jun 2, 2026, at 3:53 PM, NeilBrown wrote:
> > On Wed, 20 May 2026, Chuck Lever wrote:
> >> 
> >> On Tue, May 19, 2026, at 6:02 PM, Benjamin Coddington wrote:
> >> > On 19 May 2026, at 14:44, Chuck Lever wrote:
> >> >
> >> >> On Tue, May 19, 2026, at 5:08 PM, Benjamin Coddington wrote:
> >> >>> Just to be clear - the issue I'm exploring isn't the same as when all the
> >> >>> kNFSD threads are slow due to their workload.  This is very much a
> >> >>> multi-client dynamic where one client (or a group of automated client
> >> >>> instances) are able to easily starve another simply because they create the
> >> >>> most connections.
> >> >>>
> >> >>> That's different from the other problem that we've discussed a bunch at
> >> >>> bakeathon and on the list previously.
> >> >>>
> >> >>> This is not so much a deadlock issue as it is an issue
> >> >>> of per-client fairness.  I think this problem is in a different class.
> >> >>
> >> >> Does dynamic svc thread creation have any impact?
> >> >
> >> > I haven't tested it - I think it would just pin to max-threads for the
> >> > workload in question.
> >> 
> >> If the aggregate workload consumes all the threads, then that doesn’t
> >> sound like xprt scheduling is the bottleneck. But I should look at
> >> numbers instead of speculating.
> >> 
> >> Are you seeing connection loss in these scenarios?
> >> 
> >> 
> >> > I'm probably not understanding you here, because for the problem I'm
> >> > interested in fair would look like prioritizing each client's request
> >> > queue equally, no matter how many xprts each client has.
> >> 
> >> Then for NFSv4.1 and later, NFSD might schedule work on the session, and
> >> manage each session’s workload by raising and lowering the number of slots
> >> in its slot table.
> >
> > I agree that managing slot numbers is likely to be a good approach.
> > It doesn't make much sense to allocate to any client more slots than the
> > maximum number of threads - does it?
> >
> > We already have code to reduce slots numbers based on memory pressure.
> > We could extend that to reduce based on demand compared to number of
> > threads.
> >
> > e.g. let every client have a least one unused slot until the total slots
> > across all clients reaches the maximum number of threads.  Then apply
> > pressure evenly like we do for memory shortage.
> 
> Sensible!
> 
> But these days the number of threads isn’t fixed. As load goes up the
> number of threads increases.

Yes, but there is still an upper limit.  That is what I was thinking of.

But in practice the upper limit might be less than the configured upper
limit if allocating a new server fails (hmm...  I wonder if it can
fail...  it should probably use GFP_RETRY_MAYFAIL or maybe even
GFP_NORETRY when adding extra threads).
To be able to compare the number of slots to the number of threads we
would need to track an effective-maximum which is probably the largest
maximum seen.

Hmmm..  should we add a shrinker to stop nfsd threads when memory is
tight?

Thanks,
NeilBrown


> 
> 
> > Idle clients will get pushed back to 1 slot, active client will tend
> > towards a "fair" share based on how comparatively busy they are.
> >
> > This wouldn't help for v3 of course but I don't think we need these
> > advanced features for v3.
> 
> Ben’s employer might disagree with that :-)
> 
> I think NFSD might also need to consider LOCALIO workloads.
> 
> -- 
> Chuck Lever
>