Re: NFS delegations behavior analysis

On Tue, Jun 23, 2026, at 7:04 AM, Mkrtchyan, Tigran wrote:
> ----- Original Message -----
>> From: "Jeff Layton" <[email protected]>
>> To: "Piyush Sachdeva" <[email protected]>, "linux-nfs" <[email protected]>, "Chuck Lever" <[email protected]>,
>> "trondmy" <[email protected]>, [email protected], [email protected]
>> Cc: [email protected]
>> Sent: Tuesday, 23 June, 2026 12:50:16
>> Subject: Re: NFS delegations behavior analysis
>
>> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>>> Hi,
>>> Lately I have been running micro benchmarks around the `ls` command and
>>> reading through the code documentation of the NFS client to better
>>> understand the client side caching behavior with and without
>>> delegations.
>>> 
>>> Understanding so far:
>>> Delegations (both file and directory) are granted by the server to the
>>> client, indefinitely (until revoked or under the watermark) to cache
>>> attributes. The caching of data is a result of the attribute
>>> cache. Hence forth, a directory delegation will cache the directory
>>> attributes and the names of the files in the directory, and a file
>>> delegation will cache the attributes of the file and the file data.
>>> 
>>> Workload run:
>>> I focused on the 2 workloads below, doing 2 passes of a large flat
>>> directory (with close to 100K files) -
>>> a cold pass, and warm pass using the cache from the cold pass:
>>> - lslr - ls -lR on both runs
>>> - lsmix - ls -R (cold) and then ls -lR (warm)
>>> 
>>> I also played with the rdirplus behavior using both the default
>>> heuristic behavior and the `rdirplus=force` set at mount time.
>>> 
>>> Numbers:
>>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>>> ==================================================================
>>> 
>>>                  |         LSLR          |         LSMIX
>>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>>> -----------------+-------------+-----------+-------------+---------
>>> READDIR calls    |    27       |     0     |   27        |    0
>>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>>    call type     | readdirplus |    --     | readdirplus |    --
>>> LOOKUP           |     1       |     0     |    1        |    0
>>> GETATTR          |     3       |  100,000  |    2        | 100,001
>>> ACCESS           |     2       |     0     |    2        |    0
>>> -----------------+-------------+-----------+-------------+---------
>>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>>> 
>>> 
>>> Observations:
>>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>>> directory, the client gets a directory delegation - caching the
>>> directory attributes and file names. However, as we don't have file
>>> delegations due to no open(2) calls to any of the files. Henceforth,
>>> the cache of file attributes is governed by `actimeo`.
>>> Now here is the interesting bit, if the next `ls -l` is issued after
>>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>>> calls for every file in the directory. As a result, the performance of
>>> this warm `ls -l` run ends up being worse than the cold pass. I am
>>> guessing this is most likely due to the compounded "rdirplus" being more
>>> efficient than stat() calls.
>>> 
>>> 
>>> Proposal:
>>> For large directories, this ends up being a massive problem, taking 1-2
>>> minutes when enumerating a directory on the warm passes.
>>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>>   instead of issuing the stat(2) storm to the server: When the client
>>>   notices that there are cache misses, which would be the case of file
>>>   attributes, instead of fetching file names from the directory-delegation
>>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>>   the server, nonetheless.
>>> - A more tedious would be the to cache file attributes as well, as a part
>>>   of the directory delegation. This would end up requiring a change in the
>>>   NFS protocol spec though.
>>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>>   what if, the client could do 1 GETATTR call for getting attributes
>>>   for multiple files.
>> 
>> 
>> ls is such a hard workload to get right, because we don't really get an
>
> 100% agree. And there were a couple of attempts to address this issue
> (second ls that is slow).
>
>> indication in the kernel of what userland's intentions are. It's
>> basically a readdir() call followed by a bunch of stat()'s, but at the
>> point where we're getting the readdir() call, we don't know if userland
>> intends to stat() those files or not. We have to make a guess about
>> that intention.
>> 
>> In this case, it sounds like the directory cache was valid, so the
>> client decided it didn't need to do a READDIR at all, but the
>> individual files had caches that timed out.
>> 
>> So imagine you're the kernel client and have been given that second
>> readdir() call: Why should you decide to do a READDIRPLUS at that point
>> instead of a regular READDIR?
>
> May we need some kind of client-side heuristics, like on the server side
> for open-delegations, where after seeing some `stats` for files in the
> In the same directory, the client will decide to switch to READDIR (v4)
> to get all attributes in one go.

We do something like that already. I don't think we'll ever have a readdir
plus heuristic that makes everybody happy without userspace somehow telling
us their intentions. I know a statx-based readdir plus system call probably
sounds crazy, but something like that would go a long way to take all the
guesswork out of things on our end.

Anna

>
> Best regards,
>    Tigran.
>
>> --
>> Jeff Layton <[email protected]>
>
> Attachments:
> * smime.p7s