Per-entry WSL EA queries during readdir cause severe ls -la slowdown on high-latency links (6.12)

Gustavo Nobrega <[email protected]>
Newsgroups org.kernel.vger.linux-cifs
Message-ID <[email protected]>
Hi Steve, linux-cifs,

I'm reporting what looks like a missing negative cache for WSL extended-
attribute queries during directory listing. On a high-latency link this turns
a 3ms operation into 21 seconds.

Environment
-----------
Client:  Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2)
Server:  Windows Server, SMB 3.1.1
Link:    site-to-site VPN between datacenters, ~76ms RTT
Share:   FTP landing zone, 511 subdirectories in the parent

Mount options:
  vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000,
  forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix,
  reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,
  acdirmax=300,acregmax=30

Symptom
-------
  # time ls -la /mnt/share > /dev/null
  real    0m39.031s

An identical mount of the same share from a 5.10 client completes in 0.003s.

/proc/fs/cifs/Stats after a single ls -la (511 entries):

  QueryDirectories: 3 total 0 failed
  Creates:        512 total 0 failed
  Closes:         512 total 0 failed
  QueryInfos:     511 total 0 failed

Enumeration itself is 3 round trips. The client then issues a compounded
create/query/close against every single entry.

Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:

  ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
      status=0xc0000052 rc=-61

  $ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
      268 0xc0000052
        1 0xc000019c
        1 0x80000006

0xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one
listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV)
on every entry. The server has none and says so, every time, at 76ms per
round trip.

Notably the mount forces uid/gid, so any ownership data returned would be
discarded regardless.

Command distribution for the same listing:

  $ grep -oP '(?<=cmd=)\d+' trace | sort -n | uniq -c
      548 5    (CREATE)
      282 6    (CLOSE)
        6 14   (QUERY_DIRECTORY)
      553 16   (QUERY_INFO)

The CREATE/CLOSE imbalance suggests retries on top of the base cost.

Tracepoint event distribution for the same listing:

  $ grep -oP 'smb3_\w+(?=:)' trace | sort | uniq -c | sort -rn | head
      830 smb3_cmd_enter
      830 smb3_add_credits
      295 smb3_cmd_done
      292 smb3_waitff_credits
      277 smb3_enter
      276 smb3_exit_done
      270 smb3_cmd_err
        9 smb3_tcon_ref
        7 smb3_close_enter
        3 smb3_query_dir_enter

What doesn't help
-----------------
  noserverino          511 -> 271 queries, 39s -> 21s  (only thing that helped)
  symlink=none         no change
  reparse=wsl          no change
  vers=3.0             no change
  dir_cache_timeout    no change (set post-mount, so possibly not applied)

There is no mount option to disable the WSL EA probing. Checking the module's
parameter tables, wsl_query_ea/wsl_query_eas appear to be internal only.

Repeat listings within acdirmax=300 also take ~21s, which suggests the
negative result is not being cached per-inode.

Suggestion
----------
Two options that would each solve this:

1. Cache the negative EA result. If a file returns STATUS_NO_EAS_ON_FILE,
   don't re-probe it on every stat within the attribute cache window.
   Arguably worth caching per-tcon as well, since a server that has no WSL
   EAs anywhere is the common case against Windows.

2. Add a mount option to skip WSL EA queries entirely. Mounts that force
   uid/gid/file_mode discard the results anyway.

Happy to provide full tracepoint output, run patches, or test against other
dialects and servers. The reproducer is straightforward: any Windows share
with a few hundred entries over a link with meaningful latency.

Thanks,
Gustavo Nobrega
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.