Per-entry WSL EA queries during readdir cause severe ls -la slowdown on high-latency links (6.12)
Gustavo Nobrega <[email protected]>
| Newsgroups | org.kernel.vger.linux-cifs |
|---|---|
| Message-ID | <[email protected]> |
Hi Steve, linux-cifs,
I'm reporting what looks like a missing negative cache for WSL extended-
attribute queries during directory listing. On a high-latency link this turns
a 3ms operation into 21 seconds.
Environment
-----------
Client: Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2)
Server: Windows Server, SMB 3.1.1
Link: site-to-site VPN between datacenters, ~76ms RTT
Share: FTP landing zone, 511 subdirectories in the parent
Mount options:
vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000,
forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix,
reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,
acdirmax=300,acregmax=30
Symptom
-------
# time ls -la /mnt/share > /dev/null
real 0m39.031s
An identical mount of the same share from a 5.10 client completes in 0.003s.
/proc/fs/cifs/Stats after a single ls -la (511 entries):
QueryDirectories: 3 total 0 failed
Creates: 512 total 0 failed
Closes: 512 total 0 failed
QueryInfos: 511 total 0 failed
Enumeration itself is 3 round trips. The client then issues a compounded
create/query/close against every single entry.
Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:
ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
status=0xc0000052 rc=-61
$ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
268 0xc0000052
1 0xc000019c
1 0x80000006
0xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one
listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV)
on every entry. The server has none and says so, every time, at 76ms per
round trip.
Notably the mount forces uid/gid, so any ownership data returned would be
discarded regardless.
Command distribution for the same listing:
$ grep -oP '(?<=cmd=)\d+' trace | sort -n | uniq -c
548 5 (CREATE)
282 6 (CLOSE)
6 14 (QUERY_DIRECTORY)
553 16 (QUERY_INFO)
The CREATE/CLOSE imbalance suggests retries on top of the base cost.
Tracepoint event distribution for the same listing:
$ grep -oP 'smb3_\w+(?=:)' trace | sort | uniq -c | sort -rn | head
830 smb3_cmd_enter
830 smb3_add_credits
295 smb3_cmd_done
292 smb3_waitff_credits
277 smb3_enter
276 smb3_exit_done
270 smb3_cmd_err
9 smb3_tcon_ref
7 smb3_close_enter
3 smb3_query_dir_enter
What doesn't help
-----------------
noserverino 511 -> 271 queries, 39s -> 21s (only thing that helped)
symlink=none no change
reparse=wsl no change
vers=3.0 no change
dir_cache_timeout no change (set post-mount, so possibly not applied)
There is no mount option to disable the WSL EA probing. Checking the module's
parameter tables, wsl_query_ea/wsl_query_eas appear to be internal only.
Repeat listings within acdirmax=300 also take ~21s, which suggests the
negative result is not being cached per-inode.
Suggestion
----------
Two options that would each solve this:
1. Cache the negative EA result. If a file returns STATUS_NO_EAS_ON_FILE,
don't re-probe it on every stat within the attribute cache window.
Arguably worth caching per-tcon as well, since a server that has no WSL
EAs anywhere is the common case against Windows.
2. Add a mount option to skip WSL EA queries entirely. Mounts that force
uid/gid/file_mode discard the results anyway.
Happy to provide full tracepoint output, run patches, or test against other
dialects and servers. The reproducer is straightforward: any Windows share
with a few hundred entries over a link with meaningful latency.
Thanks,
Gustavo Nobrega