Re: Per-entry WSL EA queries during readdir cause severe ls -la slowdown on high-latency links (6.12)

Gustavo Nobrega <[email protected]> writes:

> I'm reporting what looks like a missing negative cache for WSL extended-
> attribute queries during directory listing. On a high-latency link this turns
> a 3ms operation into 21 seconds.

Yes - we currently don't cache xattrs.

> Environment
> -----------
> Client:  Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2)
> Server:  Windows Server, SMB 3.1.1
> Link:    site-to-site VPN between datacenters, ~76ms RTT
> Share:   FTP landing zone, 511 subdirectories in the parent
>
> Mount options:
>   vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000,
>   forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix,
>   reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,
>   acdirmax=300,acregmax=30
>
> Symptom
> -------
>   # time ls -la /mnt/share > /dev/null
>   real    0m39.031s
>
> An identical mount of the same share from a 5.10 client completes in 0.003s.
>
> /proc/fs/cifs/Stats after a single ls -la (511 entries):
>
>   QueryDirectories: 3 total 0 failed
>   Creates:        512 total 0 failed
>   Closes:         512 total 0 failed
>   QueryInfos:     511 total 0 failed
>
> Enumeration itself is 3 round trips. The client then issues a compounded
> create/query/close against every single entry.
>
> Root cause
> ----------
> smb3 tracepoints show what those per-entry queries actually are:
>
>   ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
>       status=0xc0000052 rc=-61
>
>   $ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
>       268 0xc0000052
>         1 0xc000019c
>         1 0x80000006
>
> 0xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one
> listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV)
> on every entry. The server has none and says so, every time, at 76ms per
> round trip.

It's not that the client thinks the files have WSL EAs.  For some reason
the query directory response doesn't seem to satisfy the client in terms
of metadata in order to instantiate those files during directory
listing.  That is why it is sending query info for each file to get more
metadata information.  The WSL EAs are queried unconditionally when
querying information about a file.  This is done for saving any
potentially additional roundtrips in case they have.

The extra query info usually when the client finds a file with a reparse
attribute set, 'cifsacl' or 'modefromsid' mount option being used,
'mfsymlinks' mount option is set and a regular file could be a potential
symlink, etc.  If that isn't the case, then the bug is somewhere else.

I'll look at the network traces sent off-list and then let you know.

Thanks for the detailed report!