Re: [REGRESSION] nfsd TCP socket lockup on 14.3-RELEASE-p9/p10 - Confirmed on Multiple Systems

On Mon, Apr 20, 2026 at 10:11:21AM -0400, Olgun Adak wrote:
> Hello FreeBSD Community,
> 
> We’ve run into a consistent nfsd lockup after moving from 14.3-RELEASE-p8
> to p10. We have verified this across two identical bare-metal systems.
> Reverting to p8 via bectl immediately restores stability on both systems,
> so this appears to be a regression introduced in the p9/p10 cycle.
> 
> *The symptoms:*
> 
> Under NFSv3 load, the nfsd service hangs and becomes unresponsive to all
> clients. Looking at procstat -kk, we see a deadlock pattern where threads
> are stuck waiting on soiolock:
> 
> _sx_xlock_hard -> soiolock -> sosend_generic -> sosend -> svc_vc_reply

Would you be able to share full "procstat -kka" output from an affected
system?

> 
> Several threads are blocked in _sx_xlock_hard while others sit in sbwait.
> 
> *The environment:*
> 
> The systems are bare-metal with 2 x dual-port Mellanox ConnectX-6 100GbE
> (mlx5en) cards. We see the issue regardless of MTU (1500 and 9000).
> 
> Offloads:
> 
>    -
> 
>    TSO: Enabled
>    -
> 
>    LRO: The issue persists regardless of LRO state (tested with LRO
>    disabled and with software-only LRO). Hardware LRO is disabled in all cases.
> 
> Relevant tunables:
> 
> kern.ipc.soacceptqueue=1000
> kern.ipc.somaxconn=2000
> kern.ipc.maxsockbuf=67108864
> net.inet.tcp.sendbuf_max=67108864
> net.inet.tcp.sendspace=16777216
> net.inet.tcp.sendbuf_inc=262144
> net.inet.tcp.recvbuf_max=67108864
> net.inet.tcp.recvspace=16777216
> vfs.nfsd.srvmaxio=1048576
> 
> We have kept the p10 Boot Environments intact and can boot back into them
> to run any additional debug commands or test patches if someone can help
> point us in the right direction.
> 
> Best regards,
> -Olgun Adak