nfs server issues
Richard Purdie <[email protected]> Fri, 03 Jul 2026 17:58:52 +0100
| Newsgroups | gmane.os.freebsd.devel.file-systems |
|---|---|
| Message-ID | <8657340090e3a4ac441cc7c4e9ea92ff2673927f.camel@linuxfoundation.org> |
Hi, I was hoping someone might be able to give me some pointers on how to debug an NFS issue we keep running into. I asked on #freebsd and they suggested I should send an email. We have a FreeBSD 14.4 NFS server which we connect to with Linux nfs clients using NFS 4.2. The clients are many different Linux distros (Alma, Fedora, Ubuntu, Debian, OpenSUSE) of differing versions. We reboot the clients on Monday, by Friday, the clients start locking up showing "Remote I/O error" for something like "echo xxx > /nfs/testfile". Reads work, writes don't and it affects all of them eventually. Having things break just before or at the weekend is getting annoying! I was able to get one of the clients working again by terminating all the processes using the mount point and then remounting. We can obviously restart the nfs server but that just buys time until it happens again eventually. On the server, nfsd is using 5580% CPU and there appear to be very high open file (10-15k) and lock (100s) counts. The output from nfsdumpstate shows that: Flags OpenOwner Open LockOwner Lock Deleg OldDeleg Clientaddr ClientID CB 1 14837 208 208 0 0 fd01:172:16::1a 4c696e7578204e465376342e32207562756e7475323430342d766b2d31 CB 1 10945 136 136 0 0 fd01:172:16::242:2154 4c696e7578204e465376342e3220616c6d61382d766b2d31 CB 1 11006 157 157 0 0 fd01:172:16::14 4c696e7578204e465376342e322064656269616e31322d766b2d35 CB 1 10330 127 127 0 0 2a01:4f9:3070:2b44::2 4c696e7578204e465376342e32207562756e7475323230342d766b2d32 CB 1 14515 205 205 0 0 fd01:172:16::242:2162 4c696e7578204e465376342e322064656269616e31312d766b2d33 CB 1 13184 261 261 0 0 fd01:172:16::242:2152 4c696e7578204e465376342e32207562756e7475323531302d766b2d31 CB 1 14713 264 264 0 0 fd01:172:16:1::11 4c696e7578204e465376342e322064656269616e31322d766b2d32 CB 1 8275 123 123 0 0 fd01:172:16::18 4c696e7578204e465376342e3220616c6d61382d766b2d32 CB 1 14778 165 165 0 0 fd01:172:16::242:2158 4c696e7578204e465376342e322064656269616e31312d766b2d31 CB 1 2525 16 16 0 0 fd01:172:16::16 4c696e7578204e465376342e32207562756e7475323230342d766b2d61726d32 CB 1 13050 163 163 0 0 fd01:172:16::2:18 4c696e7578204e465376342e322064656269616e31322d766b2d38 CB 1 11290 155 155 0 0 fd01:172:16::242:2157 4c696e7578204e465376342e322064656269616e31322d766b2d31 CB 0 0 0 0 0 0 2a01:4f9:3081:38ea::2 4c696e7578204e465376342e3220706572662d64656269616e31322d766b CB 1 15116 191 191 0 0 fd01:172:16::1:26 4c696e7578204e465376342e32207562756e7475323430342d766b2d33 CB 1 9935 181 181 0 0 fd01:172:16::242:2150 4c696e7578204e465376342e32206665646f726134332d766b2d32 CB 1 14844 183 183 0 0 fd01:172:16::12 4c696e7578204e465376342e322064656269616e31322d766b2d33 CB 1 22771 356 356 0 0 fd01:172:16::37 4c696e7578204e465376342e32206f70656e737573653135362d766b2d31 CB 1 1 0 0 0 0 2a01:4f9:3b:4ec5::2 4c696e7578204e465376342e3220706572662d616c6d61382d766b CB 1 11093 163 163 0 0 fd01:172:16::242:2160 4c696e7578204e465376342e3220616c6d61392d766b2d32 CB 1 15780 179 179 0 0 fd01:172:16::1:25 4c696e7578204e465376342e32207562756e7475323430342d766b2d32 CB 1 16558 333 333 0 0 fd01:172:16::13 4c696e7578204e465376342e322064656269616e31322d766b2d34 CB 1 6297 93 93 0 0 2a01:4f9:3090:14cc::2 4c696e7578204e465376342e32206665646f726134342d766b2d31 CB 1 13180 188 188 0 0 fd01:172:16::242:2155 4c696e7578204e465376342e3220726f636b79382d766b2d31 CB 1 11373 196 196 0 0 fd01:172:16::19 4c696e7578204e465376342e3220726f636b79392d766b2d32 CB 1 14881 195 195 0 0 fd01:172:16::2:19 4c696e7578204e465376342e322064656269616e31322d766b2d39 CB 1 4068 63 63 0 0 fd01:172:16::38 4c696e7578204e465376342e32207562756e7475323630342d766b2d61726d31 CB 1 11171 157 157 0 0 fd01:172:16::1:28 4c696e7578204e465376342e32207562756e7475323230342d766b2d34 CB 1 14374 226 226 0 0 fd01:172:16::15 4c696e7578204e465376342e322064656269616e31322d766b2d36 CB 1 14584 290 290 0 0 fd01:172:16::1:27 4c696e7578204e465376342e32207562756e7475323230342d766b2d33 CB 1 19905 516 516 0 0 fd01:172:16::242:2143 4c696e7578204e465376342e32206f70656e737573653136302d766b2d312e796f63746f2e696f CB 1 0 0 0 0 0 2a01:4f9:3071:1625::2 4c696e7578204e465376342e32207562756e7475323630342d766b2d31 CB 1 13805 214 214 0 0 fd01:172:16::242:2153 4c696e7578204e465376342e32207562756e7475323530342d766b2d31 CB 1 11157 125 125 0 0 fd01:172:16::242:2165 4c696e7578204e465376342e32207562756e7475323230342d766b2d31 CB 1 6668 103 103 0 0 fd01:172:16:1::33 4c696e7578204e465376342e322064656269616e31332d766b2d61726d31 CB 1 8055 105 105 0 0 2a01:4f9:3071:1624::2 4c696e7578204e465376342e32207562756e7475323630342d766b2d32 CB 1 2669 23 23 0 0 fd01:172:16::36 4c696e7578204e465376342e32207562756e7475323430342d766b2d61726d32 CB 1 10060 554 554 0 0 fd01:172:16:1::29 4c696e7578204e465376342e322064656269616e31332d766b2d31 CB 1 11182 532 532 0 0 fd01:172:16:1::242:2159 4c696e7578204e465376342e322064656269616e31332d766b2d32 CB 1 11565 195 195 0 0 2a01:4f9:3081:33de::2 4c696e7578204e465376342e322073747265616d392d766b31 CB 1 16998 338 338 0 0 fd01:172:16::242:2161 4c696e7578204e465376342e32206f70656e737573653136302d766b2d32 CB 1 3597 36 36 0 0 fd01:172:16::35 4c696e7578204e465376342e32207562756e7475323430342d766b2d61726d31 CB 1 7276 188 188 0 0 2a01:4f9:3051:510f::2 4c696e7578204e465376342e322073747265616d31302d766b2d61726d31 CB 1 15294 210 210 0 0 fd01:172:16::2:17 4c696e7578204e465376342e322064656269616e31322d766b2d37 CB 1 9599 192 192 0 0 fd01:172:16::222:2160 4c696e7578204e465376342e3220616c6d6131302d766b2d31 The open file and lock counts are odd as there shouldn't be any! I also noticed some duplicate ClientIDs which I wasn't sure was an issue or not. One of the zero counts lines is the client I remounted on. I was hoping I could find out which files were open/locked but I couldn't work out if there is a way to do that. Any pointers on how to try and debug the issue further, or what the issue might be would be welcome. We'll probably have to just give in and restart the server again but this will happen again in a week or two based on past experience so I'd love to have a plan ready for when it does happen again. Cheers, Richard (trying to help fix Yocto Project infrastructure issues)