Re: files disappearing randomly - kernel 2.6.27 cifs.ko 1.54 - VFS: No response for cmd 50

On Mon, 20 Apr 2009 17:26:55 +0200
Jodok Ole Muellers <[email protected]> wrote:

> Hello,
> 
> We recently noticed that one of our smb mounts sometimes only shows a small amount of files.
> We e.g. do a "ls -1 | wc -l" and noticed that sometimes only 148 files are shown but 
> the directory contains 2074 files. Just a few seconds later "ls -1 | wc -l" will again show 2074 files.
> We also monitored the share with a Windows client but never saw files disappearing.
> 
> Kernel is Linux vam-uti3 2.6.27-gentoo-r10 #1 SMP Tue Apr 14 14:04:02 CEST 2009 i686 Intel(R) Xeon(TM) CPU 2.40GHz GenuineIntel GNU/Linux
> 
> # modinfo /lib/modules/2.6.27-gentoo-r10/kernel/fs/cifs/cifs.ko
> filename:       /lib/modules/2.6.27-gentoo-r10/kernel/fs/cifs/cifs.ko
> version:        1.54
> description:    VFS to access servers complying with the SNIA CIFS Specification e.g. Samba and Windows
> license:        GPL
> author:         Steve French <[email protected]>
> srcversion:     C4B33C4846085F49140276C
> depends:        
> vermagic:       2.6.27-gentoo-r10 SMP mod_unload PENTIUM4 
> parm:           CIFSMaxBufSize:Network buffer size (not including header). Default: 16384 Range: 8192 to 130048 (int)
> parm:           cifs_min_rcv:Network buffers in pool. Default: 4 Range: 1 to 64 (int)
> parm:           cifs_min_small:Small network buffers in pool. Default: 30 Range: 2 to 256 (int)
> parm:           cifs_max_pending:Simultaneous requests to server. Default: 50 Range: 2 to 256 (int)
> 
> This is the share that causes problems:
> 
> # cat /proc/fs/cifs/DebugData 
> ...
> 4) Name: 10.2.13.177  Domain: ASCHENDORFF Uses: 1 OS: Windows Server 2003 R2 3790 Service Pack 2
>         NOS: Windows Server 2003 R2 5.2 Capability: 0x1f3fd
>         SMB session status: 1   TCP status: 1
>         Local Users To Server: 2 SecMode: 0x3 Req On Wire: 0
>         Shares:
>         1) \\prod-fs3\Dispatch Mounts: 1 Type: NTFS DevInfo: 0x20 Attributes: 0x700ff
> PathComponentMax: 255 Status: 0x1 type: DISK 
> 
> 
> Therefore we started to have a look into the /var/log/messages 
> Which contains a lot of these messages:
> 
> Apr 20 12:11:43 vam-uti3 CIFS VFS: Unexpected lookup error -5
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64280
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55802
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 117 mid 55801
> Apr 20 12:11:43 vam-uti3 CIFS VFS: Unexpected lookup error -11
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55803
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64286
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55808
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55809
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64294
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55816
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55817
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55825
> Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55828
> Apr 20 12:12:04 vam-uti3 CIFS VFS: server not responding
> Apr 20 12:12:04 vam-uti3 CIFS VFS: No response for cmd 114 mid 55830
> Apr 20 12:12:04 vam-uti3 CIFS VFS: Unexpected lookup error -112
> Apr 20 12:12:04 vam-uti3 CIFS VFS: No response to cmd 115 mid 55831
> Apr 20 12:12:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
> Apr 20 12:12:04 vam-uti3 CIFS VFS: Unexpected lookup error -11
> 
> I did a "echo > 4 /proc/fs/cifs/cifsFYI" at around the same timepoint when
> we were missing most of the files (saw only about 143 files out of 1901) in ls output.
> I attached the /var/log/messages 
> 
> I also have a tcpdump capture avilable but I lag to post it 
> since it contains smb passwords and probably other sensitive information.
> 
> 
> Also we sometimes can't access the mount point at all:
> vam-uti3:/data/dispatch/incoming/prod-fs3-ISDN_EINGANG # ls
> ls: cannot open directory .: Resource temporarily unavailable
> 
> Apr 20 12:24:04 vam-uti3 CIFS VFS: server not responding
> Apr 20 12:24:04 vam-uti3 CIFS VFS: server not responding
> Apr 20 12:24:04 vam-uti3 CIFS VFS: No response to cmd 115 mid 25210
> Apr 20 12:24:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
> Apr 20 12:24:04 vam-uti3 CIFS VFS: No response for cmd 114 mid 25209
> Apr 20 12:24:04 vam-uti3 CIFS VFS: Unexpected lookup error -11
> Apr 20 12:24:04 vam-uti3 CIFS VFS: Unexpected lookup error -112
> Apr 20 12:24:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
> Apr 20 12:24:34 vam-uti3 CIFS VFS: server not responding
> Apr 20 12:24:34 vam-uti3 CIFS VFS: No response to cmd 115 mid 25215
> Apr 20 12:24:34 vam-uti3 CIFS VFS: Send error in SessSetup = -11
> Apr 20 12:24:34 vam-uti3 CIFS VFS: Unexpected lookup error -11
> Apr 20 12:24:34 vam-uti3 CIFS VFS: No response for cmd 114 mid 25214
> Apr 20 12:24:34 vam-uti3 CIFS VFS: Unexpected lookup error -112
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26581
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 34408
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26594
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 34432
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 162 mid 34439
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26606
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 162 mid 34443
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26619
> Apr 20 12:24:39 vam-uti3 CIFS VFS: No response to cmd 46 mid 34459
> Apr 20 12:24:39 vam-uti3 CIFS VFS: Send error in read = -11
> 
> 
> Help and hints how to proceed/debug further are very much appreciated!
> 
> TIA Jodok Muellers
> 
> 

Short answer: try a newer kernel

Longer answer: sounds like the problem that these commits were intended
to fix:

ddb4cbfc53aa0913ee8da059fcbf628d14f40f63
a364bc0b37f14ffd66c1f982af42990a9d77fa43

Cheers,
-- 
Jeff Layton <[email protected]>