Re: block size problem ?

Peter T. Breuer a écrit :
> "Also sprach [email protected]:"
> 
>>There is just a little problem remaining : enbd block size is set at 4096 both
>>size. Disk is formatted in XFS, with blocksize 4096 and I have this : Is it
>>harmful ?
> 
> 
> xfs blocksize is always 512.  If you don't use that, inefficiencies
> will result. That's not serious.

Ok, I changed it to 512 and there are no warnings about blocksize now.

But now, another enbd problem :  sometimes, after connecting, nda device is 
totally unuseable :  i have "buffer IO errors", and device is not accessible.
I tried to stop client and server, restart it, but nothing changes. when 
removing module i've got an oops. Rebooting solves sometimes the problem, but 
not everytime.

the strangest is that i have 2 enbd devices. the first connected is working fine 
, and problems occurs sometimes with the second one.

here my enbd.conf :
for client :

module A merge_requests=1 show_errs=1

# 'client' ID DEVICE    SERVER     PORT CONNECTIONS ... OPTIONS
# E.g.
# client   0  /dev/nd/a dit000.lab 1035 -n 4            -m -b 4096

client A-1 /dev/nda ldap-croise 12345 -n 4 -m  -b 512
client B-1 /dev/ndb ldap-croise 12350 -n 4 -m  -b 512

for server :

# 'server' ID PORT RESOURCE RESOURCE ... OPTIONS
# E.g.
# server   0  1035 /dev/users/lab
# server   1  2039 /dev/users/lm

server A-1 12345 /dev/sda -b 512
server B-1 12350 /dev/sdb -b 512




here are the logs clientside : corresponding to enbd stop and and enbd start
stop :

ENBD #3134[5]: enbd_disable disabled device nda
ENBD #2194[28]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[22]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[22]: enbd_soft_reset run enbd_request on nda
ENBD #2194[29]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[23]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[23]: enbd_soft_reset run enbd_request on nda
ENBD #2194[30]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[24]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[24]: enbd_soft_reset run enbd_request on nda
ENBD #1805[22]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #1805[23]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #1805[24]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #1805[25]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #2194[31]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[25]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[25]: enbd_soft_reset run enbd_request on nda
ENBD #1805[26]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #2194[32]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[26]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[26]: enbd_soft_reset run enbd_request on nda
ENBD #1805[27]: enbd_get_req Our slot has been vamooshed (result -19).
ENBD #2194[33]: enbd_clr_queue unqueued 0 reqs
ENBD #2133[2]: enbd_clr_kernel_queue removed 0 requests
ENBD #2535[11]: enbd_enable set VALID on nda
ENBD #3700[42]: fixup_slot failed to find slot for pid 6557 for ioctl 4004ab04 
arg 199d
ENBD #3850[42]: enbd_ioctl CLEAR_SOCK called on full device nda arg 199d
ENBD #4234[71]: enbd_release (3): erasing slot pid 6557
ENBD #3700[43]: fixup_slot failed to find slot for pid 6555 for ioctl 4004ab04 
arg 199b
ENBD #3850[43]: enbd_ioctl CLEAR_SOCK called on full device nda arg 199b
ENBD #4234[72]: enbd_release (1): erasing slot pid 6555
ENBD #4234[73]: enbd_release (2): erasing slot pid 6556
ENBD #2194[34]: enbd_clr_queue unqueued 0 reqs
ENBD #3224[27]: enbd_soft_reset INVALIDATE DEVICE nda
ENBD #3234[27]: enbd_soft_reset run enbd_request on nda
ENBD #4234[74]: enbd_release (0): erasing slot pid 6554
ENBD #3767[27]: fixup_slot failed to find slot for pid 6851 ioctl 4004ab10 arg 
bfa4ee58 (312d41)

start :

ENBD #2535[12]: enbd_enable set VALID on nda
ENBD #824[105]: enbd_open DISK CHECK wanted in open on device nda
ENBD #4404[105]: enbd_media_changed MEDIA CHANGED called
ENBD #4410[105]: enbd_media_changed REMOTE CHECK done locally, no remote ioctl!
ENBD #4508[200]: enbd_revalidate REVALIDATE called on nda
ENBD #4514[190]: enbd_revalidate asked to revalidate remotely dead drive nda
ENBD #4508[201]: enbd_revalidate REVALIDATE called on nda
ENBD #4514[191]: enbd_revalidate asked to revalidate remotely dead drive nda
  nda:remote device invalid.
printk: 22 messages suppressed.
Buffer I/O error on device nda, logical block 0
Buffer I/O error on device nda, logical block 1
Buffer I/O error on device nda, logical block 2
Buffer I/O error on device nda, logical block 3
Buffer I/O error on device nda, logical block 4
Buffer I/O error on device nda, logical block 5
Buffer I/O error on device nda, logical block 6
Buffer I/O error on device nda, logical block 7
Buffer I/O error on device nda, logical block 6
Buffer I/O error on device nda, logical block 7
  unable to read partition table


here syslog for start :

Jul  7 18:05:12 se3 enbd-client: enbd-client: client says target 0 is 
ldap-croise:12345
Jul  7 18:05:12 se3 enbd-client: enbd-client: client identifier is A-1
Jul  7 18:05:12 se3 enbd-client: enbd-client: client channels is 4
Jul  7 18:05:12 se3 enbd-client: enbd-client: client cksum flag is 1
Jul  7 18:05:12 se3 enbd-client: enbd-client: client says blksize is 512
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) opened device /dev/nda
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager opened NBD 
device /dev/nda (2b00)
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) set kernel bdflush 
sync boundary to 80% from -1210043408%
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) left kernel bdflush 
async boundary at -1079703484%
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager detaches from 
shell
Jul  7 18:05:12 se3 enbd-client: nbd/pidfile  6851: <# 144> lockpidfile 
returning err 0 for lock attempt
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager made pidfile 
/var/run/enbd-client-A-1.pid with pid 6851
Jul  7 18:05:12 se3 enbd-client: nbd/time  6851: <#  64> mygettimeofday time was 
0+0E-6 s and is 1120752312+854830E-6 s
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) starts introduction 
sequence on ldap-croise:12345
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) got size 159966560256
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) got signature [A-1], 
had []
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) negotiated blksize 512
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) negotiated pulse_intvl 10
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) got session port 12346 ok
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) introduction sequence 
ends ok
Jul  7 18:05:12 se3 kernel: ENBD #3767[27]: fixup_slot failed to find slot for 
pid 6851 ioctl 4004ab10 arg bfa4ee58 (312d41)
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) set device size 
159966560256
Jul  7 18:05:12 se3 enbd-client: enbd-client  6851: <#1693> setdeviceblksize 
client (-1) Warning! kernel says blksz is 4096 != 512
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) set md5sum flags on device
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) sets session slots to 0-3
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager launched 
daemon 0 (6852) for ldap-croise:12346
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager launched 
daemon 1 (6853) for ldap-croise:12346
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (-1) manager launched 
daemon 2 (6854) for ldap-croise:12346
Jul  7 18:05:12 se3 enbd-client: enbd-client: client (0) opened device /dev/nda
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (-1) manager launched 
daemon 3 (6855) for ldap-croise:12346
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (0) opened socket 5 to 
ldap-croise:12346
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (1) opened device /dev/nda
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (2) opened device /dev/nda
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (3) opened device /dev/nda
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (1) opened socket 5 to 
ldap-croise:12346
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (3) opened socket 5 to 
ldap-croise:12346
Jul  7 18:05:13 se3 enbd-client: enbd-client: client (2) opened socket 5 to 
ldap-croise:12346
Jul  7 18:05:14 se3 enbd-client: enbd-client: client (0) read passwd ok from 
ldap-croise:12346
Jul  7 18:05:14 se3 enbd-client: enbd-client: client (0) got cliserv magic ok 
from ldap-croise:12346
Jul  7 18:05:14 se3 enbd-client: enbd-client: client (1) read passwd ok from 
ldap-croise:12346
Jul  7 18:05:14 se3 enbd-client: enbd-client: client (1) got cliserv magic ok 
from ldap-croise:12346
Jul  7 18:05:15 se3 enbd-client: enbd-client: client (0) got a signature ok from 
ldap-croise:12346
Jul  7 18:05:15 se3 enbd-client: enbd-client: client (0) begins main loop
Jul  7 18:05:15 se3 kernel: ENBD #2535[12]: enbd_enable set VALID on nda
Jul  7 18:05:15 se3 enbd-client: enbd-client: client (1) got a signature ok from 
ldap-croise:12346
Jul  7 18:05:15 se3 enbd-client: enbd-client: client (1) begins main loop
Jul  7 18:05:16 se3 enbd-client: enbd-client: client (2) read passwd ok from 
ldap-croise:12346
Jul  7 18:05:16 se3 enbd-client: enbd-client: client (2) got cliserv magic ok 
from ldap-croise:12346
Jul  7 18:05:16 se3 enbd-client: enbd-client: client (2) got a signature ok from 
ldap-croise:12346
Jul  7 18:05:16 se3 enbd-client: enbd-client: client (2) begins main loop
Jul  7 18:05:17 se3 enbd-client: enbd-client: client (3) read passwd ok from 
ldap-croise:12346
Jul  7 18:05:17 se3 enbd-client: enbd-client: client (3) got cliserv magic ok 
from ldap-croise:12346
Jul  7 18:05:17 se3 enbd-client: enbd-client: client (3) got a signature ok from 
ldap-croise:12346
Jul  7 18:05:17 se3 enbd-client: enbd-client: client (3) begins main loop
Jul  7 18:05:18 se3 enbd-client: enbd-client: client (-1) opened device /dev/nda
Jul  7 18:05:18 se3 kernel: ENBD #824[105]: enbd_open DISK CHECK wanted in open 
on device nda
Jul  7 18:05:18 se3 kernel: ENBD #4404[105]: enbd_media_changed MEDIA CHANGED called
Jul  7 18:05:18 se3 kernel: ENBD #4410[105]: enbd_media_changed REMOTE CHECK 
done locally, no remote ioctl!
Jul  7 18:05:18 se3 kernel: ENBD #4508[200]: enbd_revalidate REVALIDATE called 
on nda
Jul  7 18:05:18 se3 kernel: ENBD #4514[190]: enbd_revalidate asked to revalidate 
remotely dead drive nda
Jul  7 18:05:18 se3 kernel: ENBD #4508[201]: enbd_revalidate REVALIDATE called 
on nda
Jul  7 18:05:18 se3 kernel: ENBD #4514[191]: enbd_revalidate asked to revalidate 
remotely dead drive nda
Jul  7 18:05:18 se3 kernel:  nda:remote device invalid.
Jul  7 18:05:18 se3 kernel: printk: 22 messages suppressed.
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 0
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 1
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 2
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 3
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 4
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 5
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 6
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 7
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 6
Jul  7 18:05:18 se3 kernel: Buffer I/O error on device nda, logical block 7
Jul  7 18:05:18 se3 kernel:  unable to read partition table


here /proc/nbdinfo

Device a:       Open
[a] State:      verify, rw, merge requests, enabled, remote invalid, last error 
0, lives 6, bp 0
[a] Queued:     +0R/0W curr (check 0R/0W) +0R/0W max
[a] Buffersize: 262144  (sectors=512, blocks=512)
[a] Blocksize:  512     (log=9)
[a] Size:       156217344KB
[a] Blocks:     0
[a] Sockets:    4       (+)     (+)     (+)     (*)
[a] Requested:  0       (0)     (0)     (0)     (0)     0R/0W   max 0
[a] Despatched: 0       (0)     (0)     (0)     (0)     0R/0W   md5 0W (0 eq, 0 
ne, 0 dn)
[a] Errored:    3.3205K (0)     (0)     (0)     (0)     0+3.320K
[a] Pending:    0       (0)     (0)     (0)     (0)     0R/0W+0R/0W
[a] B/s now:    0       (0R+0W)
[a] B/s ave:    0       (0R+0W)
[a] B/s max:    0       (0R+0W)
[a] Spectrum:
[a] Kthreads:   0       (0 waiting/0 running/1 max)
[a] Cthreads:   4       (+)     (+)     (+)     (+)
[a] Cpids:      4       (6852)  (6853)  (6854)  (6855)

and oops after module removing  :

Jul  4 15:32:55 se3 kernel: Buffer I/O error on device nda, logical block 536870640
Jul  4 15:32:59 se3 kernel: ENBD #5287[0]: enbd_cleanup module cleaned up.
Jul  4 15:32:59 se3 kernel: Unable to handle kernel paging request at virtual 
address f8922cf8
Jul  4 15:32:59 se3 kernel:  printing eip:
Jul  4 15:32:59 se3 kernel: c02c8e0a
Jul  4 15:32:59 se3 kernel: *pde = 017ea067
Jul  4 15:32:59 se3 kernel: *pte = 00000000
Jul  4 15:32:59 se3 kernel: Oops: 0000 [#1]
Jul  4 15:32:59 se3 kernel: PREEMPT
Jul  4 15:32:59 se3 kernel: Modules linked in: parport_pc lp parport uhci_hcd 
ehci_hcd i2c_piix4 i2c_core sworks_agp ohci_hcd megaraid via_rhine r8169 e1000
Jul  4 15:32:59 se3 kernel: CPU:    0
Jul  4 15:32:59 se3 kernel: EIP:    0060:[get_disk+18/160]    Not tainted VLI
Jul  4 15:32:59 se3 kernel: EFLAGS: 00010282   (2.6.12.enbd-fr1)
Jul  4 15:32:59 se3 kernel: EIP is at get_disk+0x12/0xa0
Jul  4 15:32:59 se3 kernel: eax: f8922ce0   ebx: f73cd480   ecx: 0000002b   edx: 
00000000
Jul  4 15:32:59 se3 kernel: esi: 00000000   edi: f73cd480   ebp: c02c8618   esp: 
f35cfe9c
Jul  4 15:32:59 se3 kernel: ds: 007b   es: 007b   ss: 0068
Jul  4 15:32:59 se3 kernel: Process mdadm (pid: 3989, threadinfo=f35ce000 
task=f43e2020)
Jul  4 15:32:59 se3 kernel: Stack: f73cd480 00000000 f3b28540 c02c862e f73cd480 
c02c1297 02b00000 f73cd480
Jul  4 15:32:59 se3 kernel:        00000001 fffffffa c17e5480 c17e1300 0000000f 
c02c86b5 f7cb9400 02b00000
Jul  4 15:32:59 se3 kernel:        f35cff00 c01533f1 02b00000 f35cff00 00000001 
c17e5480 f39de480 c17e1300
Jul  4 15:32:59 se3 kernel: Call Trace:
Jul  4 15:32:59 se3 kernel:  [exact_lock+10/28] exact_lock+0xa/0x1c
Jul  4 15:32:59 se3 kernel:  [kobj_lookup+191/428] kobj_lookup+0xbf/0x1ac
Jul  4 15:32:59 se3 kernel:  [get_gendisk+21/44] get_gendisk+0x15/0x2c
Jul  4 15:32:59 se3 kernel:  [do_open+53/784] do_open+0x35/0x310
Jul  4 15:32:59 se3 kernel:  [blkdev_open+30/96] blkdev_open+0x1e/0x60
Jul  4 15:32:59 se3 kernel:  [dentry_open+252/500] dentry_open+0xfc/0x1f4
Jul  4 15:32:59 se3 kernel:  [filp_open+82/92] filp_open+0x52/0x5c
Jul  4 15:32:59 se3 kernel:  [get_unused_fd+80/180] get_unused_fd+0x50/0xb4
Jul  4 15:32:59 se3 kernel:  [sys_open+55/112] sys_open+0x37/0x70
Jul  4 15:32:59 se3 kernel:  [syscall_call+7/11] syscall_call+0x7/0xb



Oh yes, another thing too : this is with the other device that works correctly 
in fr1 array : syslog is blown continuously with these messages :

Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (0) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (1) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (1) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (2) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (2) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (3) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (3) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (0) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (0) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (2) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (2) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (1) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (1) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (3) yes it can, result 0
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: seeing if (3) can discard one 
of 1 write acks pending
Jul  8 06:28:03 se3 enbd-client: enbd-netserver: (0) yes it can, result 0