Re: Re: General question

On Mon, 6 Jun 2005 00:47:56 +0200
[email protected] (Peter T. Breuer) wrote:

> In article <[email protected]> you wrote:
> > On Sun, 5 Jun 2005 19:28:55 +0200 (MET DST)
> > "Peter T. Breuer" <[email protected]> wrote:
> 
> > > "Also sprach Stephan von Krawczynski:"
> > > > unplugging the network between the two enbd-servers.
> > > > Now something interesting happens:
> > > > Bonnie stops right away (freezes), the raid1 in fact freezes. You
> > > > cannot do
> > > 
> > > You are supposed to use FR1 with enbd, not raid1.
> 
> > I did.
> 
> OK - well that turns on show_errs when the enbd device is used inside a
> raid1 (FR1) device, so I don't know what is going on. I would suspect
> that you are NOT using fr1, although you think you are ...

Well, really, I am using it. It is version fr1-2.17. Btw I found no reference
to show_err in the patches there...
 
> > > > anything with it. You cannot even setraidfaulty the now-gone
> > > > enbd-server partition.
> > > 
> > > It sounds like you are not using enbd with show_errs set (so raid1 sees
> > > no error, but simply blocks forever when enbd cannot complete a
> > > transaction, since enbd blocks forever waiting for a reconnect).  But
> > > anyway, use FR1. Then you don't need to set anything.
> 
> > Well, as stated, I did use fr1.  See:
> 
> Are you sure that you are using the NEW md module? That's the part that
> is responsible for telling the enbd module that it is in a raid array,
> and for reacting to a network error message from enbd.
> 
> > :~ # lsmod
> > Module                  Size  Used by    Not tainted
> > fr1                    18356   1 
> > bitmap                  6416   0  [fr1]
> > md                     49664   2  [fr1]
> > enbd                   71788   8 

It wouldn't open fr1 module if it were the wrong version of md, not?

> > Dumb question: how do I set show_errs ?
> 
> You don't - it's done for you if you ue the md/fr1 pair from the fr1
> patch. But it should be shown in the output of /proc/nbdinfo (State
> line).  And you can turn it on manually by echoing show_errs=1 to
> /proc/nbdinfo, or by using it as a module parameter.  I think there
> might be inputs for it in /proc/sys/dev too, but don't quote me on that!

Ok, I set show_err=1 via proc and see the following:

If I shoot one enbd-server process manually the raid goes into degraded but
continues to work. So this is ok.
Unfortunately one has to restart the respective enbd-client, too, when the
corresponding server process died (or the corresponding machine died). Obvious
reason is documented (the magic does not match). In fact this is bad because
the enbd-client side has to know that it must restart the client process, which
is not very straight forward.
But I experienced another problem: unplugging the network.
If you do that the raid goes in degraded mode (correct).
But if you re-plug the network again, it does _not_ reconstruct, but loops with:

(client-side)
enbd-client 26138: client (1) read passwd ok from nrhost2:60002
enbd-client 26138: client (1) got cliserv magic ok from nrhost2:60002
enbd-client 26138: client (1) got a signature ok from nrhost2:60002
enbd-client 26138: client (1) begins main loop
enbd-client 26138: <#1488> newproto get request from kernel failed (-1) Bad file descriptor
enbd-client 26137: <#1488> newproto get request from kernel failed (-1) Bad file descriptor
enbd-client 26138: <#1488> newproto get request from kernel failed (-1) Bad file descriptor
enbd-client 26137: <#1488> newproto get request from kernel failed (-1) Bad file descriptor
enbd-client 26138: <#1488> newproto get request from kernel failed (-1) Bad file descriptor
enbd-client 26137: <#1449> newproto kernel errored 4 times when we asked for a new req: Bad file descriptor
enbd-client 26138: <#1449> newproto kernel errored 4 times when we asked for a new req: Bad file descriptor
enbd-client 26137: <# 156> unplug requested unplug (0) Bad file descriptor on nrhost2:60002 so clear socket
enbd-client 26137: client (0) last error Bad file descriptor
enbd-client 24989: sighandler relaunches child from manager 
enbd-client 26138: <# 156> unplug requested unplug (1) Bad file descriptor on nrhost2:60002 so clear socket
enbd-client 26138: client (1) last error Bad file descriptor
enbd-client 24989: sighandler relaunches child from manager 
enbd-client 24989: client (-1) reaped dead child 26138 (died of signal 0 with exit status 0)
enbd-client 24989: client (-1) reaped dead child 26137 (died of signal 0 with exit status 0)

and:

Jun  6 10:44:02 mail-a01 kernel: raid1: bitmap f6fab800 already active!
Jun  6 10:44:02 mail-a01 kernel: raid1: Disk failure on ndb, disabling device. 
Jun  6 10:44:02 mail-a01 kernel:        Operation continuing on 1 devices
Jun  6 10:44:02 mail-a01 kernel: device not enabled.
Jun  6 10:44:02 mail-a01 kernel: ENBD enbd.c #3467[58]: do_nbd_request ending req f7e02580 with prejudice
Jun  6 10:44:02 mail-a01 kernel: end_request: I/O error, dev 2b:10 (nbd), sector 8624
Jun  6 10:44:02 mail-a01 kernel: md: md_do_sync() got signal ... exiting
Jun  6 10:44:02 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:44:02 mail-a01 kernel:  --- wd:1 rd:2 nd:2
Jun  6 10:44:02 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:44:02 mail-a01 kernel:  disk 1, s:0, o:0, n:1 rd:1 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel:  disk 2, s:1, o:0, n:2 rd:2 us:1 dev:ndb
Jun  6 10:44:02 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:44:02 mail-a01 kernel:  --- wd:1 rd:2 nd:2
Jun  6 10:44:02 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:44:02 mail-a01 kernel:  disk 1, s:0, o:0, n:1 rd:1 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel:  disk 2, s:1, o:0, n:2 rd:2 us:1 dev:ndb
Jun  6 10:44:02 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:44:02 mail-a01 kernel:  --- wd:1 rd:2 nd:2
Jun  6 10:44:02 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:44:02 mail-a01 kernel:  disk 1, s:0, o:0, n:1 rd:1 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel:  disk 2, s:1, o:0, n:2 rd:2 us:1 dev:ndb
Jun  6 10:44:02 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:44:02 mail-a01 kernel:  --- wd:1 rd:2 nd:2
Jun  6 10:44:02 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:44:02 mail-a01 kernel:  disk 1, s:0, o:0, n:1 rd:1 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel:  disk 2, s:1, o:0, n:2 rd:2 us:1 dev:ndb
Jun  6 10:44:02 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:44:02 mail-a01 kernel: md: recovery thread finished ...
Jun  6 10:44:02 mail-a01 kernel: md: recovery thread got woken up ...
Jun  6 10:44:02 mail-a01 kernel: md0: no spare disk to reconstruct array! -- continuing in degraded mode
Jun  6 10:44:02 mail-a01 kernel: md: recovery thread finished ...
Jun  6 10:44:10 mail-a01 kernel: ENBD enbd.c #4993[71]: enbd_ioctl ENBD_CLEAR_SOCK 26137 from pid 26137
Jun  6 10:44:10 mail-a01 kernel: entered with arg 0 for pid 26137, tgid 26137
Jun  6 10:44:10 mail-a01 kernel: entered with arg 0 for pid 26137, tgid 26137
Jun  6 10:44:10 mail-a01 kernel: ENBD enbd.c #4993[72]: enbd_ioctl ENBD_CLEAR_SOCK 26138 from pid 26138
Jun  6 10:44:10 mail-a01 kernel: entered with arg 1 for pid 26138, tgid 26138
Jun  6 10:44:10 mail-a01 kernel: entered with arg 1 for pid 26138, tgid 26138

The only way to reconstruct the raid is again restarting the enbd-client. Then:

Jun  6 10:49:49 mail-a01 kernel: md0: retained bitmap f6fab800
Jun  6 10:49:49 mail-a01 kernel: raid1: skipped clean sectors 0-127
Jun  6 10:49:49 mail-a01 kernel: raid1: synced dirty sectors 128-143
Jun  6 10:49:49 mail-a01 kernel:  unknown partition table
Jun  6 10:49:49 mail-a01 kernel: ENBD enbd.c #1338[3]: enbd_reread_partitions validated device ndb
Jun  6 10:49:50 mail-a01 kernel: raid1: skipped clean sectors 144-8631
Jun  6 10:49:51 mail-a01 kernel: raid1: synced dirty sectors 8632-38831
Jun  6 10:49:51 mail-a01 kernel: raid1: skipped clean sectors 38832-325591
Jun  6 10:49:52 mail-a01 kernel: ENBD enbd.c #5798[9]: enbd_media_changed change ndb requested
Jun  6 10:49:52 mail-a01 kernel: ENBD enbd.c #5805[9]: enbd_media_changed REMOTE CHECK done locally, no remote ioctl!
Jun  6 10:49:57 mail-a01 kernel: ENBD enbd.c #4993[73]: enbd_ioctl ENBD_CLEAR_SOCK 26296 from pid 26296
Jun  6 10:49:57 mail-a01 kernel: entered with arg 1 for pid 26296, tgid 26296
Jun  6 10:49:57 mail-a01 kernel: decremented active socket count to 1
Jun  6 10:49:57 mail-a01 kernel: entered with arg 1 for pid 26296, tgid 26296
Jun  6 10:49:57 mail-a01 kernel: decremented active socket count to 1
Jun  6 10:49:59 mail-a01 kernel: raid1: synced dirty sectors 325592-475511
Jun  6 10:50:00 mail-a01 kernel: raid1: skipped clean sectors 475512-1975991
Jun  6 10:50:00 mail-a01 kernel: md: md0: sync done.
Jun  6 10:50:00 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:50:00 mail-a01 kernel:  --- wd:1 rd:2 nd:2
Jun  6 10:50:00 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:50:00 mail-a01 kernel:  disk 1, s:0, o:0, n:1 rd:1 us:0 dev:[dev 00:00]
Jun  6 10:50:00 mail-a01 kernel:  disk 2, s:1, o:1, n:2 rd:2 us:1 dev:ndb
Jun  6 10:50:00 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:50:00 mail-a01 kernel: RAID1 conf printout:
Jun  6 10:50:00 mail-a01 kernel:  --- wd:2 rd:2 nd:2
Jun  6 10:50:00 mail-a01 kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:nda
Jun  6 10:50:00 mail-a01 kernel:  disk 1, s:0, o:1, n:1 rd:1 us:1 dev:ndb
Jun  6 10:50:00 mail-a01 kernel:  disk 2, s:0, o:0, n:2 rd:2 us:0 dev:[dev 00:00]
Jun  6 10:50:00 mail-a01 kernel:  disk 3, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
Jun  6 10:50:00 mail-a01 kernel: md: recovery thread finished ...

And everything is fine again.

Can we do anything to get this a bit more automatic, without additional daemons or manual interaction?

Regards,
Stephan