Re: How to automatically add a nbd to a raid1?

On Thursday 11 December 2008 20:13, Peter T. Breuer wrote:
> "Also sprach Michael Rendell:"
>
> > > >   I'd like to set up a raid1 disk that has a local disk and a remote
> > > > (enbd) disk.  Was wondering how to set things up so that, on/after
> > > > reboot, the enbd device gets added to the raid when the connection to
> > > > the enbd-server is made?
> > >
> > > There are scripts to handle that in the distribution package. I'm
> > > travelling today but I'll put up a complete answer later (about
> > > midnight CET).
> >
> > When booting, there is a window of time between when the raid drives are
> > started and when the network/enbd get going.
>
> Yes, but don't worry about it. The raid will come up in degraded mode
> if it comes up first (shrug .. it's your choice), then enbd will come
> up, and add itself to the degraded array via whatever scripts exist.
>
> Then the array will silently bring the enbd device up to date or vice
> versa in the background. That won't cost much because RAID is
> intelligent these days and only updates out-of-date blocks, as recorded
> on the bitmap.
>
> > Am using auto-detect on the
> > raid drives,
>
> Uh, I think that means partition type e0 and md compiled into the
> kernel, which does a special hack at bootup to look for them ...
>
> > so there is no file saying which drives are a part of which
> > raid array, so the raid is set up/started without any /dev/ndA disks,
>
> That's kind of understandable. But the raid disks themselves contain
> all the required info in their superblocks. Each will say what the
> other members of the array are. You should be able to interrogate them
> individually with mdadm --whatever and confirm my contention!

While "mdadm --detail --verbose /dev/mdNN" doesn't list the removed
devices, one can figure out where a new device should go by looking
at the output of "mdadm --examine /dev/ndA" (match the UUID and raid level).

> So
> while I can understand that the kernel itself doesn't do any thinking
> and helping, one could remedy the omission very shortly after booting
> has commenced and make good.
>
> > and therefor mdmonitor doesn't pass on the "fail" event to the pwprog
> > script.
>
> I don't recall  Oh ...  that "pwprog" rings a bell. That's my script?
> Yes, I see it is. OK. Yep, I see it now. So, hmm, it's meant to be run
> by mdadm --monitor. I see. Feel free to improve it!
>
> It seems to pick up the Fail and SpareActive notices from md. And it
> does appropriate bits of wizardry. It seems to ping the remote enbd
> end in order to notice when it's available again, and add it into the
> array (see the "repair" function).  One would kind of feel that would
> add enbd into the initial degraded array when it comes up ... but you
> say there's no Fail notice from md since it starts degraded. Hem hem.
>
>
> But how does this script know what the enbd components of each array are
> supposed to be anyway? It obviously does, but blessed if I see how.
> Maybe you could add a few comments to it for me :).
>
> Oh, I see. It relies on md to tell it what the array and its components
> are, and only reacts if they are enbd-ish enough. Quite a clever
> script - worth commenting!
>
> > ...
> >
> > > mdadm --monitor works fine. As I recall that's what is used. ENBD
> > > signals udev when conditions change.
> >
> >   OK - udev is what I was missing - should be able to add a RUN key to
> > the udev rules to run a program that can see if the device should
> > be added to a raid array (mdadm --examine, etc.).
>
> Yerrs .. I remember checking that enbd does indeed produce udev events.
> I don't recall what they are, though. I see the driver code says:
>
>    * 12-04-08 notify udev of events ptb 2.4.36 (+ backport)
>
> and there are special hacks in the driver code to deal with some nasty
> probes that udev instigates. And I see
>
>      kobject_uevent(kobj, KOBJ_ONLINE);
>      ENBD_ALERT ("notified udev nd%s is up\n", lo->devnam);
>
>      ...
>
>      kobject_uevent(kobj, KOBJ_OFFLINE);
>      ENBD_ALERT ("notified udev nd%s is down\n", lo->devnam);
>
> so it looks like there are two kinds of event notified to udev. I don't
> know offhand what they look like in udev logs, but I see the alert
> notices above in syslog. I don't see udev notices anywhere ... well, it
> would help to set udev_log to info or debug in /etc/udev/udev.conf.
>
> You'd better do that and tell me what udev gets! I have to go to sleep
> for another trip tomorrow morning! I recall now that I have tested it,
> so presumably I knew at one point what udev would see, but I've
> completely forgotten what it is.

Have had a look at the udev events by adding the line
    KERNEL=="nd[a-z]", SUBSYSTEM=="block", RUN+="/root/enbd-udev-watch"
to /etc/udev/rules.d/90-enbd.rules, where the /root/enbd-udev-watch
script looks like:
    #!/bin/sh

    exec >> /tmp/enbd-udev-watch.log 2>&1
    echo
    echo "Run at " $(date)
    echo "Env {:"
    env | sed 's/^/ /';
    echo "}"
    echo
It shows that ACTION=add events happen for all 16 /dev/nd[a-p] devices when
the kernel module is loaded.  As this is before the enbd-client is
started (and before it gets the connection going), it isn't useful
for this purpose.

Think the simplest thing to do is to write a script that looks
at /etc/enbd-client.conf for a list of devices and then 
watches /proc/nbdinfo for these devices to be Open and enabled.
It can then use "mdadm --examine" to determine if it is an element
in a raid array and, if so, search for the array and add it
(it can then forget about that device as mdmonitor will take over then;
when there are no devices left in it list, it can exit).
Can send the script when its done in if that is of use.

In the process of looking into these things, noticed that
enbd-client's -n N option could do with some range checking
to avoid going past the end of an array:

--- nbd/enbd-client.c.Orig	2008-12-12 11:57:08.000000000 -0330
+++ nbd/enbd-client.c	2008-12-12 12:24:24.000000000 -0330
@@ -452,6 +452,12 @@
 set_channels (struct nbd_client_session *session, int n)
 {
 
+    if (n > ENBD_MAXCONN) {
+	PERR ("client specified too many channels (%d) - limiting to %d.\n",
+	    n, ENBD_MAXCONN);
+	session->nchan = n = ENBD_MAXCONN;
+    }
+
     // PTB only accept channels after we know server
     if (!session->client.hostname) {
         session->pending_nchan = n;

(discovered this when trying to speed up the re-sync process by adding
many channels).  Note that the array is session->clients[ENBD_MAXCONN+1]
so the "n > ENBD_MAXCONN" above perhaps should be "n > ENBD_MAXCONN + 1".

Also noticed that "less /proc/nbdinfo" causes less to segfault;
strace shows:
    open("/proc/nbdinfo", O_RDONLY|O_LARGEFILE) = 4
    _llseek(4, 1, [1], SEEK_SET)            = 0
    _llseek(4, 0, [0], SEEK_SET)            = 0
 ** read(4, "Device a:\tOpen \n[a] State:\tverif"..., 64) = 3161
don't have a fix for this at the moment, as can use "cat /proc/nbdinfo | 
less".
If time allows, will have a look at this.

Thanks for your suggestions and for enbd itself!

Best wishes,
Michael