Re: How to automatically add a nbd to a raid1?
Michael Rendell <[email protected]> Wed, 17 Dec 2008 16:41:54 -0330
| Newsgroups | gmane.linux.enbd.general |
|---|---|
| Message-ID | <[email protected]> |
On Monday 15 December 2008 11:54, Peter Breuer wrote:
> "Also sprach Michael Rendell:"
>
> > > That's kind of understandable. But the raid disks themselves contain
> > > all the required info in their superblocks. Each will say what the
> > > other members of the array are. You should be able to interrogate them
> > > individually with mdadm --whatever and confirm my contention!
> >
> > While "mdadm --detail --verbose /dev/mdNN" doesn't list the removed
> > devices,
>
> It should .. morally, I mean. It KNOWS what devices are notionally in
> the array.
It may know, but it's not telling (via the mdadm interface anyway);
seems the --verbose option doesn't do much for the --detail mode.
...
> > the kernel module is loaded. As this is before the enbd-client is
> > started (and before it gets the connection going), it isn't useful
> > for this purpose.
>
> Yes. But what you want is the ONLINE and OFFLINE notices that enbd
> sends to udev, not the ADD.
>
> The ADD and REMOVE events are just generated by the kernel disk
> partition subsystem as the driver registers control of its majors and
> minors. They're what udev rules send to the MAKEDEV and REMOVEDEV
> scripts in the enbd distr, via the provided enbd.rules file.
Used udevmonitor to see what events are generated and there
only appear to be the add and remove events (no online/offline).
Am running enbd-2.4.35 and running the stock CentOS-5 kernel
(2.6.18-92.1.18.el5).
Have been testing/playing with the rc.d init scripts and
the pwprog scripts and have a number of patches for them,
some of which may be of use.
Patches for pwprog:
- renamed to enbd-pwprog to help me keep things straight
- when writing lock files, write the running process's PID
to the lock file and not "$$" ($$ after a "&" is not the
currently running process PID - it is the original process PID).
- update the pid in the lock file after a "&"
- added a "disable" file so can play with a drive without
pwprog starting up and fixing things; also useful when
shutting down the system.
- added disable/enable command line options to create/remove
the disable file.
- put logging functions in (they were in an old version of
pwprog that you wrote).
Patches for etc/init/enbd script:
- added a comment header so "chkconfig" will work (you may not want this)
- added a default/sysconfig CLIENT_WAIT_FOR_SERVER variable to
control whether, when starting enbd-client, to wait for the
server to be pingable (find waiting not useful when using -t 0).
- added a default/sysconfig OUTPUT_TO_DEV_NULL variable to send
enbd-client and enbd-server stdout/stderr to dev null so
the screen when booting looks nicer :-)
- similar lock file / pid changes as for pwprog (see above)
- when stopping the client, wait a bit longer for the SIGTERM to
have effect (found it wasn't quite long enough);
also, use a single kill of $pid and fuser output instead
(not sure what the purpose of the fuser kill was, so this
may not be OK; change made to get rid of the two sleep commands
and just have one).
- commented out the "echo .. | socket" as there is no socket command
on this system (other calls to it were already commended out);
- touch/remove the /var/lock/subsys/enbd when starting/stopping
both the client and server (needed for the /etc/rc.d/rc script,
if the lock files doesn't exist, the service isn't stopped when
the system is shutting down). Only done if it looks like it
is being run by the rc script (i.e., $runlevel is set in the
environment).
- use the new enable/disable commands to pwprog when being
started/stopped.
--- enbd-2.4.35/nbd/pwprog.sh 2008-12-04 16:16:19.000000000 -0330
+++ /usr/sbin/enbd-pwprog 2008-12-17 15:43:16.000000000 -0330
@@ -7,12 +7,14 @@
# pwprog: args are for example <SpareActive /dev/md1 /dev/ndb>
-LOGPL="daemon.notice"
-LOGID="pwprog"
+LOGFAC="daemon"
+LOGID="enbd-pwprog"
INTVL=10
DELAY=3
+lockPid=$$
PIDEX=".pid"
+DISABLE_FILE=/var/lock/enbd-pwprog.disable
trap "echo sob sob>/dev/null" CHLD
@@ -30,25 +32,57 @@
local id="${1#/dev/nd}"
[ `cat /proc/sys/dev/enbd/devices/$id/validated` = 1 ]
}
+pwprog_disabled() {
+ if [ -f $DISABLE_FILE ] ; then
+ local mddev=$1 nbdev=$2
+ info_log "$mddev($nbdev): disable in effect - exiting"
+ return 0
+ fi
+ return 1
+}
+#
+# send message $2 to log with priority $1
+#
log() {
- logger -p $LOGPL -t "$LOGID" -- "$1"
+ logger -p $LOGFAC.$1 -t "$LOGID" -- "$2"
+}
+
+#
+# send debugging message $1
+#
+debug_log() {
+ log debug "$1"
+}
+#
+# send info message $1
+#
+info_log() {
+ log notice "$1"
+}
+#
+# send error message $1
+#
+err_log() {
+ log error "$1"
}
repair() {
local mddev="$1"
local nbdev="$2"
+ pwprog_disabled $mddev $nbdev && return 1
# Temporarily remove from array
mdadm --manage $mddev --remove $nbdev || return 1
- #echo "removed $nbdev from $mddev"
+ debug_log "removed $nbdev from $mddev"
sleep $DELAY
+ pwprog_disabled $mddev $nbdev && return 1
# Hot add to array
mdadm --manage $mddev --re-add $nbdev || return 2
- echo "hot added $nbdev to $mddev"
+ debug_log "hot added $nbdev to $mddev"
- log "[chkmd_log: $nbdev was added to $mddev and resynced on `date`]"
+ info_log "$mddev($nbdev): added and is rebuilding"
return 0
}
@@ -68,19 +102,19 @@
}
pidfile() {
- local pidfl=/var/run/pwprog-thread-`basename "$1"`$PIDEX
+ local pidfl=/var/run/$LOGID-thread-`basename "$1"`$PIDEX
echo "$pidfl"
}
lock() {
local pidfl=`pidfile "$1"`
if [ ! -s $pidfl ]; then
- echo $$ > $pidfl
+ echo $lockPid > $pidfl
return 0
fi
local p=`head -1 $pidfl`
if ! kill -0 $p >/dev/null 2>&1; then
- echo $$ > $pidfl
+ echo $lockPid > $pidfl
return 0
fi
# process with lock pid exists and is running
@@ -95,7 +129,7 @@
rm -f $pidfl
return 0
fi
- if [ "$p" = $$ ]; then
+ if [ "$p" = $lockPid ]; then
rm -f $pidfl
return 0
fi
@@ -131,24 +165,32 @@
# Launch daemon to fix things when the net comes back
- while true; do
-
- # Trigger revalidation of removable device
- head -0c $nbdev >/dev/null
-
- if can_contact "$server" && enabled $nbdev && validated $nbdev;
then
-
- echo "network is now up and $nbdev is working"
-
- repair $mddev $nbdev && unlock $nbdev && exit 0
-
- else
- echo "$nbdev has been faulted and we are waiting for it... "
-
- fi
-
- sleep $INTVL
- done &
+ info_log "$mddev($nbdev) has been faulted and we are waiting for it... "
+ {
+ # $$ doesn't change in subprocesses, so ask another process
+ lockPid=`sh -c 'echo $PPID'`
+ # Update the pid file with our actual pid
+ echo $lockPid > `pidfile $nbdev`
+ while true; do
+
+ pwprog_disabled $mddev $nbdev && { unlock $nbdev; exit 0; }
+
+ # Trigger revalidation of removable device
+ head -0c $nbdev >/dev/null
+
+ if can_contact "$server" && enabled $nbdev && validated $nbdev; then
+
+ debug_log "network is now up and $mddev($nbdev) is working"
+
+ repair $mddev $nbdev && { unlock $nbdev; exit 0; }
+
+ else
+ debug_log "$mddev($nbdev) has been faulted and we are waiting for it...
"
+ fi
+
+ sleep $INTVL
+ done
+ } &
fi
}
@@ -157,11 +199,13 @@
# Check that we are interested in this call
is_enbd "$3" || exit 0
local nbdev="$3"
+ local mddev="$2"
local pidfl=`pidfile "$nbdev"`
# Everything is miraculously OK so kill any running repair daemon
if [ -s $pidfl ]; then
local p=`head -1 $pidfl`
+ debug_log "$mddev($nbdev) is OK again - aborting repair"
kill -0 $p >/dev/null 2>&1 && abort $p
unlock $nbdev
fi
@@ -178,8 +222,29 @@
spare_active_action "$@"
;;
+ enable)
+ rm -f $DISABLE_FILE
+ ;;
+ disable)
+ touch $DISABLE_FILE
+ # Kill off any processes that are currently waiting...
+ for pidfl in /var/run/$LOGID-thread-*$PIDEX; do
+ for try in 0 1; do
+ [ ! -s $pidfl ] && break
+ p=`head -1 $pidfl`
+ if ! kill -0 $p >/dev/null 2>&1; then
+ rm -f $pidfl
+ break
+ fi
+ kill $p >/dev/null 2>&1
+ sleep 1
+ done
+ done
+ exit $?
+ ;;
+
*)
- log "args are <$*>"
+ err_log "args are <$*>"
;;
esac
--- enbd-2.4.35/nbd/etc/init.d/enbd 2008-04-23 01:52:48.000000000 -0230
+++ /etc/rc.d/init.d/enbd 2008-12-17 16:03:23.000000000 -0330
@@ -1,4 +1,10 @@
#! /bin/sh
+# enbd Enhanced network block device (server and client).
+#
+# chkconfig: 2345 74 26
+# description: Enhanced network block device (server and client).
+# processname: enbd-client, enbd-server,
+# config: /etc/enbd.conf
############################################################################
#
# Enbd init.d startup and control script
@@ -104,7 +110,8 @@
# see if a given host responds to pings.
#
can_contact() {
- ping -c 1 $1 | grep -vqs "100% packet loss" && return 0
+ [ "$CLIENT_WAIT_FOR_SERVER" = no ] && return 0
+ ping -c 1 $1 | grep -qs " 0% packet loss" && return 0
return 1
}
@@ -142,10 +149,11 @@
kill -0 "$pid" >/dev/null 2>&1 && return 1
fi
fi
- rm -f $1
- local mytmp="`mydirname $pidfile`/.try_lock_`mybasename $pidfile`.$$"
+ rm -f $pidfile
+ local mypid=`sh -c 'echo $PPID'` # $$ (after a "&") is not useful here.
+ local mytmp="`mydirname $pidfile`/.try_lock_`mybasename $pidfile`.$mypid"
touch $mytmp || return 1
- if ! echo $$ >> $mytmp ; then
+ if ! echo $mypid >> $mytmp ; then
rm -f $mytmp
return 1
fi
@@ -214,31 +222,41 @@
message="`client_ready $server $port`"
done
fi
- $name ${server}:$port $args &
+ if [ "$OUTPUT_TO_DEV_NULL" = yes ] ; then
+ $name ${server}:$port $args > /dev/null 2>&1 &
+ else
+ $name ${server}:$port $args &
+ fi
# the executable will write this itself in >= 2.4.24
#local pid=$!
#echo $pid > /var/run/nbd-client-$id.pid
## tell the server to restart its nbd-server
#telnet $server nbd
- echo done
+ echo $name $id done
}
#########################################################################
# how to do lookup and murder via a pidfile
#
stop_foo() {
local pidfile=$1
+ local delay=2 p pids
[ ! -s $pidfile ] && return 1
pid="`head -1 $pidfile`"
for sig in USR1 TERM 9; do
- kill -$sig $pid >/dev/null 2>&1
- sleep 1
- fuser -k -$sig $pidfile >/dev/null 2>&1
- sleep 1
+ pids=$pid
+ for p in '' `fuser $pidfile 2> /dev/null`; do
+ [ X"$p" = X ] && continue
+ [ X"$p" = X"$pid" ] && continue
+ pids="$pids $p"
+ done
+ kill -$sig $pids > /dev/null 2>&1
+ sleep $delay
if ! kill -0 $pid >/dev/null 2>&1 ; then
#echo removing $pidfile because process $pid is dead
#rm -f $pidfile
return 0
fi
+ delay=4
done
return 1
}
@@ -264,7 +282,7 @@
echo -n stopping $CLIENTNAME $id
if stop_foo /var/run/${CLIENTNAME}-$id.pid; then
- ( echo notice client-stop $port $ipaddrs; echo quit ) | socket $server
$SSTATDNAME
+ #( echo notice client-stop $port $ipaddrs; echo quit ) | socket
$server $SSTATDNAME
echo done
return
fi
@@ -361,7 +379,11 @@
message="`server_ready $resources`"
done
fi
- $name $port $resources $options &
+ if [ "$OUTPUT_TO_DEV_NULL" = yes ] ; then
+ $name $port $resources $options > /dev/null 2>&1 &
+ else
+ $name $port $resources $options &
+ fi
# the executable will write this itself in >= 2.4.24
#local pid=$!
#echo $pid > /var/run/${SERVERNAME}-$id.pid
@@ -478,6 +500,11 @@
to_what="$2"
to_which="$3"
+ # If being run by /etc/rc.d/rc at boot/shutdown time, need to
+ # play with /var/lock/subsys/xxx file.
+ isRC=no
+ [ X"$runlevel" != X ] && [ X"$to_what$to_which" = X ] && isRC=yes
+
[ "$to_what" = all ] && to_what=""
[ "$to_which" = "-a" ] && to_which=""
@@ -518,6 +545,9 @@
${do_what}_module $id $rest
;;
client)
+ # Allow/prevent pwprog (run by mdmonitor) from acting.
+ [ X"$do_what" = Xstop ] && enbd-pwprog disable
+ [ X"$do_what" = Xstart ] && enbd-pwprog enable
# client configuration found! Check if client binary is
# available, issue a warning when missing.
if [ ! -x "$THECLIENT" ] ; then
@@ -554,6 +584,9 @@
fi
done
done
+ [ X"$isRC-$do_what" = Xyes-start ] && touch /var/lock/subsys/$NAME
+ [ X"$isRC-$do_what" = Xyes-stop ] && rm -f /var/lock/subsys/$NAME
+ true # exit code
;;
restart)
shift
--- enbd-2.4.35/nbd/etc/default/enbd.Orig 2008-12-17 15:52:52.000000000 -0330
+++ enbd-2.4.35/nbd/etc/default/enbd 2008-12-17 15:53:34.000000000 -0330
@@ -10,3 +10,9 @@
# ENBD_SERVER_CONFIG=/etc/enbd-server.conf
# ENBD_CLIENT_CONFIG=/etc/enbd-client.conf
+# Send output of enbd-client/enbd-server to /dev/null? (default: no)
+# OUTPUT_TO_DEV_NULL=yes
+
+# Wait for server to be up (pingable) before starting enbd-client?
+# (default: yes)
+# CLIENT_WAIT_FOR_SERVER=no
Best wishes,
Michael