Re: How to automatically add a nbd to a raid1?

On Thursday 18 December 2008 20:03, Peter Breuer wrote:
...
> More obviously, the interaction between pwprog and the init script needs
> considerable thinking about.  At the moment you're disabling pwprog (as
> run by mdadm --monitor) while the init script runs.  The init script
> effectively runs for quite a while since it starts daemon scripts which
> hang around waiting for the other end of a connection to come up before
> starting the corresponding daemons.
>
> I'm a little concerned that there may be a multiplicity of waiting init
> script slave daemon scripts, and that they may compete for or confuse
> the status of the lockout file that lets mdadm start working, or simply
> compete with or confuse the pwprog script run by mdadm --monitor.


Yes - the disable/enable should be outside the while loop that
reads the config file; it can be moved to the top of the start/stop case:
  case $do_what in
  start|stop)
    if [ X"$to_what" = X -o X"$to_what" = Xclient ] && [ X"$to_which" = X ]
    then
      # Allow/prevent pwprog (run by mdmonitor) from acting.
      [ X"$do_what" = Xstop ] && enbd-pwprog disable
      [ X"$do_what" = Xstart ] && enbd-pwprog enable
    fi
    (
      if [ "$SERVER_CONFIG" != "$CLIENT_CONFIG" ]; then


>
> There's only one pwprog disable file. It will be removed by the init
> script running start, which allows pwprog to run thereafter.  It will
> be established by the init script running stop, which turns off
> the pwprog run from mdadm. The latter is clearly right. Is the start
> action right?
>
> If the init script can't start all its daemons at once, that's when it
> starts its watchers. Hopefully mdadm won't know about the missing enbd
> devices at this point, so it won't ever tell pwprog to try and recover
> enbd devices (which don't yet exist) into raid arrays. That saves
> pwprog from competing with the init script. But it leaves a hole ..
> the init script watchers will start the enbd connections. Then nothing
> .. mdadm won't act. One needs to use the udev notice or the init script
> itself to put the new enbd devices into raid arrays. After that one can
> leave everything to pwprog.
>
> So I don't see there's anything wrong with your approach.  Nevertheless,
> I feel bad about something ..  it seems to me that there ought to be one
> lockout file for pwprog per enbd device, not one lockout file full stop.
> But I don't know why I feel that.  Perhaps it'll come to me ..  tomorrow
> I have to be on a train.


Also felt that it could be done on a device by device level - means
the enable/disable would go back inside the while loop.  Can make
the needed changes to the pwprog and init scripts if you'd like.
If one wanted a "disable all" option for pwprog, it would mean pwprog
would have to parse the config file (straight forward, but means the
parsing is done in two places).


> ...

> It seems to me that you are basically looking for a mechanism to stop
> pwprog acting in opposition to an orderly shutdown procedure.
> 
> It's not a bad idea to have a control file .. but it feels clumsy.
> 
> On the plus side it is a control mechanism that can be used in all
> circumstances. 
> 
> On the minus side, the intended use is very init-sequence specific, so
> why go the file route?
> 
> One could simply shut md down before shutting enbd down.  Then there
> would be no interference from mdadm/pwprog when enbd shutdown goes on
> its killing spree...  not so?

It may be that not all md devices are using nd devices, so stopping them
(especially if the root file system is on one) may not be possible.
Could try stopping only those that use nd devices, but again, it may
be that the file system is needed (no reason why the root file system
couldn't be mirrored to another machine), so the best one could do is
fail/remove the nd partition, but that would trigger the pwprog...

As it happens, have written a script that does stop (or fail/remove) md
devices with the intent that it be run before the "init/enbd stop" script
when shutting down :-)  It does the "pwprog disable" / enable to avoid the
above problem.  Wasn't sure if this was generally useful (doesn't deal with
other potential uses of enbd devices, such as in lvm volumes), so didn't send
it before.  Makes sense to put the general disable/enable in this script,
but also makes sense to put individual device disable/enabling in the init
script (since it is designed to be able to start/stop individual devices).
Have included the script at the end.


> I would have alternatively suggested for consideration (not terribly
> seriously) that pwprog suicide when runlevel is going to 0 or 6!  It can
> ask /sbin/runlevel.  Or it can examine the init links directly to figure
> out if enbd is going up or down in this runlevel transition, in general.

Yes - this is possible; means pwprog would have to know the details
of the init system (something that seems to vary a bit: Debian style
vs Redhat style vs Gentoo style vs ...).  Is nice to have such things
confined to the init script itself!


Best wishes,
Michael

/etc/rc.d/init.d/ script to cleanly shutdown md devices before enbd clients
go away:

#!/bin/sh
#
# enbd-md	look for md (raid) devices using enbd devices and unmount
#		the file systems and stop the raids in preparation for
#		stopping the enbd service.
#
# chkconfig: 345 75 25
# description: Prepare for the shutting down of enbd (myTel).

# Source function library.
. /etc/rc.d/init.d/functions

prog=enbd-md

verbose()
{
    echo $0: "$@"
    :
}

logError()
{
    echo "$0:" "$@" 1>&2
    logger -p daemon.error -t "$prog" -- "$@"
}

findMDs()
{
    awk '/^md[0-9][0-9]* : (active|inactive) (raid[0-9]) / {
	    md = $1;
	    level = $4;
	    for (i = 5; i < NF; ++i) {
		dev = $i;
		sub(/\[.*/, "", dev);
		isFailed = $i ~ /\(F\)/ ? "failed" : "ok";
		if (dev ~ /^nd[a-p]$/) {
		    print md, level, isFailed, dev;
		}
	    }
	}' /proc/mdstat
}

umountFSOnDev()
{
    typeset dev=$1
    if grep "^[ 	]*$dev[ 	]" /proc/mounts > /dev/null; then
	typeset sig=TERM cnt=0
	for cnt in 0 1 2; do
	    verbose "umountFSOnDev: Attempting to umount $dev"
	    if umount $dev; then
		verbose "umountFSOnDev: umount $dev succeeded"
		return 0
	    fi
	    verbose "umountFSOnDev: trying to signal ($sig) processes using $dev"
	    /sbin/fuser -k -m -$sig $dev
	    sleep 4
	    sig=KILL
	done
	# failed to unmount
	return 1
    else
	verbose "umountFSOnDev: nothing mounted on $dev"
	# not mounted
	return 0
    fi
}

doStart()
{
    echo -n "Starting $prog: "
    touch /var/lock/subsys/$prog
    # Allow pwprog to work again
    enbd-pwprog enable
    echo
    return 0
}

doStop()
{
    typeset mds=`findMDs`
    typeset md level nd
    typeset anyErrors=0

    echo -n "Stopping $prog: "

    # Toast any running pwprog processes and prevent new ones
    enbd-pwprog disable

    set -- $mds
    while [ $# -gt 0 ]; do
	md=/dev/$1
	level=$2
	isFailed=$3
	nd=/dev/$4
	shift 4 || {
	    logError "doStop: internal error: invalid md list (n args: $*)"
	    exit 1
	}
	verbose "doStop: checking $level on $md which uses $nd"
	if [ X"$level" = Xraid1 ] ; then
	    if umountFSOnDev $md; then
		verbose "doStop: $md: attempting to stop raid"
		if mdadm --stop $md; then
		    verbose "doStop: $md: stop succeeded"
		    continue
		fi
		logError "doStop: $md: unable to stop raid"
	    fi
	    verbose "doStop: $md: attempting to fail $nd"
	    if [ X"$isFailed" != Xfailed ] && ! mdadm --fail $md $nd; then
		logError "doStop: $md: unable to mark $nd as failed"
		anyErrors=1
	    else
		verbose "doStop: $md: fail succeeded"
		sleep 2
	    fi
	    verbose "doStop: $md: attempting to remove $nd"
	    if ! mdadm --remove $md $nd; then
		logError "doStop: $md: unable to remove $nd"
		anyErrors=1
	    else
		verbose "doStop: $md: remove succeeded"
	    fi
	else
	    logError "doStop: can't handle level $level (for $md, $nd)"
	    anyErrors=1
	    continue
	fi
    done
    rm -f /var/lock/subsys/$prog
    echo
    return $anyErrors
}

exitCode=0
case "$1" in
  start)
	doStart
	exitCode=$?
        ;;
  stop)
	doStop
	exitCode=$?
        ;;
  *)
        echo "Usage: enbd-md {start|stop}"
        exitCode=1
esac

exit $exitCode