Re: cluster install

MLH <[email protected]> wrote:
> Jan Schaumann wrote:

> > Actually, we're using rsync

> What might be the chances of letting us see/work on what you have?

Sure, why not.  If you find anything odd or obviously stupid, I'll be
happy to take suggestions. :)

The basic setup is as follows:

Main file server (doppelbock):

(doppelbock) pwd
/usr/local/cluster-maint
(doppelbock) ls
NASTY_HACKS               etc-c009                  etc-c023
aftersync                 etc-c010                  etc-c024
aftersync.not             etc-c011                  etc-c025
aftersync.old             etc-c012                  etc-c026
aftersync.really.old      etc-c013                  etc-c027
beforesync.old            etc-c014                  etc-c028
bin                       etc-c015                  etc-c029
etc-c002                  etc-c016                  etc-c030
etc-c003                  etc-c017                  etc-c031
etc-c004                  etc-c018                  firstboot
etc-c005                  etc-c019                  logs
etc-c006                  etc-c020                  motd
etc-c007                  etc-c021                  rsync-excludes
etc-c008                  etc-c022
(doppelbock) cat NASTY_HACKS
/etc/rc.d/mountcritlocal paxes /var-ro over mfs mounted /var

/usr/local/node needs an up-to-date MAKEDEV
Problem: nodes mfs mount /dev/, so we can't update that version
Hack: hack /sbin/init to not use /dev/MAKEDEV and /dev/MAKEDEV.local
         but fetch it from /etc/ where we placed it (ie /usr/local/node/etc
        on doppelbock!)  In addition, /dev is created too small for our
        purposes, so we increased the size.
        See /usr/src/sbin/init/init.c around line 1405 and 1446
(doppelbock) ls etc-c002
inetd.conf rc.conf
(doppelbock) cat rsync-excludes 
boot
kern
proc
share
var
dev
(doppelbock) ls bin
push.sh tool.sh
(doppelbock)

push.sh and tool.sh are attached -- they are used to push out an update
to all nodes.  The nodes image is in /usr/local/node on the file
server, with various shared directories located on the file server and
symlinked to from the nodes base system (users home directories are set
to /share/home/<username> in /etc/passwd et al).  Installing new
packages is thus as easy as on a single system:  simply go into
/usr/pkgsrc/category/package on doppelbock and run 'make install'.
Done.


(doppelbock) cd /usr/local/node
(doppelbock) ls -l usr/X11R6 usr/obj usr/pkg* usr/src var-ro/db/pkg
lrwxr-xr-x  1 root  wheel  12 Dec 15  2002 usr/X11R6 -> /share/X11R6
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/obj -> /share/obj
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/pkg -> /share/pkg
lrwxr-xr-x  1 root  wheel  13 Dec 15  2002 usr/pkgsrc -> /share/pkgsrc
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/src -> /share/src
lrwxr-xr-x  1 root  wheel  13 Dec 15  2002 var-ro/db/pkg -> /share/db/pkg
(doppelbock) diff -bu etc/rc.d/mountcritlocal /etc/rc.d/mountcritlocal
--- etc/rc.d/mountcritlocal     2003-08-27 18:04:57.000000000 -0400
+++ /etc/rc.d/mountcritlocal    2002-09-08 15:33:33.000000000 -0400
@@ -20,10 +20,6 @@
        #
        mount_critical_filesystems local
 
-       # XXX: local change
-       # /var is mfs mounted over a read-only /var-ro
-       cd /var-ro; pax -rwp e . /var
-
        #       clean up left-over files.
        #       this could include the cleanup of lock files and
        #       /var/run, etc.
        #
(doppelbock) cd /usr/src/sbin/init
(doppelbock) cvs diff -bu
cvs server: Diffing .
Index: init.c
===================================================================
RCS file: /cvsroot/src/sbin/init/init.c,v
retrieving revision 1.61
diff -b -u -r1.61 init.c
--- init.c      2003/08/07 10:04:25     1.61
+++ init.c      2003/10/22 02:08:11
@@ -1401,8 +1401,8 @@
        /* Mount an mfs over /dev so we can create devices */
        switch ((pid = fork())) {
        case 0:
-               (void)execl(INIT_MOUNT_MFS, "mount_mfs", "-i", "192",
-                   "-s", "768", "-b", "4096", "-f", "512", "swap", "/dev",
+               (void)execl(INIT_MOUNT_MFS, "mount_mfs", "-i", "128",
+                   "-s", "10240", "-b", "4096", "-f", "512", "swap", "/dev",
                    NULL);
                _exit(1);
                /*NOTREACHED*/
@@ -1443,7 +1443,11 @@
        switch ((pid = fork())) {
        case 0:
                if (chdir("/dev") == 0)
+#if 0
                        (void)execl(INIT_BSHELL, "sh", "./MAKEDEV", "init",
+#else
+                       (void)execl(INIT_BSHELL, "sh", "/etc/MAKEDEV", "all",
+#endif
                            NULL); 
                _exit(1);
 
(doppelbock)


That's all I can think of right now.  We currently don't have a decent
automatic procedure to drop in new nodes -- the last few times I had to
reinstall a node I actually took the node, opened it up, added a CD
drive, booted off an install CD, disklabel, newfs, nfs mount from
doppelbock and pax things over.  Obviously, I, too, would be interested
in a decent automatic-install tool.

Hope you find this information useful.  If you have ideas of how to do
things more elegantly, please let me know.

-Jan

-- 
Wenn ich tot bin, mir soll mal Einer mit Auferstehung oder so
kommen, ich hau ihm eine rein! (Anonym)

#!/bin/sh
#
# rsync the nodes with /usr/local/node
#set -x

RSYNC_RSH=rsh
BASEDIR=/usr/local
MAINTDIR=${BASEDIR}/cluster-maint
NODESROOT=${BASEDIR}/node

RSYNC="rsync -aS -c"
RSYNC_ROOTFLAGS="-H --delete --exclude-from=${MAINTDIR}/rsync-excludes"

RSYNC_PASSWORD=XXXXXXXXXXXXXXXXXXXXXX
export RSYNC_PASSWORD


ETCFILES="inetd.conf rc.conf"

usage="Usage: $0: "'
	[--help] [--dont] [--keep-as] [--myexcludes=<excludes>] <hostnames|IPs>'

die()
{
	echo "$0: Error $@"
	exit 1
}

while [ $# -gt 0 ]; do
	case $1 in
	--dont)		DONTDOIT="echo" ;;
	--keep-as)	KEEP_AFTERSYNC="YES" ;;
	--myexcludes=*)	MYEXCLUDES="`echo $1 | sed -e 's|--myexcludes=||'`" ;;
	--help)		echo "$usage"; exit ;;
	-h)		echo "$usage"; exit ;;
	-*)		die "$usage" ;;
	*)		HOSTS="$1 ${HOSTS}" ;;
	esac
	shift
done

if [ "${MYEXCLUDES}" != "" ]; then
	for exclude in ${MYEXCLUDES}; do
		EXCLUDES="--exclude ${exclude} ${EXCLUDES}"
	done
fi

for i in ${HOSTS}; do

	# first things first
	#
	echo "New host: $i" >> ${MAINTDIR}/logs/sync.$$.log

	if [ -e ${MAINTDIR}/beforesync ]; then
		${DONTDOIT} ${RSYNC} ${MAINTDIR}/beforesync $i:/tmp >> 	\
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i chmod u+x /tmp/beforesync >> \
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i /tmp/beforesync >>		\
			${MAINTDIR}/logs/sync.$$.log 2>&1
	fi

	# otherwise we can't write anything!
	#
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o rw /
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o rw /usr
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o rw /var-ro

	${DONTDOIT} ${RSYNC} ${RSYNC_ROOTFLAGS}				\
			${EXCLUDES} ${NODESROOT}/ push@$i::slash >>	\
			${MAINTDIR}/logs/sync.$$.log 2>&1

	for file in ${MAINTDIR}/etc-${i}/*; do
		${DONTDOIT} ${RSYNC} ${file} push@$i::slash/etc/${file##*/} >>	\
			${MAINTDIR}/logs/sync.$$.log 2>&1
	done

	# anything else?
	#
	if [ -e ${MAINTDIR}/aftersync ]; then
		${DONTDOIT} ${RSYNC} ${MAINTDIR}/aftersync push@$i::slash/tmp >> \
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i chmod u+x /tmp/aftersync >>	\
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i /tmp/aftersync >>		\
			${MAINTDIR}/logs/sync.$$.log 2>&1 &

		if [ x"${KEEP_AFTERSYNC}" != x"YES" ]; then
			if [ -f ${MAINTDIR}/aftersync.old ]; then
				${DONTDOIT} mv -f ${MAINTDIR}/aftersync.old \
					${MAINTDIR}/aftersync.really.old
			fi
			${DONTDOIT} mv ${MAINTDIR}/aftersync ${MAINTDIR}/aftersync.old
		fi
	fi

	# Update motd
	if [ -e ${MAINTDIR}/motd ]; then
		${DONTDOIT} ${RSYNC} ${MAINTDIR}/motd push@$i::slash/tmp >> \
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i chmod u+x /tmp/motd >>	\
			${MAINTDIR}/logs/sync.$$.log 2>&1
		${DONTDOIT} ${RSYNC_RSH} -n $i /tmp/motd >>		\
			${MAINTDIR}/logs/sync.$$.log 2>&1 &
	fi

	# and back to read-only!
	#
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /var-ro
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /usr
	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /

	# Finally, in case we made any changes to /var-ro, synch the mfs
	# mounted /var
	${DONTDOIT} ${RSYNC_RSH} -n $i rsync -aS /var-ro/ /var

done

#!/bin/sh
#
# loop through the available IPs and perform some of the usual
# tasks
#
#set -x

#defaults
BASEIP="172.16.4."
START=2
END=31
CMD="/sbin/ping -c 1 -w 1 @IP@"
QUIET=no
SYNCALL=no
MAINTDIR=/usr/local/cluster-maint

usage="Usage: $0: "'
	[--start=<num>] [--end=<num>] [--cmd=<cmd>]
	[-h|-help] [--ipbase=<x.x.x.>] [--quiet]
	[--syncall]'

die()
{
	echo "$0: Error: $@"
	exit 1
}

while [ $# -gt 0 ]; do
        case $1 in
	--start=*)	START=`echo $1 | sed -e 's|--start=||'` ;;
	--end=*)	END=`echo $1 | sed -e 's|--end=||'` ;;
	--cmd=*)	CMD="`echo $1 | sed -e 's|^--cmd=||'`" ;;
	--ipbase=*)	BASEIP=`echo $1 | sed -e 's|--ipbase=||' | \
				awk --re-interval '/^((([0-9]{1,2})|((1[0-9]{2})|(2([0-4][0-9]|5[0-5]))))\.){3}/ { print $1; }'`
			if [ -z ${BASEIP} ]; then
				die "Not a valid base-ip of the format: " \
					"'[0-255].[0-255].[0-255].'"
			fi
			;;
        --help)         echo "$usage"; exit ;;
        -h)             echo "$usage"; exit ;;
	--quiet)	QUIET=yes ;;
	--syncall)	SYNCALL=yes ;;
        -*)		echo "$usage"; exit 1 ;;
        esac
        shift
done

if [ "${SYNCALL}" = "yes" ]; then
	MACHINES="`ls -d ${MAINTDIR}/etc-* | sed -e s/.*etc-*//`"
	i=0
	for m in ${MACHINES}; do
		i=$(($i+1))
		if [ $i -lt 10 ]; then
			${MAINTDIR}/bin/push.sh --keep-as $m &
		else
			${MAINTDIR}/bin/push.sh --keep-as $m
			i=0
		fi
	done
else

i=${START}
while [ $i -lt ${END} ]; do
        IP=${BASEIP}${i}
	CMD1=${CMD%@IP@*}
	CMD2=${CMD#*@IP@}
	LOCALCMD="${CMD1} ${IP} ${CMD2}"
	if [ "${QUIET}" = "yes" ]; then
		${LOCALCMD} >/dev/null
	else
		${LOCALCMD}
	fi
	if [ $? -gt 0 ]; then
		RET="Failed!"
	else
		RET="Success!"
	fi
	if [ "${QUIET}" != "yes" ] || [ "${RET}" = "Failed!" ]; then
		echo "${LOCALCMD} : ${RET}"
	fi
        i=$(( $i + 1 ))
done

fi # syncall

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (NetBSD)

iD8DBQE/letdfFtkr68iakwRAhbQAKDVV5N4boI/io2ii3T5glKeZ6eGCwCgo3ww
AOmsgLrdfXA4XquI9CmLMNY=
=agUp
-----END PGP SIGNATURE-----