SkippedSync error - cannot get node2 to drbd sync

Dear OpenSSI team,

This is our 1st post to this list and want to take the opportunity to say
thank you for bringing together a solution which is very impressive indeed
:-)

We hope to have a 2-node, Fedora3 OpenSSI 1.9dev with drbd root failover HA
cluster running shortly in production, supporting services for 1000 users.

We have followed the instructions to the best of our ability, per:
http://openssi.org/cgi-bin/view?page=docs2/1.9/fedora/INSTALL.html
http://wiki.openssi.org/go/FC2_DRBD_Root_Failover_HOWTO

The only issue we are having is that our second node, configured for root
failover with a drbd root partition, will not sync, ie:

The following is only with our 1st node powered up...

[root@cluster1 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by root@cluster1, 2007-01-18 08:12:12
0: cs:WFConnection st:Primary/Unknown ld:Consistent
ns:0 nr:0 dw:127536 dr:231025 al:7 bm:173 lo:0 pe:0 ua:0 ap:0

And the following is with the 2nd node powered up...

[root@cluster1 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by root@cluster1, 2007-01-18 08:12:12
0: cs:SkippedSyncS st:Primary/Secondary ld:Consistent
ns:360 nr:0 dw:19148 dr:56901 al:0 bm:182 lo:0 pe:0 ua:0 ap:0

Note the "SkippedSyncS" status.

The only reference we can find to this status state, is at the drbd website:
http://www.linux-ha.org/DRBD/FAQ#head-988086376cbd00bdcc9c9d199317af5f7550c5c1
SkippedSync{S,T} you should never see this. "Developers only"

Help or pointers on how to get past this issue will be very much appreciated
:-)

The instructions / howto we have pieced together is below...hopefully we
have made a simple mistake somewhere in there.....

===============================================================================
OPENSSI - Installing and getting a cluster of 2 PC's working in HA
failover...

Supported OS - Fedora Core 3 with OpenSSI 1.9
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Also, FC3 support for SATA is poor - so use PATA disks!

READ THIS page to get a feel for functionality and how it all works...
http://openssi.org/cgi-bin/view?page=docs2/1.9/Introduction-to-SSI
===============================================================================


===============================================================================
Prepping the 1st PC with our base OS installation
-------------------------------------------------------------------------------
160G PATA disk in each PC
ASUS onboard 10/100 NIC (eth1 for external connectivity)
Netgear 1Gb NIC (eht0 for frossover cable interconnect)

Make sure both PC's are configured to PXE boot (eth0) in BIOS as the 1st
device
in the Boot sequence. They will be connected via Ethernet NIC's that see
eachother on the same private subnet (eg, 10.10.10.x/24) via a cross-over
cable.

Build up PC ONE (our primary cluster 1 node) standard PC with FC3 install...

Note WORKAROUND ISSUE - ASUS M/B has unsupported NIC in basic FC3 (Realtek
RTL8201CL Via 10/100) and to access it, will need kernal arguement acpi=off
Therefore, to avoid no m/b eth, MUST at the KERNEL line at install, kick off
the install with:
boot: linux acpi=off

* 'Custom Install' profile package install
* Create partitions in the following order:
/boot			100Mb				ext3
/				151000Mb			ext3
<swap>			1024Mb
/drbd-index		MaxAvail(494Mb)		ext3	<space for DRBD meta-device>
* When finished, it will look like this:
/dev/hda1		/boot			ext3		102
/dev/hda2		/				ext3		151002
/dev/hda3						Swap		1028
/dev/hda4						Extended	494
/dev/hda5	/drbd-index		ext3		494
* grub boot loader onto the MBR of /dev/hda
* Fixed IP addresses as follows
eth0-CLUSTER INTERCONNECT (1GB NIC card)
IP		10.10.10.1
Netmask	255.255.255.0
eth1-EXTERNAL NIC (100Mb onboard NIC) <settings below to suit LAN>
IP		192.168.0.11
Netmask	255.255.255.0
Hostname & Misc <gw & dns settings below to suit LAN>
Host	cluster1
Gateway	192.168.0.1
DNS		192.168.0.181
NB TIP: if you get eth0 and eth1 the wrong way around, after install, go
into
gui network tool, and swap ip address information
* no firewall
* SELinux off
* Packages:
deselect	Office/Productivity
deselect	Sound & Video
select		Server Configuration Tools
select		Web Server
select		Mail Server
select		Development Tools
deselect	Printing Support

Once the system is all rebooted and completed the install process (ok to
enable
network time protocol), you are ready to continue....
===============================================================================


===============================================================================
PRIMARY NODE configuration
-------------------------------------------------------------------------------
Root filesystem failover is one of the features of OpenSSI. You either need
to
use DRBD or shared disk hardware for this to work. If you choose to use
DRBD,
you should also refer to README.drbd. If you use shared disk hardware, then
the root filesystem must be installed on the shared disk.
WE WILL BE USING DRDB, as we do NOT have SHARED STORAGE.

Firstly, we will not be running X on the host cluster, so disable it
starting
at boot time:
# vi /etc/inittab
And set '3' in the following line...
id:3:initdefault:

Copy the "openssi-fc3-1.9.2.i686.tar.gz" file to cluster1 and put it in the
following, created directory:
# mkdir /usr/src/openssi

Then extract it:
# cd /usr/src/openssi/
# tar xzf openssi-fc3-1.9.2.i686.tar.gz

Next, run the install script:
# cd openssi-fc3-1.9.2.i686/
# ./install

After it installs your packages, it will ask you a few questions about how
you
want to configure your cluster and your first node:
Node Number: 1
Network card: eth0	(note TIP above, and make sure this is the interconnect
NIC)
Network Boot Protocol: PXE
(did Etherboot due to 2nd nic, but we will not boot from it as a primary)
Clustername (should resolve to CVIP address): cluster1  <should be in
/etc/hosts>
Root system failover: YES <we will be using DRBD>

You will get a screen that says: "The following configuration has been
entered":
Node number:            1
NIC for interconnect:   eth0
IP address:             10.10.10.1
Network hardware addr:  00:18:4D:6F:4E:E4
Network boot protocol:  Etherboot
Local boot device:      /dev/hda1
Clustername:            cluster1
Root failover:          Yes
Potential initnode:     Yes
NFS support:            yes
Enter "W" to (W)rite new configuration

It will now run a short compile/configure script, that should end in output
similar to the following...

Installing OpenSSI kernel:
Preparing...                ###########################################
[100%]
1:kernel-ssi-smp         ########################################### [100%]
unable to stat UUID=1e9e9886-3bc5-4bbe-b6b1-2e696e4c74e0: 2
unable to stat UUID=1e9e9886-3bc5-4bbe-b6b1-2e696e4c74e0: 2
...snip...

Wait a while until it is finished, before you can reboot to use the features
of
OpenSSI in your nice new kernel.

Would you like to reboot now (y/n) [n]:
Looking good - so enter 'y' to reboot!  UNLESS you need the ACPI
workaround....

PASSING CUSTOM ARGUEMENTS TO THE KERNEL
---------------------------------------
If kernel will not detect NIC, need to pass acpi=off to the nodes as they
join.

Nodes that PXE boot get their kernel arguments from the following file that
is
read everytime a node PXE boots: /tftpboot/pxelinux.cfg/default

Nodes that Etherboot get their kernel arguments from the /tftpboot/combined
image, which contains the kernel, kernel arguments and initrd for an
etherbooting node.

To modify the kernel arguments, edit the --append option to the mkelf-linux
command at the bottom of the /sbin/ssi-ksync-network script (running the
'openssi-config-node' script to add/remove nodes re-creates the boot images
in the /tftpboot/ directory (with our new kernel arguements).

We will do both....

In another terminal window is a handy way to do this...
# vi /tftpboot/pxelinux.cfg/default

Change the line:
append initrd=initrd ro
To:
append initrd=initrd ro acpi=off

# vi /sbin/ssi-ksync-network

Change (2nd last line in the file):
$kernel $initrd >/tftpboot/combined
To:
--append='acpi=off' $kernel $initrd >/tftpboot/combined

And then reboot. The PXE/Etherboot changes will come into effect later, when
we need them to :-)

All going well you are now in your nice new kernel. What we will do, is
DELETE
the original, non-SSI kernels from the system as we will not be using
them...
...and if we tried to boot from them, they would just CLOBBER our partitions
and not know how to read from our DRBD devices anyway. Also, this supresses
errors generated when adding/deleting nodes.
# rpm -e kernel kernel-smp

REBOOT
(when I didn't do this, I could not get the new node 2 to join until I did)
===============================================================================


===============================================================================
SECONDARY FAILOVER NODE #2
-------------------------------------------------------------------------------
Ideally, this node is added to the OpenSSI cluster using network booting
(PXE).
However, Etherboot works just fine. The DHCP server will be automatically
configured and started to allow the new node(s) to join the cluster.

-------------------------
<No PXE - Need Etherboot>
However, if the NIC that will serve as the interconnect is not PXE, you will
need to Etherboot it (only once, as our #2 failover node will normally
boot from its hard disk.

Find out what NIC you have, with:
# lspci -v

ONBOARD
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev
7c)
Subsystem: ASUSTeK Computer Inc.: Unknown device 80a7
Flags: bus master, medium devsel, latency 64, IRQ 185
I/O ports at d400 [size=256]
Memory at febff400 (32-bit, non-prefetchable) [size=256]
Capabilities: [40] Power Management version 2

1GB NIC
05:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
Subsystem: Netgear: Unknown device 311a
Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 153
I/O ports at c800 [size=256]
Memory at feaffc00 (32-bit, non-prefetchable) [size=256]
Expansion ROM at feac0000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2

The 1st one is the external NIC - eth1
The 2nd one is the interconnect NIC - eth0

We want the interconnect one (RTL-8169), so with this information proceed to
the Etherboot website...

Go to the www.rom-o-matic.net website, to the latest production release
page, ie:
http://www.rom-o-matic.net/5.4.2/

In the Choose NIC/ROM type, we need to select "r8169:r8169" for our 1GB NIC,
output format as (.iso), then click on Customise, and set as follows:
Ask boot: [  ]  (empty, so it immediately Etherboots)
Boot First: BOOT_NIC
Boot Second: BOOT_NOTHING
Boot Third: BOOT_NOTHING
[tick] BAR_PROGRESS
(don't display a page ful of dots while downloading img, rotate a | symbol)
Then click on [Get ROM]

Burn it to CD-ROM, and place into the CD tray of cluster2 (ie, Failover
Node #2). (Note - we will not be leaving it there, once we have cluster2's
mirrored HD built, cluster2 will boot off its HD directly).

</No PXE - Need Etherboot>
--------------------------

Power up the computer and get it to start its network boot. Wait until it
gets
to the point where it is querying the DHCP server for boot/ip info, then
shut it down again. Out cluster1 will not have cached its request and MAC
address.

On the first node (or any node already in the cluster), run:
# openssi-config-node

and select "Add a new node". It will ask you a few questions about how you
want
to configure your new node:
Verbose: y
Node Number: 2
(This is where you can select from the list, as the cluster2 interconnect
NIC will show what was cached)
Network card: eth0
Scan for IP: n
Static IP: 10.10.10.2
Network Boot Protocol: PXE
(did Etherboot due to 2nd nic, but we will not boot from it as a primary)
Nodename: cluster2  <should be in /etc/hosts>
Root system failover: YES <we will be using DRBD>
Save the configuration.

Now go to the 2nd node, power it on, and then its network boot will go
straight into the cluster via network boot.....

The program will now do all the work to admit the node into the cluster.
Wait
for the new node to join.

A "nodeup" message on the first node's console will indicate this. You can
confirm its membership with the cluster command:
# cluster -V

The following steps tell you how to configure the new node's hardware,
including
its swap space, local boot device, and external NICs.

--------------------
OPTION 1 (preferred)

Copy the partition scheme from the cluster1 to cluster2 (assumes same HD)
which will save a lot of fdisk mucking around :-)
# sfdisk -d /dev/hda | onnode 2 sfdisk /dev/hda
--------------------
OPTION 2 (only if issues with #1 above - but it should have worked!)

Follow the steps below to manually create partitions...

Configure the new node with one or more swap devices using fdisk (or a
similar tool) and mkswap:
# onnode 2 fdisk /dev/hda

Partition disk, as follows (make sure delete existing paritions 1st with the
'd'
command) then createas follows:

BOOT
n - new
p - primary
1 - partition number
default - first cylinder
+100M - size of partition <size same as hd on cluster1>
a - toggle bootable flag
1 - partition number

ROOT
n - new
p - primary
2 - partition number
default - first cylinder
+151000M - size of partition <size same as hd on cluster1>

SWAP
n - new
p - primary
3 - partition number
default - first cylinder
+1028M - size of partition <size same as hd on cluster1>
t - change system type
3 - partition number
82 - linux swap

EXTENDED
n - new
e - extended
default - first cylinder
default - size of partition = all remaining space

DRBD-INDEX
n - new
default - first cylinder
+149M - size of partition <size same as hd on cluster1>

SAVE & EXIT
w - write and save new partition table
-------------------

Format the swap partition:
# onnode 2 mkswap /dev/hda3
Setting up swapspace version 1, size = 1077506 kB

Add the device name(s) to the file /etc/fstab, as documented in README.fstab
# vi /etc/fstab

Change:
LABEL=SWAP-hda3 swap    swap    defaults,node=1 0       0
To:
#LABEL=SWAP-hda3 swap    swap    defaults,node=1 0       0
LABEL=SWAP-hda3 swap    swap    defaults,node=* 0       0

Activate the swap device(s) with the swapon command:
# onnode 2 swapon /dev/hda3

Swap space is not particularly SSI, with each node providing its own
swap space. To determine which nodes are using which swap, you can execute
the following:
# onall swapon -s

Now, as we have enabled root failover you MUST configure a local boot device
on the new node. It is highly recommended that the boot device have the
same name as the first node's boot device. Remember that we assumed at
the beginning of these instructions that the first node's boot device is
located on the first partition of the first drive (i.e. /dev/hda1).

Format the boot device with an ordinary Linux filesystem, such as ext3:
# onnode 2 mkfs.ext3 /dev/hda1

Now run ssi-chnode anywhere in the cluster (no need to use onnode with this
command). Select the new node, enter its local boot device name, and
ssi-chnode will copy over the necessary files. Commands to do this are:
# ssi-chnode
Node: 2
PXE: P
(note, use E if using Etherboot)
Boot device: /dev/hda1

Finally, you need to manually install a GRUB boot block on the new node:
# onnode 2 grub --device-map=/boot/grub/device.map
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

This will ensure that the boot loader is installed on hda of Node 2, (so
that
it is able to boot). The failover node will not boot from the network
anymore. It will boot from its own hard drive, and depending on whether it
finds a root node on the cluster, it will either join the root node or
become the root node.

Eject the CD in Node 2 (cluster2) so that it will boot from the hard drive,
or
if you have been using PXE, change the boot sequence in BIOS. Now, reboot
cluster2 with the following command:
# clusternode_shutdown -r -A 5 -t 5 -N 2 now

Reboot and watch the node boot and joins the SSI cluster :-)
===============================================================================


===============================================================================
DRBD - highly available (HA) data storage, data protection, data redundancy,
and data replication (sub-second failover performance)
-------------------------------------------------------------------------------
http://www.openssi.org/cgi-bin/view?pf=1&page=docs2/1.9/fedora/README.drbd

If you do not have shared disk hardware (ie fibre channel, SAN), an
alternate
solution for HA filesystems is provided by the Distributed Replicated Block
Device (DRBD) project.

Copy "drbd-ssi-1.9.2.tar.gz" & "kernel-ssi-2.6.10-ssi_3devel.src.rpm" to
cluster1 and put in the following, created directory:
# mkdir /usr/src/openssi-drbd
# cd /usr/src/openssi-drbd

Install the kernel sources, so we can compile against its headers
# rpm -Uvh kernel-ssi-2.6.10-ssi_3devel.src.rpm
# rpmbuild -ba --target=i686 /usr/src/redhat/SPECS/kernel-2.6.spec

The above command takes a LONG TIME (about 1.5 to 2hrs)

The sources end up in a weird spot, so make it accessible in a more
conventional fashion with...
# ln -s /usr/src/redhat/BUILD/kernel-ssi-2.6.10/linux /usr/src/linux

Next, run the install script for the SSI patched DRBD:
# cd /usr/src/openssi-drbd/
# tar xzf drbd-ssi-1.9.2.tar.gz
# cd drbd-ssi-1.9.2/drbd/

Because we have not installed the 'docbook tools', we will not be able to
compile the 'documentation' subdirectory. Trouble is, that that directory
is in the build script, so we will edit the Makefile so the 'documentation'
directory is ignored.
# vi Makefile

Change:
SUBDIRS     = user scripts documentation drbd #testing #benchmark
ALLSUBDIRS  = user scripts benchmark documentation drbd testing
To:
SUBDIRS     = user scripts drbd #documentation #testing #benchmark
ALLSUBDIRS  = user scripts benchmark drbd testing #documentation

# make KDIR=/usr/src/linux

Make sure it has a squeaky clean install, and that it has produced our
loadable
kernel module 'drbd.ko' as follows:
/usr/src/openssi-drbd/drbd-ssi-1.9.2/drbd/drbd/drbd.ko

# make install

It should have installed this into the following location, following a
squeaky
clean completion of the above install command:
/lib/modules/2.6.10-ssi_3develsmp/kernel/drivers/block/drbd.ko

# vi /etc/drbd.conf
Make the following configuration changes:
Change:
# resource root {
resource r0 {
To:
resource root {
# resource r0 {

Change:
# wfc-timeout  0;
To:
wfc-timeout  0;

OPTIONAL - change (under the 'syncer' block) the rate as follows:
rate 10M;		(for 100Mbit Ethernet, leaves 20% headroom capacity on network)
rate 100M;		(for GigaBit Ethernet, leaves 20% headroom capacity on network)

Change "on amd {" to "on cluster1 {"
Change "disk /dev/hde5" to "disk /dev/hda2"
nodenum    1;
Change "address 192.168.22.11:7788;" to "address 10.10.10.1:7788;" (the
interconnect IP)
Change "meta-disk internal" to "meta-disk /dev/hda5[0];"

Change "on alf {" to "on cluster2 {"
Change "disk /dev/hdc5" to "disk /dev/hda2"
nodenum    2;
Change "address 192.168.22.12:7788;" to "address 10.10.10.2:7788;" (the
interconnect IP)
Change "meta-disk internal" to "meta-disk /dev/hda5[0];"

Comment out the remainder of the file (resource r1, r2 & r3 blocks), which
are
additional device definitions we won't be using. Use the 'skip' block
syntax, ie:

skip {
TEXT WITHIN THE BRACES IS NOW COMMENTED OUT AS A CHUNK
}

Now, when we did the initial Fedora install, we created a /drbd-index
partition,
as a holding area to get the DRBD area created. Time to finish this process
so it can store DRBD meta data.
# vi /etc/fstab

Comment out the line, so that it reads:
#LABEL=/drbd-index       /drbd-index     ext3    defaults,node=1 1       2

# umount /drbd-index
# rmdir /drbd-index

# dd if=/dev/zero of=/dev/hda5

The above will take a minute or so, then we can continue with:
# modprobe drbd
# lsmod | grep drbd

WARNING: IF YOU DO NOT GET THIS MODULE LOADING OK - DO NOT CONTINUE &
REBOOT,
OR YOU WILL NOT HAVE AN OPERABLE SYSTEM ANYMORE!!!! If you followed the
'Makefile' documentation directory tweak above, you should have no issues.

# mkinitrd --drbd --cfs -f /boot/initrd-`uname -r`.img `uname -r`
# vi /boot/grub/grub.conf

Add "drdb" to the following line, so it reads:
kernel /vmlinuz-2.6.10-ssi_3develsmp ro drbd
root=UUID=b3111a92-fdc4-459c-9867-94d0c75bd9a9 acpi=off rhgb quiet

# ssi-ksync
# vi /etc/fstab

Replace the "UUID=..." part to "/dev/drbd0", ie so we get:
#UUID=b3111a92-fdc4-459c-9867-94d0c75bd9a9       /       ext3   
chard,defaults,node=1   1       1
/dev/drbd0       /       ext3    chard,defaults,node=1   1       1

Now REBOOT (and leave cluster2 off)!

One-off-procedure: Configure cluster1 drbd
------------------------------------------
You will see some messages about DRBD, let it continue to boot until it
begins
counting on the screen. This is counting up the seconds it will wait for a
peer node to appear. It will count up to 120 seconds. Look for the following
line:
To abort waiting enter 'yes' [ nn]:

Type "yes" to abort the counting.

The system will attempt to mount the root (/) filesystem but will fail
because
it is not a primary node. Then it will display:
Press Enter within 10 seconds for shell.

MAKE SURE you HIT <Enter> within 10 seconds ! And then continue with...

# drbdadm -s /bin/drbdsetup -- --do-what-I-say primary root
# exit

When the system boots up, it will wait for 120 sec...just let it do this.

Now the root (/) partition should be picked up by the drbd-enabled ramdisk
and booted. You are now running with the root as the DRBD primary.
</one-off-procedure>

When completed booting up, login in and type:
# cat /proc/drbd

This will show the status of the DRBD system, the command can also be used
to
show whether other nodes have synced up properly with the main image. You
should see:
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by root@cluster1, 2007-01-18 08:12:12
0: cs:WFConnection st:Primary/Unknown ld:Consistent
ns:0 nr:0 dw:127536 dr:231025 al:7 bm:173 lo:0 pe:0 ua:0 ap:0

You can see the waiting for connection (WFConnection) to our secondary.
TIP for later - DRBD is ready for failover once all drbd devices show
"ld:Consistent".

Bring up Node2 and sync
-----------------------
Boot up cluster2, it should boot via its hard disk and then detect that it
is
secondary in the bootup - as it will detect cluster1 via the interconnect.

You can "cat /proc/drbd" to determine the progress of the syncing. DRBD is
ready
for failover once all drbd devices show "ld:Consistent", and until then, you
should see a progress info layout similar to...

# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed:  6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
PROBLEM_PROBLEM_PROBLEM - I GET THE FOLLOWING (SKIPPED THE SYNC #$*@&#!)

version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by root@cluster1, 2007-01-18 08:12:12
0: cs:SkippedSyncS st:Primary/Secondary ld:Consistent
ns:360 nr:0 dw:19148 dr:56901 al:0 bm:182 lo:0 pe:0 ua:0 ap:0
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

===============================================================================


mtel.net.au webmail from Mach Technology http://mach.com.au


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV