Heartbeat not reclaiming IP properly

Hi,

I'm in new territory playing with Ultra Monkey, and have come
across a problem with heartbeat. I can't find anything similar to
this on the web, but maybe I'm not looking in the right places!

Topology is basically "High Capacity High Availability and Load
Balancing"[1]. The actual network segment is fairly resilient and
split over two buildings, but to all intents and purposes all
machines are plugged into the same switch. Currently this is all
for testing, so the "machines" are all user-mode-linux bridged
onto the network. There is no secondary link between the directors
at the present. There are:

lb1, lb2      ... directors
vw1, vw2, vw3 ... web servers

Using just lb1 and ldirectord works fine, load balancing between
the web servers.

I have set up heartbeat, and can reproducibly cause the following
to happen:

1. Start heartbeat on lb1. This brings up the virtual IP /
   ldirectord after a few seconds as expected.

   1a: wait until vIP is up.

2. Start heartbeat on lb2. This sees that lb1 is up, and starts
   as backup.

   2a: wait until lb2 has stabilised (30 seconds ish)

3. Stop (init script) heartbeat on lb1. This hands control over to
   lb2 correctly, and things continue as expected running through lb2.

4. Start heartbeat on lb1 again. Lb2 seems to notice that lb1 is
   back, but does not hand control over because auto_failback is
   off.

   4a: wait only about 5-10 seconds, after the first few log
   messages have arrived on lb1.

5. Stop heartbeat on lb2.

We now have, seemingly, lb2 handing control back to lb1, after
which lb2 stops. However, lb1 never brings the vIP back up again,
even though it seems to think that it is now the master. If you
then do:

6. Start heartbeat on lb2. Seems to register as a backup with lb1
   (but of course nothing is working now).

   6a: wait for things to stabilise (30s or so)

7. Stop heartbeat on lb1. This hangs with "WARN: Shutdown delayed
   until current resource activity finishes." in the logs.

The only way of stopping heartbeat on lb1 is with kill.

After just another quick test, it seems that this situation occurs
whenever lb2 is stopped while lb1 is starting up (within the first
30 seconds ish).

This does _not_ happen the other way round, i.e. if lb1 is stopped
which lb2 is starting up. Then lb2 takes over the IP pretty
quickly.

The only thing that seems biased towards lb1 is the haresources
file:

----------8<--------------------------
lb1.test        \
        ldirectord::ldirectord.cf \
        LVSSyncDaemonSwap::master \
        IPaddr2::143.210.16.173/24/eth0/143.210.16.255
----------8<--------------------------

Waiting until heartbeat has completed starting up after stopping
the opposite heartbeat seems to work fine; it's only while
starting the problem occurs.

Is this a known problem, or have I made silly mistakes in my
setup? ;-)

I can post configs and logs if needed.

Thanks!

Matthew


[1] http://www.ultramonkey.org/3/topologies/hc-ha-lb-overview.html

-- 
Matthew Newton <[email protected]>

UNIX and e-mail Systems Administrator, Network Support Section,
Computer Centre, University of Leicester,
Leicester LE1 7RH, United Kingdom


-- 
Ultra Monkey - http://www.ultramonkey.org/
To UNSUBSCRIBE, email to [email protected], with a body:
unsubscribe ultramonkey-users [email protected]
where "[email protected]" is YOUR email address.