Re: Heartbeat not reclaiming IP properly

In article <[email protected]> you wrote:
> Hi,
> 
> I'm in new territory playing with Ultra Monkey, and have come
> across a problem with heartbeat. I can't find anything similar to
> this on the web, but maybe I'm not looking in the right places!
> 
> Topology is basically "High Capacity High Availability and Load
> Balancing"[1]. The actual network segment is fairly resilient and
> split over two buildings, but to all intents and purposes all
> machines are plugged into the same switch. Currently this is all
> for testing, so the "machines" are all user-mode-linux bridged
> onto the network. There is no secondary link between the directors
> at the present. There are:
> 
> lb1, lb2      ... directors
> vw1, vw2, vw3 ... web servers
> 
> Using just lb1 and ldirectord works fine, load balancing between
> the web servers.
> 
> I have set up heartbeat, and can reproducibly cause the following
> to happen:
> 
> 1. Start heartbeat on lb1. This brings up the virtual IP /
>   ldirectord after a few seconds as expected.
> 
>   1a: wait until vIP is up.
> 
> 2. Start heartbeat on lb2. This sees that lb1 is up, and starts
>   as backup.
> 
>   2a: wait until lb2 has stabilised (30 seconds ish)
> 
> 3. Stop (init script) heartbeat on lb1. This hands control over to
>   lb2 correctly, and things continue as expected running through lb2.
> 
> 4. Start heartbeat on lb1 again. Lb2 seems to notice that lb1 is
>   back, but does not hand control over because auto_failback is
>   off.
> 
>   4a: wait only about 5-10 seconds, after the first few log
>   messages have arrived on lb1.
> 
> 5. Stop heartbeat on lb2.
> 
> We now have, seemingly, lb2 handing control back to lb1, after
> which lb2 stops. However, lb1 never brings the vIP back up again,
> even though it seems to think that it is now the master. If you
> then do:

What does the heartbeat log looklike on lb1 at this time.
Does it claim that it is taking the IP back?

> 6. Start heartbeat on lb2. Seems to register as a backup with lb1
>   (but of course nothing is working now).
> 
>   6a: wait for things to stabilise (30s or so)
> 
> 7. Stop heartbeat on lb1. This hangs with "WARN: Shutdown delayed
>   until current resource activity finishes." in the logs.
> 
> The only way of stopping heartbeat on lb1 is with kill.
> 
> After just another quick test, it seems that this situation occurs
> whenever lb2 is stopped while lb1 is starting up (within the first
> 30 seconds ish).
> 
> This does _not_ happen the other way round, i.e. if lb1 is stopped
> which lb2 is starting up. Then lb2 takes over the IP pretty
> quickly.
> 
> The only thing that seems biased towards lb1 is the haresources
> file:
> 
> ----------8<--------------------------
> lb1.test        \
>        ldirectord::ldirectord.cf \
>        LVSSyncDaemonSwap::master \
>        IPaddr2::143.210.16.173/24/eth0/143.210.16.255
> ----------8<--------------------------
> 
> Waiting until heartbeat has completed starting up after stopping
> the opposite heartbeat seems to work fine; it's only while
> starting the problem occurs.

This sounds like a split-brain kind of problem. Especially
as you only have one communication channel between lb1 and lb2 (right?).
Are you using ipfail?  That would probably help.

Also, I would be interested to know if changing the value of
auto_failback affects this behaviour.

> Is this a known problem, or have I made silly mistakes in my
> setup? ;-)
>
> I can post configs and logs if needed.

Yes, please do.

-- 
Horms                                           
H: http://www.vergenet.net/~horms/          W: http://www.valinux.co.jp/en/


-- 
Ultra Monkey - http://www.ultramonkey.org/
To UNSUBSCRIBE, email to [email protected], with a body:
unsubscribe ultramonkey-users [email protected]
where "[email protected]" is YOUR email address.