When heartbeat core process dies, failover is not succeeded.

Hello,

I use 2 Linux-Directors with heartbeat in LVS-DR,
and Linux-Directors and Real-Servers into the same hosts.

hostname : web1 (LVS master)
OS       : Red Hat Enterprise Linux 4
kernel   : 2.6.9-22.ELsmp
packages : ultramonkey 3
           ipvsadm 1.24-5
           heartbeat-1.2.3.cvs.20050927-1.rh.el.um.1
           arptables_jf-0.0.8-2
           Apache 2.0.52
           ==== haresources =========================================
           web1 \
               ldirectord::ldirectord.cf \
               LVSSyncDaemonSwap::master \
               IPaddr::192.168.21.151/24
           ==== ha.cf ===============================================
           logfile /var/log/ha-log
           keepalive 1
           deadtime 3
           auto_failback off
           bcast   bond0
           node web1
           node web2
           ==========================================================

hostname : web2 (LVS slave)
OS       : Red Hat Enterprise Linux 4
kernel   : 2.6.9-22.ELsmp
packages : ultramonkey 3
           ipvsadm 1.24-5
           heartbeat-1.2.3.cvs.20050927-1.rh.el.um.1
           arptables_jf-0.0.8-2
           Apache 2.0.52
           ==== haresources =========================================
           web1 \
               ldirectord::ldirectord.cf \
               LVSSyncDaemonSwap::master \
               IPaddr::192.168.21.151/24
           ==== ha.cf ===============================================
           logfile /var/log/ha-log
           keepalive 1
           deadtime 3
           auto_failback off
           bcast   bond0
           node web1
           node web2
           ==========================================================

case 1:
When heartbeat:control process dies in web1, I think web1 is LVS slave
, and web2 is LVS master.
But I kill(-9) heartbeat:control process in web1, web1 does not 
release its resources.

(at web1)
・VIP is not released.
・arptables is not changed.
・Loopback Interface is continuously down.
(at web2)
・VIP is up.
・arptables is changed accepting incoming ARP requests.
・LoopBack Interface is down.

=== case1.web1.ha-log ===============================================

(Non log written. The message to release VIP, to change arptables
 dropping incoming ARP requests for the VIP, to be up Loopback
 Interface should be outputed.)

=== case1.web2.ha-log ===============================================
heartbeat: 2005/11/29_12:52:07 WARN: node web1: is dead
heartbeat: 2005/11/29_12:52:07 WARN: No STONITH device configured.
heartbeat: 2005/11/29_12:52:07 WARN: Shared disks are not protected.
heartbeat: 2005/11/29_12:52:07 info: Resources being acquired from we
b1.
heartbeat: 2005/11/29_12:52:07 info: Link web1:bond0 dead.
heartbeat: 2005/11/29_12:52:07 info: Running /etc/ha.d/rc.d/status st
atus
heartbeat: 2005/11/29_12:52:07 info: No local resources [/usr/lib/hea
rtbeat/ResourceManager listkeys web2] to acquire.
heartbeat: 2005/11/29_12:52:07 info: Taking over resource group ldire
ctord::ldirectord.cf
heartbeat: 2005/11/29_12:52:07 info: Acquiring resource group: web1 l
directord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr::192.168.21
.151/24
heartbeat: 2005/11/29_12:52:08 info: Running /etc/ha.d/resource.d/ldi
rectord ldirectord.cf start
heartbeat: 2005/11/29_12:52:09 info: Running /etc/ha.d/resource.d/LVS
SyncDaemonSwap master start
heartbeat: 2005/11/29_12:52:09 info: ipvs_syncbackup down
heartbeat: 2005/11/29_12:52:09 info: ipvs_syncmaster up
heartbeat: 2005/11/29_12:52:09 info: ipvs_syncmaster obtained
heartbeat: 2005/11/29_12:52:09 info: Running /etc/ha.d/resource.d/IPa
ddr 192.168.21.151/24 start
heartbeat: 2005/11/29_12:52:09 info: Removing conflicting loopback lo
:0.
heartbeat: 2005/11/29_12:52:09 info: /sbin/ifconfig lo:0 down
heartbeat: 2005/11/29_12:52:09 info: /sbin/route -n del -host 192.168
.21.151
heartbeat: 2005/11/29_12:52:09 info: /usr/sbin/arptables-noarp-addr 1
92.168.21.151 stop: success
heartbeat: 2005/11/29_12:52:09 info: /sbin/ifconfig bond0:0 192.168.2
1.151 netmask 255.255.255.0  broadcast 192.168.21.255
heartbeat: 2005/11/29_12:52:09 info: Sending Gratuitous Arp for 192.1
68.21.151 on bond0:0 [bond0]
heartbeat: 2005/11/29_12:52:09 /usr/lib/heartbeat/send_arp -i 1010 -r
 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.21.151 bond
0 192.168.21.151 auto 192.168.21.151 ffffffffffff
heartbeat: 2005/11/29_12:52:09 info: /usr/lib/heartbeat/mach_down: ni
ce_failback: foreign resources acquired
heartbeat: 2005/11/29_12:52:09 info: mach_down takeover complete.
heartbeat: 2005/11/29_12:52:09 info: mach_down takeover complete for 
node web1.
=====================================================================


case 2:
And when heartbeat:FIFO read process dies in web1, I think web1 is LVS
slave, and web2 is LVS master, too.
But I kill(-9) heartbeat:FIFO read process in web1, web2 does not take
resources.

(at web1)
・VIP is down.
・arptables is changed dropping incoming ARP requests for the VIP,
  and mangles outgoing ARP announcements on each ARPing interface.
・Loopback Interface is up.
(at web2)
・VIP is not up.
・arptables is not changed.
・Loopback Interface is continuously up. Isn't released.

=== case2.web1.ha-log ===============================================
heartbeat: 2005/11/30_15:10:24 ERROR: Exiting HBFIFO process 27546 ki
lled by signal 9.
heartbeat: 2005/11/30_15:10:24 ERROR: Core heartbeat process died! Re
starting.
heartbeat: 2005/11/30_15:10:24 info: Heartbeat shutdown in progress. 
(27543)
heartbeat: 2005/11/30_15:10:24 info: Giving up all HA resources.
heartbeat: 2005/11/30_15:10:24 info: Releasing resource group: web1 l
directord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr::192.168.21
.151/24
heartbeat: 2005/11/30_15:10:24 info: Running /etc/ha.d/resource.d/IPa
ddr 192.168.21.151/24 stop
heartbeat: 2005/11/30_15:10:24 info: /usr/sbin/arptables-noarp-addr 1
92.168.21.151 stop: success
heartbeat: 2005/11/30_15:10:24 info: /sbin/route -n del -host 192.168
.21.151
heartbeat: 2005/11/30_15:10:24 info: /sbin/ifconfig bond0:0 down
heartbeat: 2005/11/30_15:10:24 info: Restoring loopback IP Address  1
92.168.21.151 on lo:0.
heartbeat: 2005/11/30_15:10:24 info: /sbin/ifconfig lo:0 192.168.21.1
51 netmask 255.255.255.255
heartbeat: 2005/11/30_15:10:24 info: IP Address 192.168.21.151 releas
ed
heartbeat: 2005/11/30_15:10:24 info: Running /etc/ha.d/resource.d/LVS
SyncDaemonSwap master stop
heartbeat: 2005/11/30_15:10:25 info: ipvs_syncmaster down
heartbeat: 2005/11/30_15:10:25 info: ipvs_syncbackup up
heartbeat: 2005/11/30_15:10:25 info: ipvs_syncmaster released
heartbeat: 2005/11/30_15:10:25 info: Running /etc/ha.d/resource.d/ldi
rectord ldirectord.cf stop
heartbeat: 2005/11/30_15:10:25 info: All HA resources relinquished.
heartbeat: 2005/11/30_15:10:25 ERROR: send_cluster_msg: cannot open /
var/lib/heartbeat/fifo: No such device or address
=== case2.web2.ha-log ===============================================

(Non log written. The message to be up VIP, to change arptables
 accepting incoming ARP requests for the VIP, to release Loopback
 Interface should be outputed.)

=====================================================================


Still when I kill(-9) heartbeat:control process in web2,
and when I kill(-9) heartbeat:FIFO read process in web2,
there is especially no problem in the state of the resources.

I would like you to teach when there is a solution.
Thanks.



-- 
Ultra Monkey - http://www.ultramonkey.org/
To UNSUBSCRIBE, email to [email protected], with a body:
unsubscribe ultramonkey-users [email protected]
where "[email protected]" is YOUR email address.