Re: Keepalived track_script and failover

Thanks Ryan,

There are no iptables rules. I've tried setting both to BACKUP (the outcome is below) and one to BACKUP one to MASTER (following the priority) and when I stop haproxy it still does not failover the VIPs to the next host. 

See below:
Here are the logs from each host:
Host1:
root@netlb-1-stg:/etc/keepalived# tail -f /var/log/messages
Jul 15 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Configuration is using : 7209 Bytes
Jul 15 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Using LinkWatch kernel netlink reflector...
Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Script(chk_haproxy) succeeded
Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) forcing a new MASTER election
Jul 15 15:29:42 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
Jul 15 15:32:55 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Script(chk_haproxy) failed

The last line of this log file showed up once I did a /etc/init.d/haproxy stop. 

The output of my commands on this host are seen below:

root@netlb-1-stg:~# /etc/init.d/haproxy stop
 * Stopping haproxy haproxy                                                                                                                                               [ OK ]

root@netlb-1-stg:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:a1:25:e5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.6/24 brd 192.168.12.255 scope global eth0
    inet 192.168.12.5/32 scope global eth0
    inet 192.168.12.10/32 scope global eth0
    inet6 fe80::250:56ff:fea1:25e5/64 scope link
       valid_lft forever preferred_lft forever


Host2:
root@netlb-2-stg:/etc/keepalived# tail -f /var/log/messages
Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Registering Kernel netlink reflector
Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Registering Kernel netlink command channel
Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Configuration is using : 7209 Bytes
Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Using LinkWatch kernel netlink reflector...
Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: Remote SMTP server [192.168.11.11]:25 connected.
Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: VRRP_Script(chk_haproxy) succeeded
Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: SMTP alert successfully sent.

Nothing happens on host2.

More information
root@netlb-1-stg:~# dpkg -l | grep keepalived
ii  keepalived                       1:1.2.7-3                    Failover and monitoring daemon for LVS clusters

This is the latest version of keepalived. I tried it with an older version but it would not even run the check script.

The following is in the /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1

Both VM's are running on Ubuntu 12.04.1 on ESX5 server.

Any ideas?

-----Original Message-----
From: Ryan O'Hara [mailto:[email protected]] 
Sent: July-15-13 3:29 PM
To: Sheldon Lee-Wen
Cc: [email protected]
Subject: Re: [Keepalived-devel] Keepalived track_script and failover

On Mon, Jul 15, 2013 at 04:50:20PM +0000, Sheldon Lee-Wen wrote:
> Hello,
> 
> Please forgive me if this is a dumb question, but being new to keepalived I need some help. If this is not the correct place to ask, please let me know where I should go for help.
> 
> I have keepalived setup between two servers that run haproxy. When haproxy stops running on one host, the  track_script fails (as it shows in the log) but keepalived does not fail the virtual IP addresses over to the other host. My expectation was that it would failover to the other host. What am I missing?

Can you please make sure that iptables is not blocking VRRP packets? I recently assisted someone else with this exact same problems and it turned out to be iptables.

Also, you have both nodes set as MASTER. I'm not entirely sure what will happen when you do this, but it could very well be causing problems. Either set Host2 to BACKUP state or set both to BACKUP.

Ryan


> Here are the config files for both keepalived hosts:
> 
> Host1:
> 
> global_defs {
>     notification_email {
>         <removed>
>     }
>     notification_email_from <removed>
>     smtp_server <removed>
>     smtp_connect_timeout 30
> }
> 
> vrrp_script chk_haproxy {
>     script "killall -0 haproxy"
>     interval 1
>     weight 2
> }
> 
> vrrp_instance VI_1 {
>     interface eth0
>     state MASTER
>     virtual_router_id 51
>     priority 150
> 
>     smtp_alert
> 
>     authentication {
>        auth_type PASS
>        auth_pass 1234qwer
>     }
> 
>     virtual_ipaddress {
>         192.168.12.5
>         192.168.12.10
>     }
> 
>     track_script {
>        chk_haproxy
>     }
> }
> 
> Host2:
> global_defs {
>     notification_email {
>         <removed>
>     }
>     notification_email_from <removed>
>     smtp_server <removed>
>     smtp_connect_timeout 30
> }
> 
> vrrp_script chk_haproxy {
>     script "killall -0 haproxy"
>     interval 1
>     weight 2
> }
> 
> vrrp_instance VI_1 {
>     interface eth0
>     state MASTER
>     virtual_router_id 51
>     priority 100
> 
>     smtp_alert
> 
>     authentication {
>        auth_type PASS
>        auth_pass 1234qwer
>     }
> 
>     virtual_ipaddress {
>         192.168.12.5
>         192.168.12.10
>     }
> 
>     track_script {
>        chk_haproxy
>     }
> }
> 
> Thank you in advance!
> 
> 
> 

> ----------------------------------------------------------------------
> -------- See everything from the browser to the database with 
> AppDynamics Get end-to-end visibility with application monitoring from 
> AppDynamics Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.c
> lktrk

> _______________________________________________
> Keepalived-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/keepalived-devel


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk