Re: Keepalived track_script and failover

Still no joy.

If it was blocking multicast traffic, it wouldn't failover when I shut one of the keepalived instances down would it? 


-----Original Message-----
From: Ryan O'Hara [mailto:[email protected]] 
Sent: July-15-13 4:11 PM
To: Sheldon Lee-Wen
Cc: [email protected]
Subject: Re: [Keepalived-devel] Keepalived track_script and failover

On Mon, Jul 15, 2013 at 07:41:07PM +0000, Sheldon Lee-Wen wrote:
> Thanks Ryan,
> 
> There are no iptables rules. I've tried setting both to BACKUP (the outcome is below) and one to BACKUP one to MASTER (following the priority) and when I stop haproxy it still does not failover the VIPs to the next host. 
> 
> See below:
> Here are the logs from each host:
> Host1:
> root@netlb-1-stg:/etc/keepalived# tail -f /var/log/messages Jul 15 
> 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Configuration 
> is using : 7209 Bytes Jul 15 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Using LinkWatch kernel netlink reflector...
> Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
> Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: 
> VRRP_Script(chk_haproxy) succeeded Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
> Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: 
> VRRP_Instance(VI_1) forcing a new MASTER election Jul 15 15:29:42 
> netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Transition to 
> MASTER STATE Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Entering MASTER STATE Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
> Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
> Jul 15 15:32:55 netlb-1-stg Keepalived_vrrp[22437]: 
> VRRP_Script(chk_haproxy) failed
> 
> The last line of this log file showed up once I did a /etc/init.d/haproxy stop. 
> 
> The output of my commands on this host are seen below:
> 
> root@netlb-1-stg:~# /etc/init.d/haproxy stop
>  * Stopping haproxy haproxy                                                                                                                                               [ OK ]
> 
> root@netlb-1-stg:~# ip addr show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
>     link/ether 00:50:56:a1:25:e5 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.12.6/24 brd 192.168.12.255 scope global eth0
>     inet 192.168.12.5/32 scope global eth0
>     inet 192.168.12.10/32 scope global eth0
>     inet6 fe80::250:56ff:fea1:25e5/64 scope link
>        valid_lft forever preferred_lft forever
> 
> 
> Host2:
> root@netlb-2-stg:/etc/keepalived# tail -f /var/log/messages Jul 15 
> 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Registering 
> Kernel netlink reflector Jul 15 15:32:08 netlb-2-stg 
> Keepalived_healthcheckers[12389]: Registering Kernel netlink command channel Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Opening file '/etc/keepalived/keepalived.conf'.
> Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: 
> Configuration is using : 7209 Bytes Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Using LinkWatch kernel netlink reflector...
> Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: 
> VRRP_Instance(VI_1) Entering BACKUP STATE Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: Remote SMTP server [192.168.11.11]:25 connected.
> Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: 
> VRRP_Script(chk_haproxy) succeeded Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: SMTP alert successfully sent.
> 
> Nothing happens on host2.
> 
> More information
> root@netlb-1-stg:~# dpkg -l | grep keepalived
> ii  keepalived                       1:1.2.7-3                    Failover and monitoring daemon for LVS clusters
> 
> This is the latest version of keepalived. I tried it with an older version but it would not even run the check script.
> 
> The following is in the /etc/sysctl.conf
> net.ipv4.ip_nonlocal_bind=1
> 
> Both VM's are running on Ubuntu 12.04.1 on ESX5 server.
> 
> Any ideas?

This is a long-shot, but could you try setting advert_int in your vrrp_instance?

I don't know much about ESX5. Is it possible that it is blocking multicast traffic?

Ryan


> -----Original Message-----
> From: Ryan O'Hara [mailto:[email protected]]
> Sent: July-15-13 3:29 PM
> To: Sheldon Lee-Wen
> Cc: [email protected]
> Subject: Re: [Keepalived-devel] Keepalived track_script and failover
> 
> On Mon, Jul 15, 2013 at 04:50:20PM +0000, Sheldon Lee-Wen wrote:
> > Hello,
> > 
> > Please forgive me if this is a dumb question, but being new to keepalived I need some help. If this is not the correct place to ask, please let me know where I should go for help.
> > 
> > I have keepalived setup between two servers that run haproxy. When haproxy stops running on one host, the  track_script fails (as it shows in the log) but keepalived does not fail the virtual IP addresses over to the other host. My expectation was that it would failover to the other host. What am I missing?
> 
> Can you please make sure that iptables is not blocking VRRP packets? I recently assisted someone else with this exact same problems and it turned out to be iptables.
> 
> Also, you have both nodes set as MASTER. I'm not entirely sure what will happen when you do this, but it could very well be causing problems. Either set Host2 to BACKUP state or set both to BACKUP.
> 
> Ryan
> 
> 
> > Here are the config files for both keepalived hosts:
> > 
> > Host1:
> > 
> > global_defs {
> >     notification_email {
> >         <removed>
> >     }
> >     notification_email_from <removed>
> >     smtp_server <removed>
> >     smtp_connect_timeout 30
> > }
> > 
> > vrrp_script chk_haproxy {
> >     script "killall -0 haproxy"
> >     interval 1
> >     weight 2
> > }
> > 
> > vrrp_instance VI_1 {
> >     interface eth0
> >     state MASTER
> >     virtual_router_id 51
> >     priority 150
> > 
> >     smtp_alert
> > 
> >     authentication {
> >        auth_type PASS
> >        auth_pass 1234qwer
> >     }
> > 
> >     virtual_ipaddress {
> >         192.168.12.5
> >         192.168.12.10
> >     }
> > 
> >     track_script {
> >        chk_haproxy
> >     }
> > }
> > 
> > Host2:
> > global_defs {
> >     notification_email {
> >         <removed>
> >     }
> >     notification_email_from <removed>
> >     smtp_server <removed>
> >     smtp_connect_timeout 30
> > }
> > 
> > vrrp_script chk_haproxy {
> >     script "killall -0 haproxy"
> >     interval 1
> >     weight 2
> > }
> > 
> > vrrp_instance VI_1 {
> >     interface eth0
> >     state MASTER
> >     virtual_router_id 51
> >     priority 100
> > 
> >     smtp_alert
> > 
> >     authentication {
> >        auth_type PASS
> >        auth_pass 1234qwer
> >     }
> > 
> >     virtual_ipaddress {
> >         192.168.12.5
> >         192.168.12.10
> >     }
> > 
> >     track_script {
> >        chk_haproxy
> >     }
> > }
> > 
> > Thank you in advance!
> > 
> > 
> > 
> 
> > --------------------------------------------------------------------
> > --
> > -------- See everything from the browser to the database with 
> > AppDynamics Get end-to-end visibility with application monitoring 
> > from AppDynamics Isolate bottlenecks and diagnose root cause in seconds.
> > Start your free trial of AppDynamics Pro today!
> > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg
> > .c
> > lktrk
> 
> > _______________________________________________
> > Keepalived-devel mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/keepalived-devel
> 

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk