Re: Keepalived track_script and failover

Sheldon Lee-Wen <[email protected]>
Newsgroups gmane.linux.keepalived.devel
Message-ID <103EEAB24688A147B5B2E39C44C71BF05009C57C@EX2010-MAIL2.jsitelecom.com> 
Host1 logs:

Jul 15 16:24:48 netlb-1-stg Keepalived_healthcheckers: Terminating Healthchecker child process on signal
Jul 15 16:24:48 netlb-1-stg Keepalived_vrrp: Terminating VRRP child process on signal
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Netlink reflector reports IP 192.168.12.6 added
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Netlink reflector reports IP fe80::250:56ff:fea1:25e5 added
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Registering Kernel netlink reflector
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Registering Kernel netlink command channel
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Registering gratutious ARP shared channel
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Configuration is using : 65172 Bytes
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Netlink reflector reports IP 192.168.12.6 added
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Netlink reflector reports IP fe80::250:56ff:fea1:25e5 added
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Registering Kernel netlink reflector
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Registering Kernel netlink command channel
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Configuration is using : 7219 Bytes
Jul 15 16:24:58 netlb-1-stg Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: Remote SMTP server [192.168.11.10:25] connected.
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: VRRP_Script(chk_haproxy) succeeded
Jul 15 16:24:58 netlb-1-stg Keepalived_vrrp: SMTP alert successfully sent.
Jul 15 16:25:01 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Jul 15 16:25:02 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 15 16:25:02 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 15 16:25:02 netlb-1-stg Keepalived_healthcheckers: Netlink reflector reports IP 192.168.12.5 added
Jul 15 16:25:02 netlb-1-stg Keepalived_healthcheckers: Netlink reflector reports IP 192.168.12.10 added
Jul 15 16:25:02 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.12.5
Jul 15 16:25:02 netlb-1-stg Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.12.10

Host2 Logs:
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Netlink reflector reports IP 192.168.12.7 added
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Netlink reflector reports IP fe80::250:56ff:fea1:484b added
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Registering Kernel netlink reflector
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Registering Kernel netlink command channel
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Registering gratutious ARP shared channel
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Configuration is using : 65172 Bytes
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Netlink reflector reports IP 192.168.12.7 added
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Netlink reflector reports IP fe80::250:56ff:fea1:484b added
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Registering Kernel netlink reflector
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Registering Kernel netlink command channel
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: Remote SMTP server [192.168.11.11:25] connected.
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Configuration is using : 7219 Bytes
Jul 15 16:25:09 netlb-2-stg Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: VRRP_Script(chk_haproxy) succeeded
Jul 15 16:25:09 netlb-2-stg Keepalived_vrrp: SMTP alert successfully sent.


-----Original Message-----
From: Ryan O'Hara [mailto:[email protected]] 
Sent: July-15-13 4:24 PM
To: Sheldon Lee-Wen
Cc: [email protected]
Subject: Re: [Keepalived-devel] Keepalived track_script and failover

On Mon, Jul 15, 2013 at 08:17:54PM +0000, Sheldon Lee-Wen wrote:
> Still no joy.
> 
> If it was blocking multicast traffic, it wouldn't failover when I shut one of the keepalived instances down would it? 

No, but I didn't see evidence of that in the logs.

Run keepalived with -D and verify your config is read correctly.


> -----Original Message-----
> From: Ryan O'Hara [mailto:[email protected]]
> Sent: July-15-13 4:11 PM
> To: Sheldon Lee-Wen
> Cc: [email protected]
> Subject: Re: [Keepalived-devel] Keepalived track_script and failover
> 
> On Mon, Jul 15, 2013 at 07:41:07PM +0000, Sheldon Lee-Wen wrote:
> > Thanks Ryan,
> > 
> > There are no iptables rules. I've tried setting both to BACKUP (the outcome is below) and one to BACKUP one to MASTER (following the priority) and when I stop haproxy it still does not failover the VIPs to the next host. 
> > 
> > See below:
> > Here are the logs from each host:
> > Host1:
> > root@netlb-1-stg:/etc/keepalived# tail -f /var/log/messages Jul 15
> > 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Configuration 
> > is using : 7209 Bytes Jul 15 15:29:41 netlb-1-stg Keepalived_healthcheckers[22436]: Using LinkWatch kernel netlink reflector...
> > Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
> > Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: 
> > VRRP_Script(chk_haproxy) succeeded Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
> > Jul 15 15:29:41 netlb-1-stg Keepalived_vrrp[22437]: 
> > VRRP_Instance(VI_1) forcing a new MASTER election Jul 15 15:29:42 
> > netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Transition 
> > to MASTER STATE Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: VRRP_Instance(VI_1) Entering MASTER STATE Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: Remote SMTP server [192.168.11.10]:25 connected.
> > Jul 15 15:29:43 netlb-1-stg Keepalived_vrrp[22437]: SMTP alert successfully sent.
> > Jul 15 15:32:55 netlb-1-stg Keepalived_vrrp[22437]: 
> > VRRP_Script(chk_haproxy) failed
> > 
> > The last line of this log file showed up once I did a /etc/init.d/haproxy stop. 
> > 
> > The output of my commands on this host are seen below:
> > 
> > root@netlb-1-stg:~# /etc/init.d/haproxy stop
> >  * Stopping haproxy haproxy                                                                                                                                               [ OK ]
> > 
> > root@netlb-1-stg:~# ip addr show
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >     inet 127.0.0.1/8 scope host lo
> >     inet6 ::1/128 scope host
> >        valid_lft forever preferred_lft forever
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> >     link/ether 00:50:56:a1:25:e5 brd ff:ff:ff:ff:ff:ff
> >     inet 192.168.12.6/24 brd 192.168.12.255 scope global eth0
> >     inet 192.168.12.5/32 scope global eth0
> >     inet 192.168.12.10/32 scope global eth0
> >     inet6 fe80::250:56ff:fea1:25e5/64 scope link
> >        valid_lft forever preferred_lft forever
> > 
> > 
> > Host2:
> > root@netlb-2-stg:/etc/keepalived# tail -f /var/log/messages Jul 15
> > 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Registering 
> > Kernel netlink reflector Jul 15 15:32:08 netlb-2-stg
> > Keepalived_healthcheckers[12389]: Registering Kernel netlink command channel Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Opening file '/etc/keepalived/keepalived.conf'.
> > Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: 
> > Configuration is using : 7209 Bytes Jul 15 15:32:08 netlb-2-stg Keepalived_healthcheckers[12389]: Using LinkWatch kernel netlink reflector...
> > Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: 
> > VRRP_Instance(VI_1) Entering BACKUP STATE Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: Remote SMTP server [192.168.11.11]:25 connected.
> > Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: 
> > VRRP_Script(chk_haproxy) succeeded Jul 15 15:32:08 netlb-2-stg Keepalived_vrrp[12390]: SMTP alert successfully sent.
> > 
> > Nothing happens on host2.
> > 
> > More information
> > root@netlb-1-stg:~# dpkg -l | grep keepalived
> > ii  keepalived                       1:1.2.7-3                    Failover and monitoring daemon for LVS clusters
> > 
> > This is the latest version of keepalived. I tried it with an older version but it would not even run the check script.
> > 
> > The following is in the /etc/sysctl.conf
> > net.ipv4.ip_nonlocal_bind=1
> > 
> > Both VM's are running on Ubuntu 12.04.1 on ESX5 server.
> > 
> > Any ideas?
> 
> This is a long-shot, but could you try setting advert_int in your vrrp_instance?
> 
> I don't know much about ESX5. Is it possible that it is blocking multicast traffic?
> 
> Ryan
> 
> 
> > -----Original Message-----
> > From: Ryan O'Hara [mailto:[email protected]]
> > Sent: July-15-13 3:29 PM
> > To: Sheldon Lee-Wen
> > Cc: [email protected]
> > Subject: Re: [Keepalived-devel] Keepalived track_script and failover
> > 
> > On Mon, Jul 15, 2013 at 04:50:20PM +0000, Sheldon Lee-Wen wrote:
> > > Hello,
> > > 
> > > Please forgive me if this is a dumb question, but being new to keepalived I need some help. If this is not the correct place to ask, please let me know where I should go for help.
> > > 
> > > I have keepalived setup between two servers that run haproxy. When haproxy stops running on one host, the  track_script fails (as it shows in the log) but keepalived does not fail the virtual IP addresses over to the other host. My expectation was that it would failover to the other host. What am I missing?
> > 
> > Can you please make sure that iptables is not blocking VRRP packets? I recently assisted someone else with this exact same problems and it turned out to be iptables.
> > 
> > Also, you have both nodes set as MASTER. I'm not entirely sure what will happen when you do this, but it could very well be causing problems. Either set Host2 to BACKUP state or set both to BACKUP.
> > 
> > Ryan
> > 
> > 
> > > Here are the config files for both keepalived hosts:
> > > 
> > > Host1:
> > > 
> > > global_defs {
> > >     notification_email {
> > >         <removed>
> > >     }
> > >     notification_email_from <removed>
> > >     smtp_server <removed>
> > >     smtp_connect_timeout 30
> > > }
> > > 
> > > vrrp_script chk_haproxy {
> > >     script "killall -0 haproxy"
> > >     interval 1
> > >     weight 2
> > > }
> > > 
> > > vrrp_instance VI_1 {
> > >     interface eth0
> > >     state MASTER
> > >     virtual_router_id 51
> > >     priority 150
> > > 
> > >     smtp_alert
> > > 
> > >     authentication {
> > >        auth_type PASS
> > >        auth_pass 1234qwer
> > >     }
> > > 
> > >     virtual_ipaddress {
> > >         192.168.12.5
> > >         192.168.12.10
> > >     }
> > > 
> > >     track_script {
> > >        chk_haproxy
> > >     }
> > > }
> > > 
> > > Host2:
> > > global_defs {
> > >     notification_email {
> > >         <removed>
> > >     }
> > >     notification_email_from <removed>
> > >     smtp_server <removed>
> > >     smtp_connect_timeout 30
> > > }
> > > 
> > > vrrp_script chk_haproxy {
> > >     script "killall -0 haproxy"
> > >     interval 1
> > >     weight 2
> > > }
> > > 
> > > vrrp_instance VI_1 {
> > >     interface eth0
> > >     state MASTER
> > >     virtual_router_id 51
> > >     priority 100
> > > 
> > >     smtp_alert
> > > 
> > >     authentication {
> > >        auth_type PASS
> > >        auth_pass 1234qwer
> > >     }
> > > 
> > >     virtual_ipaddress {
> > >         192.168.12.5
> > >         192.168.12.10
> > >     }
> > > 
> > >     track_script {
> > >        chk_haproxy
> > >     }
> > > }
> > > 
> > > Thank you in advance!
> > > 
> > > 
> > > 
> > 
> > > ------------------------------------------------------------------
> > > --
> > > --
> > > -------- See everything from the browser to the database with 
> > > AppDynamics Get end-to-end visibility with application monitoring 
> > > from AppDynamics Isolate bottlenecks and diagnose root cause in seconds.
> > > Start your free trial of AppDynamics Pro today!
> > > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/os
> > > tg
> > > .c
> > > lktrk
> > 
> > > _______________________________________________
> > > Keepalived-devel mailing list
> > > [email protected]
> > > https://lists.sourceforge.net/lists/listinfo/keepalived-devel
> > 

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.