Re: Keepalived preempts

On Fri, 2016-09-16 at 11:44 -0400, Ross Vandegrift wrote:

> Hello all,
> 
> I'm seeing strange VRRP preempt behavior on keepalived 1.2.16 on ubuntu
> 14.04 LTS.  Both boxes have priority 100 and nopreempt.
> 
> 10.84.128.202 was master, 10.84.129.151 was backup.  I rebooted the
> backup box, and when it returned, it preempted.
> 
> Config is simple:
> vrrp_instance instance0 {
>     state BACKUP
>     interface eth0
>     unicast_src_ip 10.84.129.151
>     unicast_peer {
>         10.84.128.202
>     }
>     virtual_router_id 1
>     priority 100
>     nopreempt
>     notify_master /etc/keepalived/master.sh
>     notify_backup /etc/keepalived/backup.sh
>     notify_fault /etc/keepalived/fault.sh
> }
> 
> Peer config is identical but unicast IPs are swapped.  I normally use AH
> auth, but it's disabled to simplify troubleshooting - that hasn't
> helped.  Maybe I've done something else wrong?
> 
> Here's the startup logs from 10.84.129.151:
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived[1092]: Starting Keepalived v1.2.16 (08/04,2015)
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived[1093]: Starting Healthcheck child process, pid=1094
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived[1093]: Starting VRRP child process, pid=1095
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Registering Kernel netlink reflector
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Registering Kernel netlink command channel
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Registering gratuitous ARP shared channel
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Initializing ipvs 2.6
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Opening file '/etc/keepalived/keepalived.conf'.
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Configuration is using : 60216 Bytes
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Using LinkWatch kernel netlink reflector...
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: VRRP_Instance(instance0) Entering BACKUP STATE
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_vrrp[1095]: Opening script file /etc/keepalived/backup.sh
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Registering Kernel netlink reflector
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Registering Kernel netlink command channel
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Opening file '/etc/keepalived/keepalived.conf'.
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Configuration is using : 5437 Bytes
> > Sep 16 14:55:21 ip-10-84-129-151 Keepalived_healthcheckers[1094]: Using LinkWatch kernel netlink reflector...
> > Sep 16 14:55:24 ip-10-84-129-151 Keepalived_vrrp[1095]: VRRP_Instance(instance0) Transition to MASTER STATE
> > Sep 16 14:55:25 ip-10-84-129-151 Keepalived_vrrp[1095]: VRRP_Instance(instance0) Entering MASTER STATE
> > Sep 16 14:55:25 ip-10-84-129-151 Keepalived_vrrp[1095]: Opening script file /etc/keepalived/master.sh
> 
> Here's the preempt logs from 10.84.128.202:
> > Sep 16 14:55:25 ip-10-84-128-202 Keepalived_vrrp[25564]: VRRP_Instance(instance0) Received higher prio advert
> > Sep 16 14:55:25 ip-10-84-128-202 Keepalived_vrrp[25564]: VRRP_Instance(instance0) Entering BACKUP STATE
> > Sep 16 14:55:25 ip-10-84-128-202 Keepalived_vrrp[25564]: Opening script file /etc/keepalived/backup.sh
> 
> Ross
> 

Ross,

I've both looked at the 1.2.16 code, and tested it with your
configurations, and I cannot reproduce a new instance of keepalived
taking over as master.

BTW, specifying nopreempt has no effect when the priorities are equal,
since a backup will never take over from another instance when their
priorities are equal.

Can you try the following:
1. Stop keepalived on ip-10-84-129-151, so that ip-10-84-128-202 becomes
master.
2. Start keepalived again on ip-10-84-129-151.
Does ip-10-84-129-151 take over as master, or does ip-10-84-128-202
remain as master? I suspect ip-10-84-128-202 will remain as master.

If ip-10-84-128-202 remains as master, then it would suggest that
keepalived is working as expected.

Does ip-10-84-129-151 start keepalived as part of the boot/init process?
My guess is that when ip-10-84-129-151 boots, it starts keepalived
before the networking is fully up, and so it doesn't receive any
advertisements from ip-10-84-128-202, and hence times out and becomes
master. You then have a split brain situation with both systems thinking
they are master and are of equal priority. When the networking starts
working, the keepalived instances start receiving adverts and if a
master receives an advert of equal priority, it then checks is own
address against the sender's address; if it's own address is lower, it
considers it is of lower priority and transitions to backup, and what we
are seeing here is certainly the lower address system (ip-10-84-128-202)
being the one to transition to backup.

The timing, as shown in the logs, looks critical here. The first time
ip-10-84-129-151 sends an advert is 14:55:24 (when it transitions to
MASTER), but it looks as though the first advert received by
ip-10-84-128-202 is at 14:55:25, which is sent by ip-10-84-129-151 when
it enters MASTER state (I'm assuming the system clocks are
synchronised), so this supports the idea that the advert sent at
14:55:24 isn't received by ip-10-84-128-202.

If you increase the adver_int time in the keepalived config to, say, 3
seconds, rather than the 1 second you are now using, then the problem
may not occur, since it will take ip-10-84-129-151 about 10 seconds to
timeout and become master, but before then (if the process startup
timing at boot is consistent) it will have received an advert from
ip-10-84-128-202. [I'm not suggesting that you should leave the
adver_int settings at 3 seconds, but this is just a way to prove if it
is a boot timing issue]

This scenario could be further simulated by adding iptables entries on
one system to block incoming and outgoing vrrp packets (iptables -I
INPUT -p 112 -j DROP; iptables -I OUTPUT -p 112 -j DROP), and letting
both systems be master. Then delete the iptables entries (iptables -D
INPUT -p 112 -j DROP; iptables -D OUTPUT -p 112 -j DROP), seeing how
they behave.

I hope that helps,

Quentin Armitage

------------------------------------------------------------------------------

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel