Two VRRP preempt_delay bugs

Tore Anderson <[email protected]>
Newsgroups gmane.linux.keepalived.devel
Message-ID <[email protected]>
Hi, current git master has two bugs relating to the preempt_delay
setting. They were both introduced by this commit:

commit c7a985db4136b374dbee25caa39aa121bc16fb7d
Author: Joachim Nilsson <[email protected]>
Date:   Tue Sep 10 15:36:36 2013 +0200

    Honor preempt_delay setting on startup.
    
    This is a fix to honor the preempt_delay setting on power-up, or reboot, by
    preventing a BACKUP router to transition to MASTER until its preempt timer
    has expired.

Bug #1 is that preempt_delay comes into play at startup even when there
are no other active VRRP speakers on the link. The keepalived.conf
manual page is very clear that preemption means something different,
namely to «preempt a lower priority machine when a higher priority
machine comes online». This is also what I understand the English word
"preemption" to mean, i.e., to take over something from someone else.

The negative consequence of this bug is that if you're recovering from
an outage that have impacted all VRRP speakers simultaneously (for
example a power loss), you prolong the outage for the duration of
preempt_delay, since none of them will go to MASTER state before
preempt_delay has expired.

>From the commit log I do realise that the patch is probably «working as
intended», but in my opinion changing the meaning of preempt_delay in
this manner is the wrong thing to do - the functionality should instead
have been implemented as a brand new setting instead called
"startup_delay" or "initial_delay" or something like that. If it really
is necessary to implement this in keepalived at all - it appears to me
that doing "sleep <n> && keepalived" in the init script would accomplish
exactly the same thing.

Bug #2 is far more critical, namely that preempt_delay is used at
reload, irrespective of the current state of the VRRP instance. So if
you have an VRRP instance in the MASTER state, and send keepalived
SIGHUP, it will sit there *with the virtual addresses active* for the
duration of preempt_delay before it will resume sending out VRRP
advertisements. If preempt_delay is set to something higher than a
couple of seconds at most, another VRRP speaker will notice the absence
of advertisements and transition to MASTER state, and you'll have a
"split-brain" active/active situation, which will last until
preempt_delay has expired and the node starts transmitting VRRP
advertisements again.

I propose simply reverting the problematic commit. This solves both
problems.

The following logs demonstrate the problem being reproduced. There
is only one VRRP speaker on the link. Configuration is as follows:

vrrp_instance eth1 {
    interface eth1
    virtual_router_id 10
    preempt_delay 30
    virtual_ipaddress {
        192.168.1.1/30
    }
}

Mar  6 11:13:04 ucstest Keepalived[21800]: Starting Keepalived v1.2.12 (03/03,2014)
Mar  6 11:13:04 ucstest Keepalived[21801]: Starting Healthcheck child process, pid=21802
Mar  6 11:13:04 ucstest Keepalived[21801]: Starting VRRP child process, pid=21804
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Initializing ipvs 2.6
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink reflector
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink command channel
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP shared channel
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Opening file '/etc/keepalived/keepalived.conf'.
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Configuration is using : 59582 Bytes
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel netlink reflector...
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't initialize ipvs: Protocol not available
Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Entering BACKUP STATE
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink reflector
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink command channel
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Opening file '/etc/keepalived/keepalived.conf'.
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Configuration is using : 4713 Bytes
Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch kernel netlink reflector...
>Bug 1: The delay at this point. There are no other VRRP speakers on the
>link, so there are nobody to preempt, yet preempt_delay comes into play.
Mar  6 11:13:34 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Transition to MASTER STATE
Mar  6 11:13:35 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Entering MASTER STATE
>The virtual address gets addded to the interface at this point.
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Got SIGHUP, reloading checker configuration
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Initializing ipvs 2.6
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't initialize ipvs: Protocol not available
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink reflector
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink command channel
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Opening file '/etc/keepalived/keepalived.conf'.
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Configuration is using : 3601 Bytes
Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch kernel netlink reflector...
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink reflector
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink command channel
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP shared channel
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Opening file '/etc/keepalived/keepalived.conf'.
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Configuration is using : 58423 Bytes
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel netlink reflector...
Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)
>Bug 2: The delay at this point. The virtual address remains configured
>on eth1, but no VRRP advertisements are transmitted. Had there been
>another VRRP speaker on the link we would have had a split-brain
>situation during this delay (except for the first few seconds.
Mar  6 11:15:25 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Transition to MASTER STATE

Tore

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works. 
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.