Two VRRP preempt_delay bugs
Tore Anderson <[email protected]>
| Newsgroups | gmane.linux.keepalived.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi, current git master has two bugs relating to the preempt_delay setting. They were both introduced by this commit: commit c7a985db4136b374dbee25caa39aa121bc16fb7d Author: Joachim Nilsson <[email protected]> Date: Tue Sep 10 15:36:36 2013 +0200 Honor preempt_delay setting on startup. This is a fix to honor the preempt_delay setting on power-up, or reboot, by preventing a BACKUP router to transition to MASTER until its preempt timer has expired. Bug #1 is that preempt_delay comes into play at startup even when there are no other active VRRP speakers on the link. The keepalived.conf manual page is very clear that preemption means something different, namely to «preempt a lower priority machine when a higher priority machine comes online». This is also what I understand the English word "preemption" to mean, i.e., to take over something from someone else. The negative consequence of this bug is that if you're recovering from an outage that have impacted all VRRP speakers simultaneously (for example a power loss), you prolong the outage for the duration of preempt_delay, since none of them will go to MASTER state before preempt_delay has expired. >From the commit log I do realise that the patch is probably «working as intended», but in my opinion changing the meaning of preempt_delay in this manner is the wrong thing to do - the functionality should instead have been implemented as a brand new setting instead called "startup_delay" or "initial_delay" or something like that. If it really is necessary to implement this in keepalived at all - it appears to me that doing "sleep <n> && keepalived" in the init script would accomplish exactly the same thing. Bug #2 is far more critical, namely that preempt_delay is used at reload, irrespective of the current state of the VRRP instance. So if you have an VRRP instance in the MASTER state, and send keepalived SIGHUP, it will sit there *with the virtual addresses active* for the duration of preempt_delay before it will resume sending out VRRP advertisements. If preempt_delay is set to something higher than a couple of seconds at most, another VRRP speaker will notice the absence of advertisements and transition to MASTER state, and you'll have a "split-brain" active/active situation, which will last until preempt_delay has expired and the node starts transmitting VRRP advertisements again. I propose simply reverting the problematic commit. This solves both problems. The following logs demonstrate the problem being reproduced. There is only one VRRP speaker on the link. Configuration is as follows: vrrp_instance eth1 { interface eth1 virtual_router_id 10 preempt_delay 30 virtual_ipaddress { 192.168.1.1/30 } } Mar 6 11:13:04 ucstest Keepalived[21800]: Starting Keepalived v1.2.12 (03/03,2014) Mar 6 11:13:04 ucstest Keepalived[21801]: Starting Healthcheck child process, pid=21802 Mar 6 11:13:04 ucstest Keepalived[21801]: Starting VRRP child process, pid=21804 Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Initializing ipvs 2.6 Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink reflector Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink command channel Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP shared channel Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Opening file '/etc/keepalived/keepalived.conf'. Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Configuration is using : 59582 Bytes Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel netlink reflector... Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't initialize ipvs: Protocol not available Mar 6 11:13:04 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Entering BACKUP STATE Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink reflector Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink command channel Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Opening file '/etc/keepalived/keepalived.conf'. Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Configuration is using : 4713 Bytes Mar 6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch kernel netlink reflector... >Bug 1: The delay at this point. There are no other VRRP speakers on the >link, so there are nobody to preempt, yet preempt_delay comes into play. Mar 6 11:13:34 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Transition to MASTER STATE Mar 6 11:13:35 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Entering MASTER STATE >The virtual address gets addded to the interface at this point. Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Got SIGHUP, reloading checker configuration Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Initializing ipvs 2.6 Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't initialize ipvs: Protocol not available Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink reflector Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering Kernel netlink command channel Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Opening file '/etc/keepalived/keepalived.conf'. Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Configuration is using : 3601 Bytes Mar 6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch kernel netlink reflector... Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink reflector Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink command channel Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP shared channel Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Opening file '/etc/keepalived/keepalived.conf'. Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Configuration is using : 58423 Bytes Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel netlink reflector... Mar 6 11:14:55 ucstest Keepalived_vrrp[21804]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9) >Bug 2: The delay at this point. The virtual address remains configured >on eth1, but no VRRP advertisements are transmitted. Had there been >another VRRP speaker on the link we would have had a split-brain >situation during this delay (except for the first few seconds. Mar 6 11:15:25 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1) Transition to MASTER STATE Tore ------------------------------------------------------------------------------ Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. With Perforce, you get hassle-free workflows. Merge that actually works. Faster operations. Version large binaries. Built-in WAN optimization and the freedom to use Git, Perforce or both. Make the move to Perforce. http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk