Re: Two VRRP preempt_delay bugs

Hi Tore,

I will investigate and fix it (with your other email on reload). Thanks for
your time.

regs,
Alexandre


On Thu, Mar 6, 2014 at 11:22 AM, Tore Anderson <[email protected]> wrote:

> Hi, current git master has two bugs relating to the preempt_delay
> setting. They were both introduced by this commit:
>
> commit c7a985db4136b374dbee25caa39aa121bc16fb7d
> Author: Joachim Nilsson <[email protected]>
> Date:   Tue Sep 10 15:36:36 2013 +0200
>
>     Honor preempt_delay setting on startup.
>
>     This is a fix to honor the preempt_delay setting on power-up, or
> reboot, by
>     preventing a BACKUP router to transition to MASTER until its preempt
> timer
>     has expired.
>
> Bug #1 is that preempt_delay comes into play at startup even when there
> are no other active VRRP speakers on the link. The keepalived.conf
> manual page is very clear that preemption means something different,
> namely to «preempt a lower priority machine when a higher priority
> machine comes online». This is also what I understand the English word
> "preemption" to mean, i.e., to take over something from someone else.
>
> The negative consequence of this bug is that if you're recovering from
> an outage that have impacted all VRRP speakers simultaneously (for
> example a power loss), you prolong the outage for the duration of
> preempt_delay, since none of them will go to MASTER state before
> preempt_delay has expired.
>
> >From the commit log I do realise that the patch is probably «working as
> intended», but in my opinion changing the meaning of preempt_delay in
> this manner is the wrong thing to do - the functionality should instead
> have been implemented as a brand new setting instead called
> "startup_delay" or "initial_delay" or something like that. If it really
> is necessary to implement this in keepalived at all - it appears to me
> that doing "sleep <n> && keepalived" in the init script would accomplish
> exactly the same thing.
>
> Bug #2 is far more critical, namely that preempt_delay is used at
> reload, irrespective of the current state of the VRRP instance. So if
> you have an VRRP instance in the MASTER state, and send keepalived
> SIGHUP, it will sit there *with the virtual addresses active* for the
> duration of preempt_delay before it will resume sending out VRRP
> advertisements. If preempt_delay is set to something higher than a
> couple of seconds at most, another VRRP speaker will notice the absence
> of advertisements and transition to MASTER state, and you'll have a
> "split-brain" active/active situation, which will last until
> preempt_delay has expired and the node starts transmitting VRRP
> advertisements again.
>
> I propose simply reverting the problematic commit. This solves both
> problems.
>
> The following logs demonstrate the problem being reproduced. There
> is only one VRRP speaker on the link. Configuration is as follows:
>
> vrrp_instance eth1 {
>     interface eth1
>     virtual_router_id 10
>     preempt_delay 30
>     virtual_ipaddress {
>         192.168.1.1/30
>     }
> }
>
> Mar  6 11:13:04 ucstest Keepalived[21800]: Starting Keepalived v1.2.12
> (03/03,2014)
> Mar  6 11:13:04 ucstest Keepalived[21801]: Starting Healthcheck child
> process, pid=21802
> Mar  6 11:13:04 ucstest Keepalived[21801]: Starting VRRP child process,
> pid=21804
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Initializing
> ipvs 2.6
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink
> reflector
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink
> command channel
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP
> shared channel
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Opening file
> '/etc/keepalived/keepalived.conf'.
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Configuration is using :
> 59582 Bytes
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel
> netlink reflector...
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't
> initialize ipvs: Protocol not available
> Mar  6 11:13:04 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1)
> Entering BACKUP STATE
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering
> Kernel netlink reflector
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Registering
> Kernel netlink command channel
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Opening file
> '/etc/keepalived/keepalived.conf'.
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Configuration is
> using : 4713 Bytes
> Mar  6 11:13:04 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch
> kernel netlink reflector...
> >Bug 1: The delay at this point. There are no other VRRP speakers on the
> >link, so there are nobody to preempt, yet preempt_delay comes into play.
> Mar  6 11:13:34 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1)
> Transition to MASTER STATE
> Mar  6 11:13:35 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1)
> Entering MASTER STATE
> >The virtual address gets addded to the interface at this point.
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Got SIGHUP,
> reloading checker configuration
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Initializing
> ipvs 2.6
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: IPVS: Can't
> initialize ipvs: Protocol not available
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering
> Kernel netlink reflector
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Registering
> Kernel netlink command channel
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Opening file
> '/etc/keepalived/keepalived.conf'.
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Configuration is
> using : 3601 Bytes
> Mar  6 11:14:54 ucstest Keepalived_healthcheckers[21802]: Using LinkWatch
> kernel netlink reflector...
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink
> reflector
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering Kernel netlink
> command channel
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Registering gratuitous ARP
> shared channel
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Opening file
> '/etc/keepalived/keepalived.conf'.
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Configuration is using :
> 58423 Bytes
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: Using LinkWatch kernel
> netlink reflector...
> Mar  6 11:14:55 ucstest Keepalived_vrrp[21804]: cant do IP_DROP_MEMBERSHIP
> errno=Bad file descriptor (9)
> >Bug 2: The delay at this point. The virtual address remains configured
> >on eth1, but no VRRP advertisements are transmitted. Had there been
> >another VRRP speaker on the link we would have had a split-brain
> >situation during this delay (except for the first few seconds.
> Mar  6 11:15:25 ucstest Keepalived_vrrp[21804]: VRRP_Instance(eth1)
> Transition to MASTER STATE
>
> Tore
>
>
> ------------------------------------------------------------------------------
> Subversion Kills Productivity. Get off Subversion & Make the Move to
> Perforce.
> With Perforce, you get hassle-free workflows. Merge that actually works.
> Faster operations. Version large binaries.  Built-in WAN optimization and
> the
> freedom to use Git, Perforce or both. Make the move to Perforce.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
> _______________________________________________
> Keepalived-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/keepalived-devel
>

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works. 
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel