Re: Graceful failover or how to eliminate failover downtime in keepalived

On Mon, 2016-06-06 at 11:47 +0200, Pim Rupert wrote:

> Hi,
> 
> Whenever there is a failover between two machines with keepalived there is a small window of downtime (about 2 seconds), because the VIP is removed by keepalived first from node A, and added later by keepalived on node B. When a failover is started due to a priority election or scheduled maintenance it seems absurd to tolerate this two seconds of downtime. How do we properly prevent this downtime? 
> 
> I know this isn't a problem that is up to Keepalived to fix, but I am keen to learn how you deal with this problem. 
> 


Pim,

RFC5798 is the place to look for this. It looks to me as thought your
vrrp instances are using advert_int 1 (the default). State transitions
are based on the advertisement interval, and VRRPv3 supports
advertisement intervals down to 10 milli-seconds, so reducing your
advertisement intervals would speed up the transitions from backup to
master, but would increase the number of adverts being sent. Setting the
priorities as high as possible, e.g. 200 and 254 will also slightly
reduce the time for a backup to become master.

Following some experiments I have done, when I set the advertisement
interval (advert_int) to 5 seconds, I have established the following:
Priority election:
When node1 forces a new MASTER election, it sends an advertisement which
node2 receives, and node2 immediately transitions to backup state. node1
then waits 1 advert_int before it transitions to master state, and then
a further advert_int before it enters master state, which is when it
adds the virtual IP address, and it comes up again.

Scheduled downtime:
When the wsrepready script fails on node 1, each instance that is
tracking that script goes into fault state and sends a priority 0
advertisement. node2 transitions to master state 1 advert_int later for
IPv6, or two advert_ints later for IPv4 (I still to confirm that it is
the difference between IPv4 and IPv6 that makes the difference between 1
and 2 advert_ints here), and then one further advert_int later they
enter master state, adding the virtual ip addresses, and things start
working again.

The above explains why you are seeing approximately 2 second delays with
advert_int set to 1.

Within the RFC I cannot see any requirement for the delays being
experienced here. With priorities set at 254, the transition to master
after a priority 0 advert should be virtually instantaneous (i.e. after
only the skew time, which in this case would be (256-254) * 5 / 256, so
approx 1/25th second).

I'll have a look at the source code to try and understand what is
happening, but it may take a while to understand it, unless someone can
already explain the reason. I'll also ask Alexandre if he can explain
what is happening.

I hope this helps,
Quentin

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel