Re: Loss of routes to VIP

On Thu, 2016-10-13 at 17:52 +0000, Matt Hughes wrote:
> 
> I have five nodes in a cluster that share the same keepalived VIP.  I
> have 50 nodes that use this VIP.  Three times in the last month, an
> reelection has resulted in some of those 50 nodes not having a route
> to the VIP.  All the nodes are on the same subnet.  
> 
> --- settle on master4; what changed?  Why did we re-elect?
> Oct 11 14:19:49 master3 Entering MASTER STATE --- master3, master4
> Oct 11 14:19:52 master3 Entering BACKUP STATE --- master4
> Oct 11 14:19:53 master0 Entering MASTER STATE --- master0, master4
> Oct 11 14:19:53 master1 Entering MASTER STATE --- master1, master0,
> master4
> Oct 11 14:19:53 master2 Entering MASTER STATE --- master2, master1,
> master0, master4
> Oct 11 14:19:55 master2 Entering BACKUP STATE --- master1, master0,
> master4
> Oct 11 14:20:00 master0 Entering BACKUP STATE --- master1, master4
> Oct 11 14:20:00 master1 Entering BACKUP STATE --- master4
> --- Incident started; Another re-election; settle on master4 again
> --- Some nodes in the cluster can see the master VIP; others cannot
> Oct 11 14:39:09 master3 Entering MASTER STATE --- master3, master4
> Oct 11 14:39:11 master1 Entering MASTER STATE --- master1, master3,
> master4
> Oct 11 14:39:12 master0 Entering MASTER STATE --- master0, master1,
> master3, master4
> Oct 11 14:39:12 master2 Entering MASTER STATE --- master2, master0,
> master1, master3, master4
> Oct 11 14:39:13 master0 Entering BACKUP STATE --- master2, master1,
> master3, master4
> Oct 11 14:39:13 master1 Entering BACKUP STATE --- master2, master3,
> master4
> Oct 11 14:39:13 master2 Entering BACKUP STATE --- master3, master4
> Oct 11 14:39:13 master3 Entering BACKUP STATE --- master4

At 14:39:12, for all nodes to be in master state, none of them could
have seen an advert from any other node in master state for at least
3.39 seconds (i.e. 3 + 10${count}/256). So for example, at the time of
the first 14:39:12 message there were 4 nodes in master state, all
sending an advert every second, but master2 was seeing none of those
adverts. That it is happening at the same time on all the nodes suggests
that it is nothing to do with keepalived, nor with the nodes, but is
more likely the network connecting the nodes. For example, is there a
switch restarting, or are you running OSPF and a link state
recalculation is occurring?

> --- Forcibly restarted keepalived on master4; settle on master3; all
> nodes can see VIP
> Oct 11 17:03:26 master3 Entering MASTER STATE --- master3, master4
> Oct 11 17:03:26 master4 Entering BACKUP STATE --- master3

I think here the first message should not include master4 since it
ceased to be master when keepalived was stopped on it, at which point
master3, as the highest priority remaining router, took over as master.
That master4 doesn't attempt to take over as master once it restarts
shows that nopreempt is working.

> 
> Here is the keepalived config on each of the five nodes.  I am running
> keepalived 1.2.24 on CoreOS.
> 
> vrrp_instance master {
>   interface eth0
> 
>   # Must have state BACKUP if using noprempt
>   state BACKUP
> 
>   # Don't preference one node over another
>   # Whoever is currently master should stay master
>   # until they restart, etc.
>   nopreempt
> 
>   # Priority isn't important here but we'll give each master a custom
> priority
>   priority 10${count}
>   virtual_router_id 1
>   advert_int 1
>   authentication {
>       auth_type PASS
>       auth_pass master
>   }
>   virtual_ipaddress {
>       ${vip}/32
>   }
> }
> 
> I was able to restore connectivity by rebooting keepalived forcing a
> new re-election, resulting in new GARPs being sent.  Any ideas as to
> why some nodes could see the VIP and others couldn't?  

The scenario that you have given above indicates more than a loss of
connectivity to the VIPs. The VRRP advert messages are sent with a
source IP address of the primary address of (in your case) eth0, and
sent to the multicast address 224.0.0.18 and source MAC address
00-00-5e-00-01-01. It is these packets not being received by the nodes
running keepalived that is causing them to transition to master. Loss of
connectivity to the VIP is of course connectivity to a different IP
address.

The GARP messages that are sent when a keepalived instance transitions
to master is for the virtual IP address you have configured. Since all
the 5 nodes became master, they will all have sent GARP messages for the
same VIP from the same MAC address, and any switches in your network
could have become very confused about which port to send packets to that
are destined for the VIP. As the nodes sort themselves out and masters 0
to 3 revert to backup mode, no more gratuitous ARPs are sent because no
keepalived instance is transitioning to master mode. Stopping keepalived
on master4, and hence master3 becoming master and causing it to send
GARP messages will update the address cache in any switches.

> 
> I'm also concerned why the master gets re-elected in quick succession
> always resulting in master4.  master4 has the highest priority but I
> have noprempt set, so it shouldn't override as far as I understand.

In the scenario you have described above, nopreempt does not come into
play. nopreempt relates to a situation where a router is in backup mode,
and it detects a master that has a strictly lower priority than itself.
Without nopreempt, it would take over as master, with nopreempt it wont.
The situation you have is multiple masters, and what happens then is if
a master receives an advert that is either of higher priority, or of
equal priority but the source IP address is higher than it's own source
IP address, then that master will revert to backup mode. Since master4
has the highest priority, once the other masters start seeing its
adverts again, they will revert to backup mode due to being lower
priority.


>From the information you have provided, your problem really does appear
to be caused by something occurring in the network between your nodes,
which is temporarily causing VRRP adverts not to be received by the
keepalived instances, and following that the network infrastructure
becomes "confused".

keepalived does have an option to work around this issue, that I think
might allow keepalived to continue working after one of these
occurrences happens. In the global data section, add:
vrrp_garp_lower_prio_rep = 1    # or whatever number of GARP messages
you want to be sent; 5 would match the default behaviour when a
transition to master occurs
You could also add:
vrrp_garp_lower_prio_delay = 5 # to send a second set of GARP messages
after a delay of this number of seconds

These options cause gratuitous ARP messages to be sent by a keepalived
instance after receiving a lower priority advert, and are designed
precisely for the scenario you appear to be having, and have the
advantage over garp_master_refresh and garp_master_refresh_repeat of not
sending GARP messages all the time. However, if masters 0 to 3 all see
adverts from master4 before it sees adverts from any of masters 0 to 3,
then master4 won't send the GARP messages. This could be resolved by a
lower priority master, on receipt of a higher priority advert, if
garp_lower_prio_rep is configured, sending another advert before
reverting to backup mode, thereby ensuring that the higher priority
master knows it needs to send the GARP messages. I'll have a look at
this.

I hope this helps,

Quentin Armitage

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel