Re: Loss of routes to VIP

This helped a lot in understanding the issue.

The issue wasn’t, as you say, that the GARPs got lost, it’s that the VRRP adverts go lost and the original master never re-announced himself after a chaotic election.

From everything I can tell using the garp_master_refresh would have kicked things back in gear. I know it’s a bit of a sledgehammer but if it makes me sleep at night… Any thoughts on reasonable values here? Does a periodic of once every ten seconds seem sane? That is, am I going to annoy any routers/switches with that kind of periodic?

Question: why does the default config send multiple GARPs back to back? When transitioning to master, I believe the default is to send 5 GARPs. Why not 1?





On October 14, 2016 at 5:05:27 AM, Quentin Armitage ([email protected]) wrote:

On Thu, 2016-10-13 at 17:52 +0000, Matt Hughes wrote:

I have five nodes in a cluster that share the same keepalived VIP.  I have 50 nodes that use this VIP.  Three times in the last month, an reelection has resulted in some of those 50 nodes not having a route to the VIP.  All the nodes are on the same subnet. 

--- settle on master4; what changed?  Why did we re-elect?
Oct 11 14:19:49 master3 Entering MASTER STATE --- master3, master4
Oct 11 14:19:52 master3 Entering BACKUP STATE --- master4
Oct 11 14:19:53 master0 Entering MASTER STATE --- master0, master4
Oct 11 14:19:53 master1 Entering MASTER STATE --- master1, master0, master4
Oct 11 14:19:53 master2 Entering MASTER STATE --- master2, master1, master0, master4
Oct 11 14:19:55 master2 Entering BACKUP STATE --- master1, master0, master4
Oct 11 14:20:00 master0 Entering BACKUP STATE --- master1, master4
Oct 11 14:20:00 master1 Entering BACKUP STATE --- master4
--- Incident started; Another re-election; settle on master4 again
--- Some nodes in the cluster can see the master VIP; others cannot
Oct 11 14:39:09 master3 Entering MASTER STATE --- master3, master4
Oct 11 14:39:11 master1 Entering MASTER STATE --- master1, master3, master4
Oct 11 14:39:12 master0 Entering MASTER STATE --- master0, master1, master3, master4
Oct 11 14:39:12 master2 Entering MASTER STATE --- master2, master0, master1, master3, master4
Oct 11 14:39:13 master0 Entering BACKUP STATE --- master2, master1, master3, master4
Oct 11 14:39:13 master1 Entering BACKUP STATE --- master2, master3, master4
Oct 11 14:39:13 master2 Entering BACKUP STATE --- master3, master4
Oct 11 14:39:13 master3 Entering BACKUP STATE --- master4
At 14:39:12, for all nodes to be in master state, none of them could have seen an advert from any other node in master state for at least 3.39 seconds (i.e. 3 + 10${count}/256). So for example, at the time of the first 14:39:12 message there were 4 nodes in master state, all sending an advert every second, but master2 was seeing none of those adverts. That it is happening at the same time on all the nodes suggests that it is nothing to do with keepalived, nor with the nodes, but is more likely the network connecting the nodes. For example, is there a switch restarting, or are you running OSPF and a link state recalculation is occurring?
--- Forcibly restarted keepalived on master4; settle on master3; all nodes can see VIP
Oct 11 17:03:26 master3 Entering MASTER STATE --- master3, master4
Oct 11 17:03:26 master4 Entering BACKUP STATE --- master3
I think here the first message should not include master4 since it ceased to be master when keepalived was stopped on it, at which point master3, as the highest priority remaining router, took over as master. That master4 doesn't attempt to take over as master once it restarts shows that nopreempt is working.

Here is the keepalived config on each of the five nodes.  I am running keepalived 1.2.24 on CoreOS.

vrrp_instance master {
  interface eth0

  # Must have state BACKUP if using noprempt
  state BACKUP

  # Don't preference one node over another
  # Whoever is currently master should stay master
  # until they restart, etc.
  nopreempt

  # Priority isn't important here but we'll give each master a custom priority
  priority 10${count}
  virtual_router_id 1
  advert_int 1
  authentication {
      auth_type PASS
      auth_pass master
  }
  virtual_ipaddress {
      ${vip}/32
  }
}

I was able to restore connectivity by rebooting keepalived forcing a new re-election, resulting in new GARPs being sent.  Any ideas as to why some nodes could see the VIP and others couldn't? 
The scenario that you have given above indicates more than a loss of connectivity to the VIPs. The VRRP advert messages are sent with a source IP address of the primary address of (in your case) eth0, and sent to the multicast address 224.0.0.18 and source MAC address 00-00-5e-00-01-01. It is these packets not being received by the nodes running keepalived that is causing them to transition to master. Loss of connectivity to the VIP is of course connectivity to a different IP address.

The GARP messages that are sent when a keepalived instance transitions to master is for the virtual IP address you have configured. Since all the 5 nodes became master, they will all have sent GARP messages for the same VIP from the same MAC address, and any switches in your network could have become very confused about which port to send packets to that are destined for the VIP. As the nodes sort themselves out and masters 0 to 3 revert to backup mode, no more gratuitous ARPs are sent because no keepalived instance is transitioning to master mode. Stopping keepalived on master4, and hence master3 becoming master and causing it to send GARP messages will update the address cache in any switches.

I'm also concerned why the master gets re-elected in quick succession always resulting in master4.  master4 has the highest priority but I have noprempt set, so it shouldn't override as far as I understand.
In the scenario you have described above, nopreempt does not come into play. nopreempt relates to a situation where a router is in backup mode, and it detects a master that has a strictly lower priority than itself. Without nopreempt, it would take over as master, with nopreempt it wont.
The situation you have is multiple masters, and what happens then is if a master receives an advert that is either of higher priority, or of equal priority but the source IP address is higher than it's own source IP address, then that master will revert to backup mode. Since master4 has the highest priority, once the other masters start seeing its adverts again, they will revert to backup mode due to being lower priority.
From the information you have provided, your problem really does appear to be caused by something occurring in the network between your nodes, which is temporarily causing VRRP adverts not to be received by the keepalived instances, and following that the network infrastructure becomes "confused".

keepalived does have an option to work around this issue, that I think might allow keepalived to continue working after one of these occurrences happens. In the global data section, add:
vrrp_garp_lower_prio_rep = 1    # or whatever number of GARP messages you want to be sent; 5 would match the default behaviour when a transition to master occurs
You could also add:
vrrp_garp_lower_prio_delay = 5 # to send a second set of GARP messages after a delay of this number of seconds

These options cause gratuitous ARP messages to be sent by a keepalived instance after receiving a lower priority advert, and are designed precisely for the scenario you appear to be having, and have the advantage over garp_master_refresh and garp_master_refresh_repeat of not sending GARP messages all the time. However, if masters 0 to 3 all see adverts from master4 before it sees adverts from any of masters 0 to 3, then master4 won't send the GARP messages. This could be resolved by a lower priority master, on receipt of a higher priority advert, if garp_lower_prio_rep is configured, sending another advert before reverting to backup mode, thereby ensuring that the higher priority master knows it needs to send the GARP messages. I'll have a look at this.

I hope this helps,

Quentin Armitage

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel