Loss of routes to VIP

I have five nodes in a cluster that share the same keepalived VIP.  I have
50 nodes that use this VIP.  Three times in the last month, an reelection
has resulted in some of those 50 nodes not having a route to the VIP.  All
the nodes are on the same subnet.

--- settle on master4; what changed?  Why did we re-elect?
Oct 11 14:19:49 master3 Entering MASTER STATE --- master3, master4
Oct 11 14:19:52 master3 Entering BACKUP STATE --- master4
Oct 11 14:19:53 master0 Entering MASTER STATE --- master0, master4
Oct 11 14:19:53 master1 Entering MASTER STATE --- master1, master0, master4
Oct 11 14:19:53 master2 Entering MASTER STATE --- master2, master1,
master0, master4
Oct 11 14:19:55 master2 Entering BACKUP STATE --- master1, master0, master4
Oct 11 14:20:00 master0 Entering BACKUP STATE --- master1, master4
Oct 11 14:20:00 master1 Entering BACKUP STATE --- master4
--- Incident started; Another re-election; settle on master4 again
--- Some nodes in the cluster can see the master VIP; others cannot
Oct 11 14:39:09 master3 Entering MASTER STATE --- master3, master4
Oct 11 14:39:11 master1 Entering MASTER STATE --- master1, master3, master4
Oct 11 14:39:12 master0 Entering MASTER STATE --- master0, master1,
master3, master4
Oct 11 14:39:12 master2 Entering MASTER STATE --- master2, master0,
master1, master3, master4
Oct 11 14:39:13 master0 Entering BACKUP STATE --- master2, master1,
master3, master4
Oct 11 14:39:13 master1 Entering BACKUP STATE --- master2, master3, master4
Oct 11 14:39:13 master2 Entering BACKUP STATE --- master3, master4
Oct 11 14:39:13 master3 Entering BACKUP STATE --- master4
--- Forcibly restarted keepalived on master4; settle on master3; all nodes
can see VIP
Oct 11 17:03:26 master3 Entering MASTER STATE --- master3, master4
Oct 11 17:03:26 master4 Entering BACKUP STATE --- master3

Here is the keepalived config on each of the five nodes.  I am running
keepalived 1.2.24 on CoreOS.

vrrp_instance master {
  interface eth0

  # Must have state BACKUP if using noprempt
  state BACKUP

  # Don't preference one node over another
  # Whoever is currently master should stay master
  # until they restart, etc.
  nopreempt

  # Priority isn't important here but we'll give each master a custom
priority
  priority 10${count}
  virtual_router_id 1
  advert_int 1
  authentication {
      auth_type PASS
      auth_pass master
  }
  virtual_ipaddress {
      ${vip}/32
  }
}

I was able to restore connectivity by rebooting keepalived forcing a new
re-election, resulting in new GARPs being sent.  Any ideas as to why some
nodes could see the VIP and others couldn't?

I'm also concerned why the master gets re-elected in quick succession
always resulting in master4.  master4 has the highest priority but I have
noprempt set, so it shouldn't override as far as I understand.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel