Re: Question about VIPs dropping from keepalived pair

Well the master documentation source for keepalived is here https://github.com/acassen/keepalived/blob/master/doc/keepalived.conf.SYNOPSIS

Ignore all other howtos and blogs on the Internet because they are almost all a decade or more out of date. I know this sounds a little harsh but it also helps to red the VRRP RFC's because they explain a lot about the intent of these options, that said the are a relatively short read as far as RFC's go.

In that doc read about the authentication section of a vrrp instance it will explain how the passwords works.

Keep in mind the passwords are not encrypted they are in clear text because they are not meant to secure VRRP they are simply meant to prevent ‎accidental conflicts.

The best practice is to use dedicated crossover cables or vlans for vrrp ‎communications. The reason for this is the design in the RFC'S favors speed over security and was really only ever meant to be used on isolated networks. The way to achieve this is to ust the interface and track interface settings aling with specifying the device on the VIP. The vip uses iproute2 syntax so take averythig "ip addr add" on the command-line and plug that in to the VIP.

From: Chris

Sent: Thursday, October 1, 2015 22:17

To: [email protected]

Cc: [email protected]

Subject: Re: [Keepalived-devel] Question about VIPs dropping from keepalived pair

prmarino1,

Thank you very much for responding. I really do appreciate it. I will work on setting the priorities to be something of a gap more like of 10 rather than 100, and then work on setting up a chart to create unique virtual router id's instead of them all having the same. We'll run some tests and increase the number between priorities to see if we can cause this condition to happen on purpose, and then a series of other tests.

Can you please explain to me, you mentioned having a unique password set on each pair also - does this allow the pair of servers to .. securely talk to one another with this password so that some other external source (another pair of keepalived hosts within the same site) can't affect it? I am curious about the passwords aspect of keepalived and I'll do some reading on this issue, but it always helps tremendously when someone experienced is able to comment and supplement what documentation iterates.

Take care and thanks again for your time,

Chris

-Chris

On Thu, Oct 1, 2015 at 7:02 PM, <[email protected] > wrote:

On the same router ID yes that can absolutely cause a problem, and you should also have a uniqe password set on each pair too.

The interval shouldn't cause a problem and isn't unusual, in fact you are more likely to have issues with smaller intervals than larger ones especially if you use one of the centisecond interval patches.

I've run keepalived on openstack before without issue; however without knowing the details of your network I can not rule out network config issues.

One thing I noticed is that difference in priorities is way too big. The difference should be less than 50. Most likely this contributes to your issue because the priority number is used in the calculations of timeouts. Also a difference of more than 50 constitutes a hard failover to the one with the highest priority regardless of nopreempt and preempt delay settings. Both of these things are specified in the VRRP rfc's and are not specific to keepalived.

From: Chris

Sent: Thursday, October 1, 2015 21:05

To: [email protected]

Subject: [Keepalived-devel] Question about VIPs dropping from keepalived pair

Hello,

I am happy to be a part of this list now, and I apologize ahead of time if this is not the correct list to post my question to about some odd behavior from keepalived configured on our systems.

Specs: Ubuntu 14.04 LTS, instances running in Mirantis OpenStack

Keepalived version: Keepalived v1.2.19 (08/25,2015)

The problem: VIP address randomly gets dropped requiring a restart of keepalived to pick it back up again

Description: We had been running our systems out of AWS for the longest times without this happening. We have recently moved to an OpenStack solution and have a pair of haproxy servers which answer requests for a slew of external facing webapps. We have haproxy servers for both internal and external services, and this has happened on all of our keepalived pairs, regardless of what services they serve.

I have scowered the net trying to find someone with a similar situation as this and have come up empty handed so I thought to ask the all knowing community.

A couple of points that I have noticed within our keepalived.conf files:

- We have advert_int set to 10. I've read that the default value is 1. I don't know why whoever configured these prior had this value set to 10 and I am wondering what kinds of problems can occur if this value is set too high?

- I have also noticed that the virtual_router_id is set to the same integer for all of our pairs of keepalived servers, again, I don't know why it was configured this way, but I'm curious what kind of problems this can cause if any with this being the same value.

Keepalived.conf configs:

MASTER

root@haproxy-external-02:~# cat /etc/keepalived/keepalived.conf

global_defs {

router_id haproxy-external-02

}

vrrp_instance external_multi_app { #set to cluster unique id

state MASTER #primary peer(s) should be set to MASTER

priority 200 #set to 200 if master

unicast_src_ip 10.152.146.7

unicast_peer {

10.152.146.5

}

#vrrp_unicast_peer 10.152.146.5

#nopreempt #will only work if both are set to BACKUP

interface eth0

#vrrp_unicast_bind 10.152.146.7

virtual_router_id 42 #unique router id

advert_int 10

virtual_ipaddress {

10.152.146.91

}

}

BACKUP

root@haproxy-external-01:~# cat /etc/keepalived/keepalived.conf

global_defs {

router_id haproxy-external-01

}

vrrp_instance external_multi_app { #set to cluster unique id

state BACKUP #secondary peer(s) should be set to BACKUP

priority 100 #set to 100 if backup

unicast_src_ip 10.152.146.5

unicast_peer {

10.152.146.7

}

#vrrp_unicast_peer 10.152.146.7

#nopreempt #will only work if both are set to BACKUP

interface eth0

#vrrp_unicast_bind 10.152.146.5

virtual_router_id 42 #unique router id

advert_int 10

virtual_ipaddress {

10.152.146.91

}

}

Summary: I just am not sure if this is an OpenStack related problem, or if keepalived is misconfigured in a way that not having enough advertisements could cause the VIP to just go away. I do know also, that this symptom occurred when we have had to reboot our switch, the VIPs would drop off and restarting keepalived was required - which is kind of expected, I guess. But seeing this happen while the switch is up and running is bizarre to me and I am having a hell of a time trying to recreate it, aside from having constant straces running on the PIDs with the hopes that it will happen, I have not been that luck yet.

I appreciate your time and I am hoping that perhaps we have just misconfigured the conf file, or if someone else has perhaps come across this situation of VIPs just dropping. When it happens, of course it is an outage for all services attached to these haproxy servers so its quite bad when this happens.

Thank you very much again,

Sincerely,

Chris

------------------------------------------------------------------------------

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel