Re: vrrp process failed to die

Damien,

What I think could be happening here is that syslog is rate limiting
the messages to the log, and so some of them are being missed. If you
run keepalived with the --log-file=FILE_NAME, then the log details will
be written to the file, and those entries will not be rate limited. If
you also want to not have messages logged via syslog, add the -G
option. It might also be worth adding --flush-log-file if you are using
the --log-file option, to make sure that no log entries are lost due to
being buffered before keepalived dies.

The parent process, after it has sent a terminate signal, allows the
child processes (vrrp and healthchecker) 5 seconds to terminate, and if
that expires the parent assumes that the child has "got stuck" and so
forcibly terminates it (SIGKILL). What I think is happening in your
case is that the vrrp child process is not terminating within the 5
seconds, and so the parent process is sending it SIGKILL.

I think the first thing to try is extending the timeout. The is defined
in keepalived/core/main.c - CHILD_WAIT_SECS. Could you try extending
that until keepalived has time to remove all the vrrp interfaces (this
is where the --log-file option will be really helpful so that you can
see the rate at which interfaces are being deleted). If extending
CHILD_WAIT_SECS works I will look at adding that as a configurable
parameter.

If you could send to me the scripts that you are using to generate your
configuration and the parameters you are using (or just your
configuration file), I will see if I can reproduce the problem (it
would also be useful to know the specification and speed of the
processor(s) you have in you system so that I can get a feel for the
rate at which interfaces can be deleted).

I'm happy to add a short delay between creating interfaces if that
turns out to be the solution.

Quentin

On Thu, 2018-02-01 at 15:20 +0000, Damien Clabaut wrote:
> Hello,
> 
> When running Keepalived with a lot of instances, I run into the following issue:
> On shutdown, keepalived crashes and fails to remove all VRRP interfaces from system.
> 
> Logs:
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) sent 0 priority
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) removing protocol VIPs.
> (Same 2 lines for all instances)
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200) sent 0 priority
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200) removing protocol VIPs.
> Feb  1 16:16:47 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac: Success removing VMAC interface vrrp4.2.1 for vrrp_instance 42
> (Same line for several instances)
> Feb  1 16:16:51 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac: Success removing VMAC interface vrrp4.14.1 for vrrp_instance 414
> Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process failed to die - forcing termination
> Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: Stopped Keepalived v2.0.0 (01/28,2018), git commit v1.4.1-9-g03cf9be
> 
> Has anyone else ran into this issue ?
> Regards,
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel