Re: vrrp process failed to die

Hello Quentin and thank you for your answer.
I changed the timeout to 20 seconds, and like before, it seems to match.

With 5sec timeout:
Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process failed to die - forcing termination


With 20sec timeout:
Feb  1 17:25:54 bz-perflab-6-vr Keepalived[33839]: Stopping
Feb  1 17:26:14 bz-perflab-6-vr Keepalived[33839]: vrrp process failed to die - forcing termination

From quick checks at logs it looks like I can remove 6 to 8 instances per second, more or less. This may be different on other systems.
Would it be possible to make the CHILD_WAIT_SECS initialized to (Number of instances / 4) seconds on Keepalived startup ?
We (OVH) can probably spend some time patching it ourselves then pushing the code to Github.

Regards,


On Thu, 2018-02-01 at 15:59 +0000, Quentin Armitage wrote:
> Damien,
> 
> What I think could be happening here is that syslog is rate limiting the messages to the log, and so some of them are being missed. If you run
> keepalived with the --log-file=FILE_NAME, then the log details will be written to the file, and those entries will not be rate limited. If you also
> want to not have messages logged via syslog, add the -G option. It might also be worth adding --flush-log-file if you are using the --log-file
> option, to make sure that no log entries are lost due to being buffered before keepalived dies.
> 
> The parent process, after it has sent a terminate signal, allows the child processes (vrrp and healthchecker) 5 seconds to terminate, and if that
> expires the parent assumes that the child has "got stuck" and so forcibly terminates it (SIGKILL). What I think is happening in your case is that
> the vrrp child process is not terminating within the 5 seconds, and so the parent process is sending it SIGKILL.
> 
> I think the first thing to try is extending the timeout. The is defined in keepalived/core/main.c - CHILD_WAIT_SECS. Could you try extending that
> until keepalived has time to remove all the vrrp interfaces (this is where the --log-file option will be really helpful so that you can see the rate
> at which interfaces are being deleted). If extending CHILD_WAIT_SECS works I will look at adding that as a configurable parameter.
> 
> If you could send to me the scripts that you are using to generate your configuration and the parameters you are using (or just your configuration
> file), I will see if I can reproduce the problem (it would also be useful to know the specification and speed of the processor(s) you have in you
> system so that I can get a feel for the rate at which interfaces can be deleted).
> 
> I'm happy to add a short delay between creating interfaces if that turns out to be the solution.
> 
> Quentin
> 
> On Thu, 2018-02-01 at 15:20 +0000, Damien Clabaut wrote:
> > Hello,
> > 
> > When running Keepalived with a lot of instances, I run into the following issue:
> > On shutdown, keepalived crashes and fails to remove all VRRP interfaces from system.
> > 
> > Logs:
> > Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stoppingb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) sent 0 priority
> > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) removing protocol VIPs.
> > (Same 2 lines for all instances)
> > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200) sent 0 priority
> > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200) removing protocol VIPs.
> > Feb  1 16:16:47 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac: Success removing VMAC interface vrrp4.2.1 for vrrp_instance 42
> > (Same line for several instances)
> > Feb  1 16:16:51 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac: Success removing VMAC interface vrrp4.14.1 for vrrp_instance 414
> > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process failed to die - forcing termination
> > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: Stopped Keepalived v2.0.0 (01/28,2018), git commit v1.4.1-9-g03cf9be
> > 
> > Has anyone else ran into this issue ?
> > Regards,
> > 
-- 
Damien Clabaut
R&D vRouter
OVH
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot