Re: vrrp process failed to die

Damien,
It looks as though we've correctly identified to cause of the problem
at least.
I'm running the config generated from your scripts you sent before, and
with 196 vrrp instances my system is removing them in approx 3 seconds,
and removing 352 in 5 seconds when I have a config with 496 instances.
Based on these figures I would hate to make a permanent change to allow
0.25 seconds per vrrp instance. Also, the parent process at the moment
doesn't know the anything about the vrrp configuration, so wouldn't
know how long to allow.
I'm about to be busy for the next 1 1/2 hours, and then won't have much
time after that before going out for the evening. Could you for now
increase the timeout to make it large enough for your needs (probably
the number of instances / 4 would do). Later this evening I'll work on
a patch to allow the timeout to be manually configured, and I'll also
have a think to see if I can think of a more automated way.
What would be really useful for me to know is:1. How many vrrp
instances do you have configured?2. How quickly are the interfaces
created?3. Do you have any idea why the rate of interface creation is
so slow (at least relative to my system which is only an Intel Core i5
laptop at 2.3GHz).
I do have some ARMv5 processors running at 1.2GHz which will be a lot
slower, so I'll see what rate they can remove interfaces as well.
Quentin
On Thu, 2018-02-01 at 16:37 +0000, Damien Clabaut wrote:
> Hello Quentin and thank you for your answer.
> I changed the timeout to 20 seconds, and like before, it seems to
> match.
> 
> With 5sec timeout:
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process
> failed to die - forcing termination
> 
> 
> With 20sec timeout:
> Feb  1 17:25:54 bz-perflab-6-vr Keepalived[33839]: Stopping
> Feb  1 17:26:14 bz-perflab-6-vr Keepalived[33839]: vrrp process
> failed to die - forcing termination
> 
> From quick checks at logs it looks like I can remove 6 to 8 instances
> per second, more or less. This may be different on other systems.
> Would it be possible to make the CHILD_WAIT_SECS initialized to
> (Number of instances / 4) seconds on Keepalived startup ?
> We (OVH) can probably spend some time patching it ourselves then
> pushing the code to Github.
> 
> Regards,
> 
> 
> On Thu, 2018-02-01 at 15:59 +0000, Quentin Armitage wrote:
> > Damien,
> > 
> > What I think could be happening here is that syslog is rate
> > limiting the messages to the log, and so some of them are being
> > missed. If you run
> > keepalived with the --log-file=FILE_NAME, then the log details will
> > be written to the file, and those entries will not be rate limited.
> > If you also
> > want to not have messages logged via syslog, add the -G option. It
> > might also be worth adding --flush-log-file if you are using the --
> > log-file
> > option, to make sure that no log entries are lost due to being
> > buffered before keepalived dies.
> > 
> > The parent process, after it has sent a terminate signal, allows
> > the child processes (vrrp and healthchecker) 5 seconds to
> > terminate, and if that
> > expires the parent assumes that the child has "got stuck" and so
> > forcibly terminates it (SIGKILL). What I think is happening in your
> > case is that
> > the vrrp child process is not terminating within the 5 seconds, and
> > so the parent process is sending it SIGKILL.
> > 
> > I think the first thing to try is extending the timeout. The is
> > defined in keepalived/core/main.c - CHILD_WAIT_SECS. Could you try
> > extending that
> > until keepalived has time to remove all the vrrp interfaces (this
> > is where the --log-file option will be really helpful so that you
> > can see the rate
> > at which interfaces are being deleted). If extending
> > CHILD_WAIT_SECS works I will look at adding that as a configurable
> > parameter.
> > 
> > If you could send to me the scripts that you are using to generate
> > your configuration and the parameters you are using (or just your
> > configuration
> > file), I will see if I can reproduce the problem (it would also be
> > useful to know the specification and speed of the processor(s) you
> > have in you
> > system so that I can get a feel for the rate at which interfaces
> > can be deleted).
> > 
> > I'm happy to add a short delay between creating interfaces if that
> > turns out to be the solution.
> > 
> > Quentin
> > 
> > On Thu, 2018-02-01 at 15:20 +0000, Damien Clabaut wrote:
> > > Hello,
> > > 
> > > When running Keepalived with a lot of instances, I run into the
> > > following issue:
> > > On shutdown, keepalived crashes and fails to remove all VRRP
> > > interfaces from system.
> > > 
> > > Logs:
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stoppingb  1
> > > 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) sent
> > > 0 priority
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42)
> > > removing protocol VIPs.
> > > (Same 2 lines for all instances)
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200)
> > > sent 0 priority
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200)
> > > removing protocol VIPs.
> > > Feb  1 16:16:47 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac:
> > > Success removing VMAC interface vrrp4.2.1 for vrrp_instance 42
> > > (Same line for several instances)
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac:
> > > Success removing VMAC interface vrrp4.14.1 for vrrp_instance 414
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process
> > > failed to die - forcing termination
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: Stopped
> > > Keepalived v2.0.0 (01/28,2018), git commit v1.4.1-9-g03cf9be
> > > 
> > > Has anyone else ran into this issue ?
> > > Regards,
> > >

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel