Re: vrrp process failed to die

Damien,
I have done some further work on the rate at which interfaces can be
created and deleted. There is now a configure option --enable-netlink-
timers that will report the number of calls and total time in the calls
for each netlink command. What is reported as command 0 is the actual
interface creation commands, command 16 as the RTM_NEWLINK messages
that are updating the link status, command 17 is RTM_DELLINK - see
/usr/include/linux/rtnetlink.h for the other values.
What I have found on my system is that it takes 0.2ms to create an
interface, but 10ms to delete an interface (i.e. 50 times as long).
From the figures you reported below it seems as though it is taking
about 120ms to delete an interface on the system you are using.
Can you work with the new parameter child_wait_time for now? Given the
variation between our systems in how long it takes to delete an
interface, I think it makes it quite hard to calculate automatically
how long should be allowed, although I suppose we could try creating
and deleting an interface to see how long it takes, but we still have
the problem of the parent (at the moment) not knowing how many
interfaces have been created.
Quentin
		On Thu, 2018-02-01 at 16:37 +0000, Damien Clabaut
wrote:
> Hello Quentin and thank you for your answer.
> I changed the timeout to 20 seconds, and like before, it seems to
> match.
> 
> With 5sec timeout:
> Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process
> failed to die - forcing termination
> 
> 
> With 20sec timeout:
> Feb  1 17:25:54 bz-perflab-6-vr Keepalived[33839]: Stopping
> Feb  1 17:26:14 bz-perflab-6-vr Keepalived[33839]: vrrp process
> failed to die - forcing termination
> 
> From quick checks at logs it looks like I can remove 6 to 8 instances
> per second, more or less. This may be different on other systems.
> Would it be possible to make the CHILD_WAIT_SECS initialized to
> (Number of instances / 4) seconds on Keepalived startup ?
> We (OVH) can probably spend some time patching it ourselves then
> pushing the code to Github.
> 
> Regards,
> 
> 
> On Thu, 2018-02-01 at 15:59 +0000, Quentin Armitage wrote:
> > Damien,
> > 
> > What I think could be happening here is that syslog is rate
> > limiting the messages to the log, and so some of them are being
> > missed. If you run
> > keepalived with the --log-file=FILE_NAME, then the log details will
> > be written to the file, and those entries will not be rate limited.
> > If you also
> > want to not have messages logged via syslog, add the -G option. It
> > might also be worth adding --flush-log-file if you are using the --
> > log-file
> > option, to make sure that no log entries are lost due to being
> > buffered before keepalived dies.
> > 
> > The parent process, after it has sent a terminate signal, allows
> > the child processes (vrrp and healthchecker) 5 seconds to
> > terminate, and if that
> > expires the parent assumes that the child has "got stuck" and so
> > forcibly terminates it (SIGKILL). What I think is happening in your
> > case is that
> > the vrrp child process is not terminating within the 5 seconds, and
> > so the parent process is sending it SIGKILL.
> > 
> > I think the first thing to try is extending the timeout. The is
> > defined in keepalived/core/main.c - CHILD_WAIT_SECS. Could you try
> > extending that
> > until keepalived has time to remove all the vrrp interfaces (this
> > is where the --log-file option will be really helpful so that you
> > can see the rate
> > at which interfaces are being deleted). If extending
> > CHILD_WAIT_SECS works I will look at adding that as a configurable
> > parameter.
> > 
> > If you could send to me the scripts that you are using to generate
> > your configuration and the parameters you are using (or just your
> > configuration
> > file), I will see if I can reproduce the problem (it would also be
> > useful to know the specification and speed of the processor(s) you
> > have in you
> > system so that I can get a feel for the rate at which interfaces
> > can be deleted).
> > 
> > I'm happy to add a short delay between creating interfaces if that
> > turns out to be the solution.
> > 
> > Quentin
> > 
> > On Thu, 2018-02-01 at 15:20 +0000, Damien Clabaut wrote:
> > > Hello,
> > > 
> > > When running Keepalived with a lot of instances, I run into the
> > > following issue:
> > > On shutdown, keepalived crashes and fails to remove all VRRP
> > > interfaces from system.
> > > 
> > > Logs:
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stoppingb  1
> > > 16:16:46 bz-perflab-6-vr Keepalived[28725]: Stopping
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42) sent
> > > 0 priority
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (42)
> > > removing protocol VIPs.
> > > (Same 2 lines for all instances)
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200)
> > > sent 0 priority
> > > Feb  1 16:16:46 bz-perflab-6-vr Keepalived_vrrp[28726]: (6200)
> > > removing protocol VIPs.
> > > Feb  1 16:16:47 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac:
> > > Success removing VMAC interface vrrp4.2.1 for vrrp_instance 42
> > > (Same line for several instances)
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived_vrrp[28726]: vmac:
> > > Success removing VMAC interface vrrp4.14.1 for vrrp_instance 414
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: vrrp process
> > > failed to die - forcing termination
> > > Feb  1 16:16:51 bz-perflab-6-vr Keepalived[28725]: Stopped
> > > Keepalived v2.0.0 (01/28,2018), git commit v1.4.1-9-g03cf9be
> > > 
> > > Has anyone else ran into this issue ?
> > > Regards,
> > >

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel