Re: "clone" syscall in keepalived healthchecking child takes more time after switching

Damien,

The track_scripts are invoked from function vrrp_script_thread() in
vrrp_scheduler.c, and the following function vrrp_script_child_thread()
handles the termination of the scripts. The calls from there should be
quite straight forward to follow, but if not by all means ask any
further questions.

I don't think it is relevant to what you are looking at but the notify
scripts are invoked via notify_exec() in notify.c

I have a version that is planned to be made available as a beta in the
near future. This has a large number of performance enhancements,
including some in the area of executing track scripts. The updated code
is available at: https://github.com/pqarmitage/keepalived/commits/fixes
.quick , and I would be very interested to hear if makes any
improvements for you. The enhancements that may be relevant to what you
are doing are:
1. Use execve() instead of system() to invoke scripts, with any
parameters parsed at initialisation rather than when the script is run
(my understanding is that this changes it from 2 forks to 1 fork per
script invocation)
2. Ability to use track_files instead of track_scripts. keepalived uses
inotify() to monitor changes to files, and reads a number from the file
when the it changes, which is treated in the same way that the exit
code from a track_script is used. This means that scripts do not need
to be executed on a periodic basis, but rather when an external event
occurs, if the track_file is written to, keepalived will immediately
read the file and update the vrrp instances accordingly.
3. In relation to notify scripts, keepalived can write to a FIFO
instead of executing a notify script. A process can then monitor the
FIFO for any notifications (this has the side benefit that it can be
guaranteed that notifications can be processed in the correct
sequence). keepalived can be configured to execute a script to monitor
the FIFO to initiate any desired actions, just as though a notify
script had been executed, but again it saves the overhead of executing
a script.

I hope this helps, and if you do try the code that is soon to be
released as beta, I would be very glad to hear any feedback you have in
relation to it.

Regards,

Quentin

On Mon, 2017-10-02 at 15:16 -0400, Damien Clabaut wrote:
> Hello again Quentin,
> 
> I played a bit more with the templates and with strace.
> 
> As it turns out:
> 
> - One clone call means one vrrp script is launched. If I have 16 
> instances, each launching a vrrp script every 5 seconds, I get 32
> clone 
> calls over a 10 second period.
> 
> - The more instances, the longer each clone call (see attached: 
> usec_per_instance_count.png)
> 
> - The more switch (master->backup->master), the longer each clone
> call 
> (see attached: usec_per_vrrp_switch.png)
> 
> The graphs are pretty poor quality since this is a ML and I don't
> want 
> to pollute everyone's inboxes.
> 
> Could you point me to the part of the code which calls all the 
> vrrp_scripts ? I would like to study it and see if I can make some 
> progress on it.
> 
> Regards,
> 
> 
> On 2017-10-02 11:40 AM, Damien Clabaut wrote:
> > 
> > 
> > On 2017-10-02 11:25 AM, Damien Clabaut wrote:
> > > Hello Quentin and thank you very much for your answer,
> > > 
> > > My setup is a bit compilcated and built on top of Keepalived PR
> > > for 
> > > json output (https://github.com/acassen/keepalived/pull/633).
> > > 
> > > I will try to describe it as clearly as possible here, please
> > > tell me 
> > > if this is not clear.
> > > 
> > > 
> > > The network configuration is as follows:
> > > 
> > > I have 2 Linux machines connected to each other (through a switch
> > > in 
> > > my case, but that is not relevant).
> > > 
> > > Using attached templates, I create N vlans interfaces on each
> > > side of 
> > > the link connecting both servers (from 2 to N+1).
> > > 
> > > On top of each of those vlan interfaces I make 2 keepalived
> > > instances 
> > > (one in version 2 for IPv4, one in version 3 for IPv6)
> > > 
> > > instances are names: <ip_version><vlan id>. For example, IPv6 
> > > instance on vlan 42 is called 642.
> > > 
> > > 
> > > As you can see in the attached templates, each instances is
> > > tracking 
> > > a different script called offset_instance_<instance name>
> > > 
> > > All those track_scripts launch the (attached) script 
> > > "/usr/bin/get_vrrp_offset <instance name>"
> > > 
> > > get_vrrp_offset queries a JSON file (which is in a ramdisk for 
> > > performance), which contains a boolean for each VRRP instance.
> > > 
> > > If the boolean is 1, then the script returns 1, and the VRRP
> > > priority 
> > > for this instance is lowered by 100.
> > > 
> > > In any other case (JSON file not present or not valid, data not 
> > > present,...) the script returns 0 and nothing happens.
> > > 
> > > 
> > > The json script is populated with the (attached)
> > > "/usr/bin/isolate" 
> > > script.
> > > 
> > > Ran without arguments, it will set the boolean in the json file
> > > to 
> > > "1" for all existing instances (collected from the json output).
> > > 
> > > Ran with "-r", it will do the opposite.
> > > 
> > > The interesting point is, it will change 1 instance every
> > > second, 
> > > which allows us to move a large number of floating IPs without 
> > > interruption.
> > > 
> > > 
> > > Please find attached:
> > > 
> > > - /usr/bin/isolate
> > > 
> > > - /usr/bin/get_vrrp_offset
> > > 
> > > - /usr/bin/vrrp_recap (This will print a table with data from
> > > the 
> > > json output)
> > > 
> > > - make_conf.py (this will generate the keepalived config)
> > > 
> > > - template_vrrp_instance, which is called by make_conf.py
> > > 
> > > 
> > > To mount the ramdisk:
> > > 
> > > mkdir /mnt/ramdisk
> > > 
> > > mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
> > > 
> > > I will keep working on this today, and keep you updated on my
> > > findings.
> > > 
> > > Regards,
> > > 
> > > 
> > > On 2017-10-01 10:00 AM, Quentin Armitage wrote:
> > > > On Thu, 2017-09-28 at 16:02 -0400, Damien Clabaut wrote:
> > > > > Hello Keepalived devs,
> > > > > 
> > > > > While testing Keepalived version 1.3.6 on Debian8, I came
> > > > > accross a
> > > > > strange issue.
> > > > > 
> > > > > Tl;DR: It seems that after every vrrp state change, the
> > > > > "clone" system
> > > > > call takes a little more time, slowly eating up all CPU.
> > > > > 
> > > > > Details:
> > > > > 
> > > > > I start keepalived with 200 instances (100 IPv4 + 100 IPv6).
> > > > > Each instance tracks a different script.
> > > > > The keepalived healthcheck children (as described on
> > > > > http://keepalived.org/documentation.html) has PID 177548:
> > > > > 
> > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] echo $(($(cat
> > > > > /var/run/keepalived.pid) + 2))
> > > > > 177458
> > > > > 
> > > > > All instances are in the MASTER state on this machine. Strace
> > > > > -c gives
> > > > > me the count of system calls on a 60 second time period:
> > > > > 
> > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
> > > > > strace 
> > > > > -cp 177458
> > > > > % time     seconds  usecs/call     calls    errors syscall
> > > > > ------ ----------- ----------- --------- --------- ----------
> > > > > ------
> > > > >    84.96    2.676412         880      3041       689 clone
> > > > > 
> > > > > Using vrrp_scripts, I move all instances to BACKUP state,
> > > > > then back to
> > > > > MASTER state, and run the same command
> > > > > 
> > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
> > > > > strace 
> > > > > -cp 177458
> > > > > % time     seconds  usecs/call     calls    errors syscall
> > > > > ------ ----------- ----------- --------- --------- ----------
> > > > > ------
> > > > >    97.85   15.866534        4887      3247       901 clone
> > > > > 
> > > > > 
> > > > > Again, I move all instances to BACKUP then MASTER, and take a
> > > > > trace.
> > > > > 
> > > > > There are about the same number of "clone" syscalls, but now
> > > > > they take
> > > > > 8651 usecs/call
> > > > > 
> > > > > To make sure this a consequence of state change, I wait 10
> > > > > then 20
> > > > > minutes and take new traces.
> > > > > 
> > > > > There is still the same number of "clone" syscalls, and now
> > > > > they take
> > > > > 7440 usecs/call.
> > > > > 
> > > > > On a separate experiment where I kept changing the VRRP
> > > > > state, I 
> > > > > saw an
> > > > > entire core being used by this process.
> > > > > 
> > > > > Is this a known issue?
> > > > > 
> > > > > What can I do to help fix it?
> > > > > 
> > > > > Kind regards,
> > > > > 
> > > > 
> > > > Hi Damian,
> > > > 
> > > > This sounds an interesting problem. In the first place could
> > > > you please
> > > > provide a copy of your configuration files (presumably from at
> > > > least
> > > > two system  as close as possible to what you are running, and
> > > > also
> > > > copies of the scripts. I'll then see if I can reproduce the
> > > > problem in
> > > > a Debian VM, and if so see if other distros exhibit the same
> > > > problem.
> > > > I'll also have a look so see if my latest development version,
> > > > which
> > > > has quite a few changes in the way scripts are run, also
> > > > exhibits the
> > > > same problem.
> > > > 
> > > > It would be helpful if you could explain what you do to force
> > > > all the
> > > > instances into backup and then back to master, and over what
> > > > timescale.
> > > > 
> > > > At the moment I haven't a clue what the problem might be.
> > > > 
> > > > Could you write a program that could issue a few hundred
> > > > clone() system
> > > > calls, and see what happens to the performance of those around
> > > > the time
> > > > you see the different results for keepalived?
> > > > 
> > > > BTW, echo $(($(cat /var/run/keepalived.pid) + 2)) isn't a
> > > > reliable way
> > > > of getting the pid of the keepalived_vrrp process. I sometimes
> > > > see the
> > > > keepalived parent and children processes having consecutive
> > > > pids, and
> > > > sometimes there are gaps of one or two in between. Better to
> > > > use
> > > > vrrp.pid.
> > > > 
> > > > Many thanks for reporting this issue.
> > > > 
> > > > Quentin Armitage

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot