Re: "clone" syscall in keepalived healthchecking child takes more time after switching

Damien,

Re safety reading the file, yes it does include the same safety. It
uses strtol(buf, NULL, 0) to read from the beginning of the file, and
that will return 0 if the file does not start with a valid number. If
the file is not present it assumes 0.

The code is in function process_track_file() in vrrp_track.c.

The same track files can be used for different vrrp instances but
obviously then the vrrp instances will all react to the same trigger,
so if you want to manage each vrrp instance independantly then they
fill each need a separate file.

To replicate your functionality of writing 1 to a file for reducing the
priority by 100, then you want to specify weight -100 for each track
file.

Do let me know how you get on.

Quentin

On Tue, 2017-10-03 at 09:45 -0400, Damien Clabaut wrote:
> Quentin,
> 
> If I understand your point 2. correctly, what I am doing today with
> a 
> json file read by a python script could be handled by keepalived 
> directly, provided I write in one separate file per instance ?
> 
> This would be awesome. Does it include the same safety I have on
> Python 
> script (if the file is not present or does not contain expected
> data, 
> then it is seen as a 0) ?
> 
> I will try adding my patch (json output) to your branch. It looks
> like 
> it conflicts, but nothing unsolvable IMHO.
> 
> I will keep you updated.
> 
> Regards,
> 
> 
> On 2017-10-02 03:46 PM, Quentin Armitage wrote:
> > Damien,
> > 
> > The track_scripts are invoked from function vrrp_script_thread() in
> > vrrp_scheduler.c, and the following function
> > vrrp_script_child_thread()
> > handles the termination of the scripts. The calls from there should
> > be
> > quite straight forward to follow, but if not by all means ask any
> > further questions.
> > 
> > I don't think it is relevant to what you are looking at but the
> > notify
> > scripts are invoked via notify_exec() in notify.c
> > 
> > I have a version that is planned to be made available as a beta in
> > the
> > near future. This has a large number of performance enhancements,
> > including some in the area of executing track scripts. The updated
> > code
> > is available at: https://github.com/pqarmitage/keepalived/commits/f
> > ixes
> > .quick , and I would be very interested to hear if makes any
> > improvements for you. The enhancements that may be relevant to what
> > you
> > are doing are:
> > 1. Use execve() instead of system() to invoke scripts, with any
> > parameters parsed at initialisation rather than when the script is
> > run
> > (my understanding is that this changes it from 2 forks to 1 fork
> > per
> > script invocation)
> > 2. Ability to use track_files instead of track_scripts. keepalived
> > uses
> > inotify() to monitor changes to files, and reads a number from the
> > file
> > when the it changes, which is treated in the same way that the exit
> > code from a track_script is used. This means that scripts do not
> > need
> > to be executed on a periodic basis, but rather when an external
> > event
> > occurs, if the track_file is written to, keepalived will
> > immediately
> > read the file and update the vrrp instances accordingly.
> > 3. In relation to notify scripts, keepalived can write to a FIFO
> > instead of executing a notify script. A process can then monitor
> > the
> > FIFO for any notifications (this has the side benefit that it can
> > be
> > guaranteed that notifications can be processed in the correct
> > sequence). keepalived can be configured to execute a script to
> > monitor
> > the FIFO to initiate any desired actions, just as though a notify
> > script had been executed, but again it saves the overhead of
> > executing
> > a script.
> > 
> > I hope this helps, and if you do try the code that is soon to be
> > released as beta, I would be very glad to hear any feedback you
> > have in
> > relation to it.
> > 
> > Regards,
> > 
> > Quentin
> > 
> > On Mon, 2017-10-02 at 15:16 -0400, Damien Clabaut wrote:
> > > Hello again Quentin,
> > > 
> > > I played a bit more with the templates and with strace.
> > > 
> > > As it turns out:
> > > 
> > > - One clone call means one vrrp script is launched. If I have 16
> > > instances, each launching a vrrp script every 5 seconds, I get 32
> > > clone
> > > calls over a 10 second period.
> > > 
> > > - The more instances, the longer each clone call (see attached:
> > > usec_per_instance_count.png)
> > > 
> > > - The more switch (master->backup->master), the longer each clone
> > > call
> > > (see attached: usec_per_vrrp_switch.png)
> > > 
> > > The graphs are pretty poor quality since this is a ML and I don't
> > > want
> > > to pollute everyone's inboxes.
> > > 
> > > Could you point me to the part of the code which calls all the
> > > vrrp_scripts ? I would like to study it and see if I can make
> > > some
> > > progress on it.
> > > 
> > > Regards,
> > > 
> > > 
> > > On 2017-10-02 11:40 AM, Damien Clabaut wrote:
> > > > 
> > > > On 2017-10-02 11:25 AM, Damien Clabaut wrote:
> > > > > Hello Quentin and thank you very much for your answer,
> > > > > 
> > > > > My setup is a bit compilcated and built on top of Keepalived
> > > > > PR
> > > > > for
> > > > > json output (https://github.com/acassen/keepalived/pull/633).
> > > > > 
> > > > > I will try to describe it as clearly as possible here, please
> > > > > tell me
> > > > > if this is not clear.
> > > > > 
> > > > > 
> > > > > The network configuration is as follows:
> > > > > 
> > > > > I have 2 Linux machines connected to each other (through a
> > > > > switch
> > > > > in
> > > > > my case, but that is not relevant).
> > > > > 
> > > > > Using attached templates, I create N vlans interfaces on each
> > > > > side of
> > > > > the link connecting both servers (from 2 to N+1).
> > > > > 
> > > > > On top of each of those vlan interfaces I make 2 keepalived
> > > > > instances
> > > > > (one in version 2 for IPv4, one in version 3 for IPv6)
> > > > > 
> > > > > instances are names: <ip_version><vlan id>. For example, IPv6
> > > > > instance on vlan 42 is called 642.
> > > > > 
> > > > > 
> > > > > As you can see in the attached templates, each instances is
> > > > > tracking
> > > > > a different script called offset_instance_<instance name>
> > > > > 
> > > > > All those track_scripts launch the (attached) script
> > > > > "/usr/bin/get_vrrp_offset <instance name>"
> > > > > 
> > > > > get_vrrp_offset queries a JSON file (which is in a ramdisk
> > > > > for
> > > > > performance), which contains a boolean for each VRRP
> > > > > instance.
> > > > > 
> > > > > If the boolean is 1, then the script returns 1, and the VRRP
> > > > > priority
> > > > > for this instance is lowered by 100.
> > > > > 
> > > > > In any other case (JSON file not present or not valid, data
> > > > > not
> > > > > present,...) the script returns 0 and nothing happens.
> > > > > 
> > > > > 
> > > > > The json script is populated with the (attached)
> > > > > "/usr/bin/isolate"
> > > > > script.
> > > > > 
> > > > > Ran without arguments, it will set the boolean in the json
> > > > > file
> > > > > to
> > > > > "1" for all existing instances (collected from the json
> > > > > output).
> > > > > 
> > > > > Ran with "-r", it will do the opposite.
> > > > > 
> > > > > The interesting point is, it will change 1 instance every
> > > > > second,
> > > > > which allows us to move a large number of floating IPs
> > > > > without
> > > > > interruption.
> > > > > 
> > > > > 
> > > > > Please find attached:
> > > > > 
> > > > > - /usr/bin/isolate
> > > > > 
> > > > > - /usr/bin/get_vrrp_offset
> > > > > 
> > > > > - /usr/bin/vrrp_recap (This will print a table with data from
> > > > > the
> > > > > json output)
> > > > > 
> > > > > - make_conf.py (this will generate the keepalived config)
> > > > > 
> > > > > - template_vrrp_instance, which is called by make_conf.py
> > > > > 
> > > > > 
> > > > > To mount the ramdisk:
> > > > > 
> > > > > mkdir /mnt/ramdisk
> > > > > 
> > > > > mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
> > > > > 
> > > > > I will keep working on this today, and keep you updated on my
> > > > > findings.
> > > > > 
> > > > > Regards,
> > > > > 
> > > > > 
> > > > > On 2017-10-01 10:00 AM, Quentin Armitage wrote:
> > > > > > On Thu, 2017-09-28 at 16:02 -0400, Damien Clabaut wrote:
> > > > > > > Hello Keepalived devs,
> > > > > > > 
> > > > > > > While testing Keepalived version 1.3.6 on Debian8, I came
> > > > > > > accross a
> > > > > > > strange issue.
> > > > > > > 
> > > > > > > Tl;DR: It seems that after every vrrp state change, the
> > > > > > > "clone" system
> > > > > > > call takes a little more time, slowly eating up all CPU.
> > > > > > > 
> > > > > > > Details:
> > > > > > > 
> > > > > > > I start keepalived with 200 instances (100 IPv4 + 100
> > > > > > > IPv6).
> > > > > > > Each instance tracks a different script.
> > > > > > > The keepalived healthcheck children (as described on
> > > > > > > http://keepalived.org/documentation.html) has PID 177548:
> > > > > > > 
> > > > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] echo
> > > > > > > $(($(cat
> > > > > > > /var/run/keepalived.pid) + 2))
> > > > > > > 177458
> > > > > > > 
> > > > > > > All instances are in the MASTER state on this machine.
> > > > > > > Strace
> > > > > > > -c gives
> > > > > > > me the count of system calls on a 60 second time period:
> > > > > > > 
> > > > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
> > > > > > > strace
> > > > > > > -cp 177458
> > > > > > > % time     seconds  usecs/call     calls    errors
> > > > > > > syscall
> > > > > > > ------ ----------- ----------- --------- --------- ------
> > > > > > > ----
> > > > > > > ------
> > > > > > >     84.96    2.676412         880      3041       689
> > > > > > > clone
> > > > > > > 
> > > > > > > Using vrrp_scripts, I move all instances to BACKUP state,
> > > > > > > then back to
> > > > > > > MASTER state, and run the same command
> > > > > > > 
> > > > > > > [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
> > > > > > > strace
> > > > > > > -cp 177458
> > > > > > > % time     seconds  usecs/call     calls    errors
> > > > > > > syscall
> > > > > > > ------ ----------- ----------- --------- --------- ------
> > > > > > > ----
> > > > > > > ------
> > > > > > >     97.85   15.866534        4887      3247       901
> > > > > > > clone
> > > > > > > 
> > > > > > > 
> > > > > > > Again, I move all instances to BACKUP then MASTER, and
> > > > > > > take a
> > > > > > > trace.
> > > > > > > 
> > > > > > > There are about the same number of "clone" syscalls, but
> > > > > > > now
> > > > > > > they take
> > > > > > > 8651 usecs/call
> > > > > > > 
> > > > > > > To make sure this a consequence of state change, I wait
> > > > > > > 10
> > > > > > > then 20
> > > > > > > minutes and take new traces.
> > > > > > > 
> > > > > > > There is still the same number of "clone" syscalls, and
> > > > > > > now
> > > > > > > they take
> > > > > > > 7440 usecs/call.
> > > > > > > 
> > > > > > > On a separate experiment where I kept changing the VRRP
> > > > > > > state, I
> > > > > > > saw an
> > > > > > > entire core being used by this process.
> > > > > > > 
> > > > > > > Is this a known issue?
> > > > > > > 
> > > > > > > What can I do to help fix it?
> > > > > > > 
> > > > > > > Kind regards,
> > > > > > > 
> > > > > > 
> > > > > > Hi Damian,
> > > > > > 
> > > > > > This sounds an interesting problem. In the first place
> > > > > > could
> > > > > > you please
> > > > > > provide a copy of your configuration files (presumably from
> > > > > > at
> > > > > > least
> > > > > > two system  as close as possible to what you are running,
> > > > > > and
> > > > > > also
> > > > > > copies of the scripts. I'll then see if I can reproduce the
> > > > > > problem in
> > > > > > a Debian VM, and if so see if other distros exhibit the
> > > > > > same
> > > > > > problem.
> > > > > > I'll also have a look so see if my latest development
> > > > > > version,
> > > > > > which
> > > > > > has quite a few changes in the way scripts are run, also
> > > > > > exhibits the
> > > > > > same problem.
> > > > > > 
> > > > > > It would be helpful if you could explain what you do to
> > > > > > force
> > > > > > all the
> > > > > > instances into backup and then back to master, and over
> > > > > > what
> > > > > > timescale.
> > > > > > 
> > > > > > At the moment I haven't a clue what the problem might be.
> > > > > > 
> > > > > > Could you write a program that could issue a few hundred
> > > > > > clone() system
> > > > > > calls, and see what happens to the performance of those
> > > > > > around
> > > > > > the time
> > > > > > you see the different results for keepalived?
> > > > > > 
> > > > > > BTW, echo $(($(cat /var/run/keepalived.pid) + 2)) isn't a
> > > > > > reliable way
> > > > > > of getting the pid of the keepalived_vrrp process. I
> > > > > > sometimes
> > > > > > see the
> > > > > > keepalived parent and children processes having consecutive
> > > > > > pids, and
> > > > > > sometimes there are gaps of one or two in between. Better
> > > > > > to
> > > > > > use
> > > > > > vrrp.pid.
> > > > > > 
> > > > > > Many thanks for reporting this issue.
> > > > > > 
> > > > > > Quentin Armitage
> 
> 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot