Re: "clone" syscall in keepalived healthchecking child takes more time after switching

Quentin,

If I understand your point 2. correctly, what I am doing today with a 
json file read by a python script could be handled by keepalived 
directly, provided I write in one separate file per instance ?

This would be awesome. Does it include the same safety I have on Python 
script (if the file is not present or does not contain expected data, 
then it is seen as a 0) ?

I will try adding my patch (json output) to your branch. It looks like 
it conflicts, but nothing unsolvable IMHO.

I will keep you updated.

Regards,


On 2017-10-02 03:46 PM, Quentin Armitage wrote:
> Damien,
>
> The track_scripts are invoked from function vrrp_script_thread() in
> vrrp_scheduler.c, and the following function vrrp_script_child_thread()
> handles the termination of the scripts. The calls from there should be
> quite straight forward to follow, but if not by all means ask any
> further questions.
>
> I don't think it is relevant to what you are looking at but the notify
> scripts are invoked via notify_exec() in notify.c
>
> I have a version that is planned to be made available as a beta in the
> near future. This has a large number of performance enhancements,
> including some in the area of executing track scripts. The updated code
> is available at: https://github.com/pqarmitage/keepalived/commits/fixes
> .quick , and I would be very interested to hear if makes any
> improvements for you. The enhancements that may be relevant to what you
> are doing are:
> 1. Use execve() instead of system() to invoke scripts, with any
> parameters parsed at initialisation rather than when the script is run
> (my understanding is that this changes it from 2 forks to 1 fork per
> script invocation)
> 2. Ability to use track_files instead of track_scripts. keepalived uses
> inotify() to monitor changes to files, and reads a number from the file
> when the it changes, which is treated in the same way that the exit
> code from a track_script is used. This means that scripts do not need
> to be executed on a periodic basis, but rather when an external event
> occurs, if the track_file is written to, keepalived will immediately
> read the file and update the vrrp instances accordingly.
> 3. In relation to notify scripts, keepalived can write to a FIFO
> instead of executing a notify script. A process can then monitor the
> FIFO for any notifications (this has the side benefit that it can be
> guaranteed that notifications can be processed in the correct
> sequence). keepalived can be configured to execute a script to monitor
> the FIFO to initiate any desired actions, just as though a notify
> script had been executed, but again it saves the overhead of executing
> a script.
>
> I hope this helps, and if you do try the code that is soon to be
> released as beta, I would be very glad to hear any feedback you have in
> relation to it.
>
> Regards,
>
> Quentin
>
> On Mon, 2017-10-02 at 15:16 -0400, Damien Clabaut wrote:
>> Hello again Quentin,
>>
>> I played a bit more with the templates and with strace.
>>
>> As it turns out:
>>
>> - One clone call means one vrrp script is launched. If I have 16
>> instances, each launching a vrrp script every 5 seconds, I get 32
>> clone
>> calls over a 10 second period.
>>
>> - The more instances, the longer each clone call (see attached:
>> usec_per_instance_count.png)
>>
>> - The more switch (master->backup->master), the longer each clone
>> call
>> (see attached: usec_per_vrrp_switch.png)
>>
>> The graphs are pretty poor quality since this is a ML and I don't
>> want
>> to pollute everyone's inboxes.
>>
>> Could you point me to the part of the code which calls all the
>> vrrp_scripts ? I would like to study it and see if I can make some
>> progress on it.
>>
>> Regards,
>>
>>
>> On 2017-10-02 11:40 AM, Damien Clabaut wrote:
>>>
>>> On 2017-10-02 11:25 AM, Damien Clabaut wrote:
>>>> Hello Quentin and thank you very much for your answer,
>>>>
>>>> My setup is a bit compilcated and built on top of Keepalived PR
>>>> for
>>>> json output (https://github.com/acassen/keepalived/pull/633).
>>>>
>>>> I will try to describe it as clearly as possible here, please
>>>> tell me
>>>> if this is not clear.
>>>>
>>>>
>>>> The network configuration is as follows:
>>>>
>>>> I have 2 Linux machines connected to each other (through a switch
>>>> in
>>>> my case, but that is not relevant).
>>>>
>>>> Using attached templates, I create N vlans interfaces on each
>>>> side of
>>>> the link connecting both servers (from 2 to N+1).
>>>>
>>>> On top of each of those vlan interfaces I make 2 keepalived
>>>> instances
>>>> (one in version 2 for IPv4, one in version 3 for IPv6)
>>>>
>>>> instances are names: <ip_version><vlan id>. For example, IPv6
>>>> instance on vlan 42 is called 642.
>>>>
>>>>
>>>> As you can see in the attached templates, each instances is
>>>> tracking
>>>> a different script called offset_instance_<instance name>
>>>>
>>>> All those track_scripts launch the (attached) script
>>>> "/usr/bin/get_vrrp_offset <instance name>"
>>>>
>>>> get_vrrp_offset queries a JSON file (which is in a ramdisk for
>>>> performance), which contains a boolean for each VRRP instance.
>>>>
>>>> If the boolean is 1, then the script returns 1, and the VRRP
>>>> priority
>>>> for this instance is lowered by 100.
>>>>
>>>> In any other case (JSON file not present or not valid, data not
>>>> present,...) the script returns 0 and nothing happens.
>>>>
>>>>
>>>> The json script is populated with the (attached)
>>>> "/usr/bin/isolate"
>>>> script.
>>>>
>>>> Ran without arguments, it will set the boolean in the json file
>>>> to
>>>> "1" for all existing instances (collected from the json output).
>>>>
>>>> Ran with "-r", it will do the opposite.
>>>>
>>>> The interesting point is, it will change 1 instance every
>>>> second,
>>>> which allows us to move a large number of floating IPs without
>>>> interruption.
>>>>
>>>>
>>>> Please find attached:
>>>>
>>>> - /usr/bin/isolate
>>>>
>>>> - /usr/bin/get_vrrp_offset
>>>>
>>>> - /usr/bin/vrrp_recap (This will print a table with data from
>>>> the
>>>> json output)
>>>>
>>>> - make_conf.py (this will generate the keepalived config)
>>>>
>>>> - template_vrrp_instance, which is called by make_conf.py
>>>>
>>>>
>>>> To mount the ramdisk:
>>>>
>>>> mkdir /mnt/ramdisk
>>>>
>>>> mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
>>>>
>>>> I will keep working on this today, and keep you updated on my
>>>> findings.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> On 2017-10-01 10:00 AM, Quentin Armitage wrote:
>>>>> On Thu, 2017-09-28 at 16:02 -0400, Damien Clabaut wrote:
>>>>>> Hello Keepalived devs,
>>>>>>
>>>>>> While testing Keepalived version 1.3.6 on Debian8, I came
>>>>>> accross a
>>>>>> strange issue.
>>>>>>
>>>>>> Tl;DR: It seems that after every vrrp state change, the
>>>>>> "clone" system
>>>>>> call takes a little more time, slowly eating up all CPU.
>>>>>>
>>>>>> Details:
>>>>>>
>>>>>> I start keepalived with 200 instances (100 IPv4 + 100 IPv6).
>>>>>> Each instance tracks a different script.
>>>>>> The keepalived healthcheck children (as described on
>>>>>> http://keepalived.org/documentation.html) has PID 177548:
>>>>>>
>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] echo $(($(cat
>>>>>> /var/run/keepalived.pid) + 2))
>>>>>> 177458
>>>>>>
>>>>>> All instances are in the MASTER state on this machine. Strace
>>>>>> -c gives
>>>>>> me the count of system calls on a 60 second time period:
>>>>>>
>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
>>>>>> strace
>>>>>> -cp 177458
>>>>>> % time     seconds  usecs/call     calls    errors syscall
>>>>>> ------ ----------- ----------- --------- --------- ----------
>>>>>> ------
>>>>>>     84.96    2.676412         880      3041       689 clone
>>>>>>
>>>>>> Using vrrp_scripts, I move all instances to BACKUP state,
>>>>>> then back to
>>>>>> MASTER state, and run the same command
>>>>>>
>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
>>>>>> strace
>>>>>> -cp 177458
>>>>>> % time     seconds  usecs/call     calls    errors syscall
>>>>>> ------ ----------- ----------- --------- --------- ----------
>>>>>> ------
>>>>>>     97.85   15.866534        4887      3247       901 clone
>>>>>>
>>>>>>
>>>>>> Again, I move all instances to BACKUP then MASTER, and take a
>>>>>> trace.
>>>>>>
>>>>>> There are about the same number of "clone" syscalls, but now
>>>>>> they take
>>>>>> 8651 usecs/call
>>>>>>
>>>>>> To make sure this a consequence of state change, I wait 10
>>>>>> then 20
>>>>>> minutes and take new traces.
>>>>>>
>>>>>> There is still the same number of "clone" syscalls, and now
>>>>>> they take
>>>>>> 7440 usecs/call.
>>>>>>
>>>>>> On a separate experiment where I kept changing the VRRP
>>>>>> state, I
>>>>>> saw an
>>>>>> entire core being used by this process.
>>>>>>
>>>>>> Is this a known issue?
>>>>>>
>>>>>> What can I do to help fix it?
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>> Hi Damian,
>>>>>
>>>>> This sounds an interesting problem. In the first place could
>>>>> you please
>>>>> provide a copy of your configuration files (presumably from at
>>>>> least
>>>>> two system  as close as possible to what you are running, and
>>>>> also
>>>>> copies of the scripts. I'll then see if I can reproduce the
>>>>> problem in
>>>>> a Debian VM, and if so see if other distros exhibit the same
>>>>> problem.
>>>>> I'll also have a look so see if my latest development version,
>>>>> which
>>>>> has quite a few changes in the way scripts are run, also
>>>>> exhibits the
>>>>> same problem.
>>>>>
>>>>> It would be helpful if you could explain what you do to force
>>>>> all the
>>>>> instances into backup and then back to master, and over what
>>>>> timescale.
>>>>>
>>>>> At the moment I haven't a clue what the problem might be.
>>>>>
>>>>> Could you write a program that could issue a few hundred
>>>>> clone() system
>>>>> calls, and see what happens to the performance of those around
>>>>> the time
>>>>> you see the different results for keepalived?
>>>>>
>>>>> BTW, echo $(($(cat /var/run/keepalived.pid) + 2)) isn't a
>>>>> reliable way
>>>>> of getting the pid of the keepalived_vrrp process. I sometimes
>>>>> see the
>>>>> keepalived parent and children processes having consecutive
>>>>> pids, and
>>>>> sometimes there are gaps of one or two in between. Better to
>>>>> use
>>>>> vrrp.pid.
>>>>>
>>>>> Many thanks for reporting this issue.
>>>>>
>>>>> Quentin Armitage

-- 
Damien Clabaut
R&D vRouter
ovh.qc.ca


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot