Re: "clone" syscall in keepalived healthchecking child takes more time after switching

Hello Quentin,

I believe I found the reason why clone calls take more time with a 
bigger number of instances.

Tl;DR: For each instance, keepalived loops over the entire list of 
track_scripts to find the matching one.

Details:

In vrrp_track.c, There is this function:

alloc_track_script(list track_list, vector_t *strvec, const char 
*vrrp_iname)

It calls:

vsc = find_script_by_name(tracked);

Which does:

for (e = LIST_HEAD(vrrp_data->vrrp_script); e; ELEMENT_NEXT(e)) {
     scr = ELEMENT_DATA(e);
         if (!strcmp(scr->sname, name))
             return scr;


So, assuming the list is sorted:

- finding the track_script for each instance in the first vlan will take 
1 lookup

- for the second vlan, 2 lookups per instance

- for the vlan n, n lookups per instance.

I can offer to patch the find_script_by_name() function to make it more 
scalable.

What do you think ?

Regards,


On 2017-10-03 01:14 PM, Quentin Armitage wrote:
> Damien,
>
> Re safety reading the file, yes it does include the same safety. It
> uses strtol(buf, NULL, 0) to read from the beginning of the file, and
> that will return 0 if the file does not start with a valid number. If
> the file is not present it assumes 0.
>
> The code is in function process_track_file() in vrrp_track.c.
>
> The same track files can be used for different vrrp instances but
> obviously then the vrrp instances will all react to the same trigger,
> so if you want to manage each vrrp instance independantly then they
> fill each need a separate file.
>
> To replicate your functionality of writing 1 to a file for reducing the
> priority by 100, then you want to specify weight -100 for each track
> file.
>
> Do let me know how you get on.
>
> Quentin
>
> On Tue, 2017-10-03 at 09:45 -0400, Damien Clabaut wrote:
>> Quentin,
>>
>> If I understand your point 2. correctly, what I am doing today with
>> a
>> json file read by a python script could be handled by keepalived
>> directly, provided I write in one separate file per instance ?
>>
>> This would be awesome. Does it include the same safety I have on
>> Python
>> script (if the file is not present or does not contain expected
>> data,
>> then it is seen as a 0) ?
>>
>> I will try adding my patch (json output) to your branch. It looks
>> like
>> it conflicts, but nothing unsolvable IMHO.
>>
>> I will keep you updated.
>>
>> Regards,
>>
>>
>> On 2017-10-02 03:46 PM, Quentin Armitage wrote:
>>> Damien,
>>>
>>> The track_scripts are invoked from function vrrp_script_thread() in
>>> vrrp_scheduler.c, and the following function
>>> vrrp_script_child_thread()
>>> handles the termination of the scripts. The calls from there should
>>> be
>>> quite straight forward to follow, but if not by all means ask any
>>> further questions.
>>>
>>> I don't think it is relevant to what you are looking at but the
>>> notify
>>> scripts are invoked via notify_exec() in notify.c
>>>
>>> I have a version that is planned to be made available as a beta in
>>> the
>>> near future. This has a large number of performance enhancements,
>>> including some in the area of executing track scripts. The updated
>>> code
>>> is available at: https://github.com/pqarmitage/keepalived/commits/f
>>> ixes
>>> .quick , and I would be very interested to hear if makes any
>>> improvements for you. The enhancements that may be relevant to what
>>> you
>>> are doing are:
>>> 1. Use execve() instead of system() to invoke scripts, with any
>>> parameters parsed at initialisation rather than when the script is
>>> run
>>> (my understanding is that this changes it from 2 forks to 1 fork
>>> per
>>> script invocation)
>>> 2. Ability to use track_files instead of track_scripts. keepalived
>>> uses
>>> inotify() to monitor changes to files, and reads a number from the
>>> file
>>> when the it changes, which is treated in the same way that the exit
>>> code from a track_script is used. This means that scripts do not
>>> need
>>> to be executed on a periodic basis, but rather when an external
>>> event
>>> occurs, if the track_file is written to, keepalived will
>>> immediately
>>> read the file and update the vrrp instances accordingly.
>>> 3. In relation to notify scripts, keepalived can write to a FIFO
>>> instead of executing a notify script. A process can then monitor
>>> the
>>> FIFO for any notifications (this has the side benefit that it can
>>> be
>>> guaranteed that notifications can be processed in the correct
>>> sequence). keepalived can be configured to execute a script to
>>> monitor
>>> the FIFO to initiate any desired actions, just as though a notify
>>> script had been executed, but again it saves the overhead of
>>> executing
>>> a script.
>>>
>>> I hope this helps, and if you do try the code that is soon to be
>>> released as beta, I would be very glad to hear any feedback you
>>> have in
>>> relation to it.
>>>
>>> Regards,
>>>
>>> Quentin
>>>
>>> On Mon, 2017-10-02 at 15:16 -0400, Damien Clabaut wrote:
>>>> Hello again Quentin,
>>>>
>>>> I played a bit more with the templates and with strace.
>>>>
>>>> As it turns out:
>>>>
>>>> - One clone call means one vrrp script is launched. If I have 16
>>>> instances, each launching a vrrp script every 5 seconds, I get 32
>>>> clone
>>>> calls over a 10 second period.
>>>>
>>>> - The more instances, the longer each clone call (see attached:
>>>> usec_per_instance_count.png)
>>>>
>>>> - The more switch (master->backup->master), the longer each clone
>>>> call
>>>> (see attached: usec_per_vrrp_switch.png)
>>>>
>>>> The graphs are pretty poor quality since this is a ML and I don't
>>>> want
>>>> to pollute everyone's inboxes.
>>>>
>>>> Could you point me to the part of the code which calls all the
>>>> vrrp_scripts ? I would like to study it and see if I can make
>>>> some
>>>> progress on it.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> On 2017-10-02 11:40 AM, Damien Clabaut wrote:
>>>>> On 2017-10-02 11:25 AM, Damien Clabaut wrote:
>>>>>> Hello Quentin and thank you very much for your answer,
>>>>>>
>>>>>> My setup is a bit compilcated and built on top of Keepalived
>>>>>> PR
>>>>>> for
>>>>>> json output (https://github.com/acassen/keepalived/pull/633).
>>>>>>
>>>>>> I will try to describe it as clearly as possible here, please
>>>>>> tell me
>>>>>> if this is not clear.
>>>>>>
>>>>>>
>>>>>> The network configuration is as follows:
>>>>>>
>>>>>> I have 2 Linux machines connected to each other (through a
>>>>>> switch
>>>>>> in
>>>>>> my case, but that is not relevant).
>>>>>>
>>>>>> Using attached templates, I create N vlans interfaces on each
>>>>>> side of
>>>>>> the link connecting both servers (from 2 to N+1).
>>>>>>
>>>>>> On top of each of those vlan interfaces I make 2 keepalived
>>>>>> instances
>>>>>> (one in version 2 for IPv4, one in version 3 for IPv6)
>>>>>>
>>>>>> instances are names: <ip_version><vlan id>. For example, IPv6
>>>>>> instance on vlan 42 is called 642.
>>>>>>
>>>>>>
>>>>>> As you can see in the attached templates, each instances is
>>>>>> tracking
>>>>>> a different script called offset_instance_<instance name>
>>>>>>
>>>>>> All those track_scripts launch the (attached) script
>>>>>> "/usr/bin/get_vrrp_offset <instance name>"
>>>>>>
>>>>>> get_vrrp_offset queries a JSON file (which is in a ramdisk
>>>>>> for
>>>>>> performance), which contains a boolean for each VRRP
>>>>>> instance.
>>>>>>
>>>>>> If the boolean is 1, then the script returns 1, and the VRRP
>>>>>> priority
>>>>>> for this instance is lowered by 100.
>>>>>>
>>>>>> In any other case (JSON file not present or not valid, data
>>>>>> not
>>>>>> present,...) the script returns 0 and nothing happens.
>>>>>>
>>>>>>
>>>>>> The json script is populated with the (attached)
>>>>>> "/usr/bin/isolate"
>>>>>> script.
>>>>>>
>>>>>> Ran without arguments, it will set the boolean in the json
>>>>>> file
>>>>>> to
>>>>>> "1" for all existing instances (collected from the json
>>>>>> output).
>>>>>>
>>>>>> Ran with "-r", it will do the opposite.
>>>>>>
>>>>>> The interesting point is, it will change 1 instance every
>>>>>> second,
>>>>>> which allows us to move a large number of floating IPs
>>>>>> without
>>>>>> interruption.
>>>>>>
>>>>>>
>>>>>> Please find attached:
>>>>>>
>>>>>> - /usr/bin/isolate
>>>>>>
>>>>>> - /usr/bin/get_vrrp_offset
>>>>>>
>>>>>> - /usr/bin/vrrp_recap (This will print a table with data from
>>>>>> the
>>>>>> json output)
>>>>>>
>>>>>> - make_conf.py (this will generate the keepalived config)
>>>>>>
>>>>>> - template_vrrp_instance, which is called by make_conf.py
>>>>>>
>>>>>>
>>>>>> To mount the ramdisk:
>>>>>>
>>>>>> mkdir /mnt/ramdisk
>>>>>>
>>>>>> mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
>>>>>>
>>>>>> I will keep working on this today, and keep you updated on my
>>>>>> findings.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> On 2017-10-01 10:00 AM, Quentin Armitage wrote:
>>>>>>> On Thu, 2017-09-28 at 16:02 -0400, Damien Clabaut wrote:
>>>>>>>> Hello Keepalived devs,
>>>>>>>>
>>>>>>>> While testing Keepalived version 1.3.6 on Debian8, I came
>>>>>>>> accross a
>>>>>>>> strange issue.
>>>>>>>>
>>>>>>>> Tl;DR: It seems that after every vrrp state change, the
>>>>>>>> "clone" system
>>>>>>>> call takes a little more time, slowly eating up all CPU.
>>>>>>>>
>>>>>>>> Details:
>>>>>>>>
>>>>>>>> I start keepalived with 200 instances (100 IPv4 + 100
>>>>>>>> IPv6).
>>>>>>>> Each instance tracks a different script.
>>>>>>>> The keepalived healthcheck children (as described on
>>>>>>>> http://keepalived.org/documentation.html) has PID 177548:
>>>>>>>>
>>>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] echo
>>>>>>>> $(($(cat
>>>>>>>> /var/run/keepalived.pid) + 2))
>>>>>>>> 177458
>>>>>>>>
>>>>>>>> All instances are in the MASTER state on this machine.
>>>>>>>> Strace
>>>>>>>> -c gives
>>>>>>>> me the count of system calls on a 60 second time period:
>>>>>>>>
>>>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
>>>>>>>> strace
>>>>>>>> -cp 177458
>>>>>>>> % time     seconds  usecs/call     calls    errors
>>>>>>>> syscall
>>>>>>>> ------ ----------- ----------- --------- --------- ------
>>>>>>>> ----
>>>>>>>> ------
>>>>>>>>      84.96    2.676412         880      3041       689
>>>>>>>> clone
>>>>>>>>
>>>>>>>> Using vrrp_scripts, I move all instances to BACKUP state,
>>>>>>>> then back to
>>>>>>>> MASTER state, and run the same command
>>>>>>>>
>>>>>>>> [root@bz-perflab-6-vr /home/dclabaut/mac_flap] timeout 60
>>>>>>>> strace
>>>>>>>> -cp 177458
>>>>>>>> % time     seconds  usecs/call     calls    errors
>>>>>>>> syscall
>>>>>>>> ------ ----------- ----------- --------- --------- ------
>>>>>>>> ----
>>>>>>>> ------
>>>>>>>>      97.85   15.866534        4887      3247       901
>>>>>>>> clone
>>>>>>>>
>>>>>>>>
>>>>>>>> Again, I move all instances to BACKUP then MASTER, and
>>>>>>>> take a
>>>>>>>> trace.
>>>>>>>>
>>>>>>>> There are about the same number of "clone" syscalls, but
>>>>>>>> now
>>>>>>>> they take
>>>>>>>> 8651 usecs/call
>>>>>>>>
>>>>>>>> To make sure this a consequence of state change, I wait
>>>>>>>> 10
>>>>>>>> then 20
>>>>>>>> minutes and take new traces.
>>>>>>>>
>>>>>>>> There is still the same number of "clone" syscalls, and
>>>>>>>> now
>>>>>>>> they take
>>>>>>>> 7440 usecs/call.
>>>>>>>>
>>>>>>>> On a separate experiment where I kept changing the VRRP
>>>>>>>> state, I
>>>>>>>> saw an
>>>>>>>> entire core being used by this process.
>>>>>>>>
>>>>>>>> Is this a known issue?
>>>>>>>>
>>>>>>>> What can I do to help fix it?
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>>
>>>>>>> Hi Damian,
>>>>>>>
>>>>>>> This sounds an interesting problem. In the first place
>>>>>>> could
>>>>>>> you please
>>>>>>> provide a copy of your configuration files (presumably from
>>>>>>> at
>>>>>>> least
>>>>>>> two system  as close as possible to what you are running,
>>>>>>> and
>>>>>>> also
>>>>>>> copies of the scripts. I'll then see if I can reproduce the
>>>>>>> problem in
>>>>>>> a Debian VM, and if so see if other distros exhibit the
>>>>>>> same
>>>>>>> problem.
>>>>>>> I'll also have a look so see if my latest development
>>>>>>> version,
>>>>>>> which
>>>>>>> has quite a few changes in the way scripts are run, also
>>>>>>> exhibits the
>>>>>>> same problem.
>>>>>>>
>>>>>>> It would be helpful if you could explain what you do to
>>>>>>> force
>>>>>>> all the
>>>>>>> instances into backup and then back to master, and over
>>>>>>> what
>>>>>>> timescale.
>>>>>>>
>>>>>>> At the moment I haven't a clue what the problem might be.
>>>>>>>
>>>>>>> Could you write a program that could issue a few hundred
>>>>>>> clone() system
>>>>>>> calls, and see what happens to the performance of those
>>>>>>> around
>>>>>>> the time
>>>>>>> you see the different results for keepalived?
>>>>>>>
>>>>>>> BTW, echo $(($(cat /var/run/keepalived.pid) + 2)) isn't a
>>>>>>> reliable way
>>>>>>> of getting the pid of the keepalived_vrrp process. I
>>>>>>> sometimes
>>>>>>> see the
>>>>>>> keepalived parent and children processes having consecutive
>>>>>>> pids, and
>>>>>>> sometimes there are gaps of one or two in between. Better
>>>>>>> to
>>>>>>> use
>>>>>>> vrrp.pid.
>>>>>>>
>>>>>>> Many thanks for reporting this issue.
>>>>>>>
>>>>>>> Quentin Armitage
>>

-- 
Damien Clabaut
R&D vRouter
ovh.qc.ca


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel