Re: lttng-dev Digest, Vol 203, Issue 7

Hi Lakshya,

On 3/21/25 8:52 AM, Gour DEV wrote:
> Thanks Kienan for digging this and explaining the behaviour,
> 
> Looks like this option is not present in the version I am using v2.13.5 and
> the MAP_POPULATE is enabled by default there
> /* memory_map: mmap */
> memory_map = mmap(NULL, memory_map_size, PROT_READ | PROT_WRITE,
>   MAP_SHARED | LTTNG_MAP_POPULATE, shmfd, 0);
> 
> I have updated to the master branch and have verified by setting the
> LTTNG_UST_MAP_POPULATE_POLICY=none, and saw that now the RSS consumption is
> not very high now.
> 
> I think using LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible is better because
> here I will definitely know how much amount of memory lttng will use at
> start only, and allocate memory accordingly to my servers.
> 
> Also, I thought using --buffers-global
> <https://lttng.org/man/1/lttng-enable-channel/v2.8/#doc-opt--buffers-global>
> would
> be better for me as I could know how much memory will be required but looks
> like that option is only available for kernel tracking, it would be very
> good if this option could be available for user buffering also, but that
> could come later.
> 

Global buffers for UST is planned for the next release. The trade-off of 
using it is that tracing overhead will increase (as the buffers are no 
longer per-CPU), but in exchange for a potentially lower memory 
footprint particularly in situations where not all possible CPUs are online.

> 
> But again, thanks for the help Kienan.
> 

My pleasure!

thanks,
kienan

> Regards
> Lakshya.
> 
> 
> 
> 
> 
> 
> On Fri, Mar 21, 2025 at 2:21 AM Kienan Stewart <[email protected]>
> wrote:
> 
>> Hi Lakshya,
>>
>> I did some digging around. What you are seeing is the result of the
>> switching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit
>> 4d4838b ("Use MAP_POPULATE to reduce pagefault when available").
>>
>> The purpose of this change is to avoid taking page faults which tracing,
>> reducing first-event in a page latency.
>>
>> In the master branch, this feature has been made configurable for users
>> who don't want to pre-populate the pages and would rather take page
>> faults while tracing[2].
>>
>> Here is an example from LTTng master with map populate per possible CPU:
>>
>> ```
>> export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible
>>
>> # Create session, channels, start tracing, and run test app
>> # top -n1 -b | grep -E '(MiB|COMMAND|lttng)'
>> MiB Mem :  32768.0 total,  21883.7 free,   1456.0 used,   9428.3
>> buff/cache
>>
>> MiB Swap:      0.0 total,      0.0 free,      0.0 used.  31312.0 avail
>> Mem
>>
>>       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>> COMMAND
>>
>>    301117 debian    20   0  880176   2760   2760 S   0.0   0.0   0:00.04
>> lttng-sessiond
>>
>>    301118 debian    20   0   43856   1376   1376 S   0.0   0.0   0:00.01
>> lttng-runas
>>
>>    301133 debian    20   0  718616 263456 263456 S   0.0   0.8   0:00.17
>> lttng-consumerd
>>
>>    301135 debian    20   0    6996      0      0 S   0.0   0.0   0:00.05
>> lttng-runas
>>
>> # cat /proc/$(pgrep lttng-sessiond)/statm
>> lttng-sessiond: 220044 690 690 345 0 29900 0
>>
>>
>>
>> # pmap $(pgrep lttng-sessiond) | grep total
>>
>>    total           880176K
>>
>> # smem -P lttng-sessiond
>>
>>
>>     PID User     Command                         Swap      USS      PSS
>>      RSS
>>
>> 301118 debian   lttng-sessiond --daemonize         0      344      881
>>     2236
>>
>> 301117 debian   lttng-sessiond --daemonize         0     5676     6683
>>     9276
>>
>> 301201 debian   /usr/bin/python3 /usr/bin/s        0     8636    10086
>>    12936
>>
>>
>> # /proc/PID/statm for lttng-consumerd
>>
>>
>>
>> lttng-consumerd: 1749 0 0 129 0 130 0
>>
>> # pmap lttng-consumerd-pid | grep total
>> total kB            6996    1700     472
>>
>>
>>
>> # smem -P lttng-consumerd
>>     PID User     Command                         Swap      USS      PSS
>>      RSS
>>
>> 301135 debian   lttng-consumerd  -u --consu        0      280      563
>>     1700
>>
>> 301211 debian   /usr/bin/python3 /usr/bin/s        0    10048    11501
>>    14404
>>
>> 301133 debian   lttng-consumerd  -u --consu        0   262376   263177
>> 265480
>>
>> # smem -m | grep -i ust
>>
>> /dev/shm/lttng-ust-wait-8-1000               1        4        4
>> /dev/shm/shm-ust-consumer-301133             1   260756   260756
>> ```
>>
>> When using the none policy:
>>
>> ```
>> # export LTTNG_UST_MAP_POPULATE_POLICY=none
>> # as above
>>
>> Running test app UID 0
>>
>>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ------cpu-----
>>
>>    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
>> id wa st
>>
>>    1  0      0  21875      0   9434    0    0    39   636 1105 2496  0  1
>> 99  0  0
>>
>> MiB Mem :  32768.0 total,  21875.0 free,   1458.2 used,   9434.7
>> buff/cache
>>
>> MiB Swap:      0.0 total,      0.0 free,      0.0 used.  31309.8 avail
>> Mem
>>
>>       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>> COMMAND
>>
>>    301616 debian    20   0  880176   2756   2756 S   0.0   0.0   0:00.04
>> lttng-sessiond
>>
>>    301617 debian    20   0   43856   1392   1392 S   0.0   0.0   0:00.01
>> lttng-runas
>>
>>    301632 debian    20   0  718612   5416   5416 S   0.0   0.0   0:00.17
>> lttng-consumerd
>>
>>    301634 debian    20   0    6992      0      0 S   0.0   0.0   0:00.05
>> lttng-runas
>>
>> lttng-sessiond: 220044 689 689 345 0 29900 0
>>
>>
>>    total           880176K
>>
>>
>>     PID User     Command                         Swap      USS      PSS
>>      RSS
>> 301617 debian   lttng-sessiond --daemonize         0      344      862
>>     2188
>> 301616 debian   lttng-sessiond --daemonize         0     5784     6759
>>     9328
>> 301700 debian   /usr/bin/python3 /usr/bin/s        0     8632    10079
>>    12928
>>
>> lttng-consumerd: 1748 0 0 129 0 129 0
>> total kB            6992    1580     468
>>     PID User     Command                         Swap      USS      PSS
>>      RSS
>> 301634 debian   lttng-consumerd  -u --consu        0      276      536
>>     1580
>> 301632 debian   lttng-consumerd  -u --consu        0     5672     6433
>>     8652
>> 301710 debian   /usr/bin/python3 /usr/bin/s        0     9996    11449
>>    14328
>>
>> /dev/shm/lttng-ust-wait-8-1000               1        4        4
>> /dev/shm/shm-ust-consumer-301632             1     4048     4048
>> ```
>>
>> thanks,
>> kienan
>>
>> [1]:
>>
>> https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94
>> [2]:
>>
>> https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7
>>
>> On 3/17/25 2:29 AM, Gour DEV wrote:
>>> Hi, Kienan
>>>
>>> Sorry for the late reply.
>>>
>>> Looks like in buster the memory is allocated by lttng-consumerd reserved
>>>
>>> I buster, the rss is less than the VIRT
>>> root@localhost:~#  COLUMNS=500 top  -b -n 1 | grep lttng
>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>> COMMAND
>>>    4095 root      20   0 1003188  31256   4660 S   0.0   0.1   0:03.81
>>> lttng-sessiond
>>>    4096 root      20   0   44260    796      0 S   0.0   0.0   0:00.01
>>> lttng-runas
>>>    4440 root      20   0 5236020  10224   8756 S   0.0   0.0   2:56.25
>>> lttng-consumerd -- here the VIRT is much more higher than RSS
>>>    4443 root      20   0   48048    540      0 S   0.0   0.0   0:00.12
>>> lttng-runas
>>>
>>>
>>>
>>> In bookworm the VIRT and RES are nearly the same only.
>>> root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top  -b -n 1 | grep lttng
>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>> COMMAND
>>>      4382 root      20   0 1098824  42600   8436 S   0.0   0.1   0:08.87
>>> lttng-sessiond
>>>      4403 root      20   0   48928   2116    996 S   0.0   0.0   0:00.00
>>> lttng-runas
>>>      5171 root      20   0 9879764   8.9g   8.9g S   0.0  28.7 108:23.53
>>> lttng-consumerd -- here the VRIT is nearly equal to RSS
>>>      5173 root      20   0    3680   1028    680 S   0.0   0.0   0:00.88
>>> lttng-runas
>>>
>>>
>>> Looks like lttng consumerd is allocating and reserving those pages, when
>>> any instrumented application starts.
>>>
>>> I am attaching the lttng status output in the mail, please do tell me if
>>> you need any more information regarding this.
>>>
>>>
>>> These is how we used to create the lttng channels and enable event which
>> is
>>> same in both buster and bookworm, (number of channels might differ)
>>>
>>> def enable_channel(channels, session, subbuf_size, subbuf_num):
>>> for c in channels:
>>> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
>>> str(subbuf_size), '--num-subbuf', str(subbuf_num),],
>>> stdout=devnull, stderr=subprocess.STDOUT)
>>>
>>>
>>> def enable_events(traces, session):
>>> for t in traces:
>>> if 'log-level-only' in t:
>>> log_opt = '--loglevel-only=' + t['log-level-only']
>>> elif 'log-level' in t:
>>> log_opt = '--loglevel=' + t['log-level']
>>> else:
>>> log_opt = ''
>>>
>>> else:
>>> call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'],
>>> '-s', session], stdout=devnull, stderr=subprocess.STDOUT)
>>>
>>>
>>> Thank You.
>>> Lakshya
>>>
>>>
>>>
>>>
>>> On Wed, Mar 12, 2025 at 8:06 PM <[email protected]>
>> wrote:
>>>
>>>> Send lttng-dev mailing list submissions to
>>>>           [email protected]
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>           https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>> or, via email, send a message with subject or body 'help' to
>>>>           [email protected]
>>>>
>>>> You can reach the person managing the list at
>>>>           [email protected]
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of lttng-dev digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>      1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>>         (Kienan Stewart)
>>>>      2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>>         (Gour DEV)
>>>>      3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>>         (Kienan Stewart)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Tue, 11 Mar 2025 14:55:21 -0400
>>>> From: Kienan Stewart <[email protected]>
>>>> To: Gour DEV <[email protected]>, [email protected]
>>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>> Message-ID: <[email protected]>
>>>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>>>
>>>> Hi Lakshya,
>>>>
>>>> On 3/11/25 12:25 PM, Gour DEV wrote:
>>>>    > Hi, Kienan
>>>>    >
>>>>    > here is the requested output
>>>>    >
>>>>    > root@localhost:~# top -b -n 1 | grep  lttng
>>>>    >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4
>>   8:39.93
>>>>    > lttng-c+
>>>>    >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1
>>   0:07.25
>>>>    > lttng-s+
>>>>    >     4825 root      20   0   48872   2188   1012 S   0.0   0.0
>>   0:00.00
>>>>    > lttng-r+
>>>>    >     4843 root      20   0    3680   1160    816 S   0.0   0.0
>>   0:00.23
>>>>
>>>> This top output for `localhost` seems very different than the output for
>>>> `localhost` in your previous message.
>>>>
>>>>
>>>>    > lttng-r+
>>>>    > root@localhost:~# nrpco
>>>>    > bash: nrpco: command not found
>>>>    > root@localhost:~# nproc
>>>>    > 16
>>>>    > root@localhost:~# cat /sys/devices/system/cpu/possible
>>>>    > 0-15
>>>>    >
>>>>
>>>> You indicated the bookworm machine has 32 cores, this is showing 16. If
>>>> you're comparing a 16 core machine to a 32 core machine, it is very
>>>> normal that the memory usage is higher on the 32 core machine.
>>>>
>>>>    >
>>>>    > Most of the process are running as asorcs user but some are running
>>>> as root.
>>>>
>>>> So you have two users with instrumented applications.
>>>>
>>>>
>>>> Given the discrepancies in the information provided I'm finding it a bit
>>>> hard to understand what you're looking at.
>>>>
>>>>
>>>> In general, a channel's shared memory footprint can be estimated
>> with[1]:
>>>>
>>>>      (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>>> (nUIDs or nPIDs)
>>>>
>>>> Note that the sub-buffer sizes you are using get rounded to the nearest
>>>> larger power of 2. See [2].
>>>>
>>>> thanks,
>>>> kienan
>>>>
>>>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
>>>> [2]:
>>>>
>> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Wed, 12 Mar 2025 14:49:07 +0530
>>>> From: Gour DEV <[email protected]>
>>>> To: Kienan Stewart <[email protected]>
>>>> Cc: [email protected]
>>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>> Message-ID:
>>>>           <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb=
>>>> [email protected]>
>>>> Content-Type: text/plain; charset="utf-8"
>>>>
>>>> Hi, Kienan
>>>>
>>>> I am attaching an screen recording of the behaviour I am seeing in this
>>>> mail. The behaviour is same irrespective of the device i use, sorry for
>>>> miscommunication in the npocs output (I assumed it was 32), but other
>> than
>>>> that all outputs are same (except the hostname as there are multiple
>>>> devices with same lttng config but this memory cosumption is seen on all
>>>> the devices).
>>>>
>>>> I had few question
>>>>
>>>> 1. Does lltng allocated all the memory it needs and mark it as dirty in
>> ram
>>>> when any process which links/uses lttng-ust runs? (here i tried with one
>>>> process but it is same for any of my process)
>>>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>>> (nUIDs or nPIDs)
>>>>
>>>> How do we calculate uid in the system is it all uids in the system? is
>> it
>>>> equal to `cat /etc/passwd | wc -l` ?
>>>>
>>>> I will put my calculations according to the above estimate based on all
>> the
>>>> channel i am creating
>>>>
>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal
>> to
>>>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on
>> the
>>>> start_lttng.py please do correct me if am wrong here.]
>>>>
>>>> But since there are only two users which uses lttng i think the correct
>>>> estimate would be
>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB
>>>>
>>>> Please do correct me If I am wrong calculations here.
>>>>
>>>> Now, there are a few things here, according to my output lttng is using
>> 11G
>>>> which is much more higher than the what is configured.
>>>>
>>>> I am attaching the lttng status and the file which is uses to create the
>>>> lttng sessions.
>>>>
>>>>
>>>>
>>>> Thank You.
>>>>
>>>>
>>>>
>>>>
>> https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing
>>>> -- recording of the behaviour which is seen
>>>>
>>>>
>> https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing
>>>> -- the file which is used to create lttng sessions
>>>>
>>>>
>>>> On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Lakshya,
>>>>>
>>>>> On 3/11/25 12:25 PM, Gour DEV wrote:
>>>>>    > Hi, Kienan
>>>>>    >
>>>>>    > here is the requested output
>>>>>    >
>>>>>    > root@localhost:~# top -b -n 1 | grep  lttng
>>>>>    >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4
>>>>    8:39.93
>>>>>    > lttng-c+
>>>>>    >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1
>>>>    0:07.25
>>>>>    > lttng-s+
>>>>>    >     4825 root      20   0   48872   2188   1012 S   0.0   0.0
>>>>    0:00.00
>>>>>    > lttng-r+
>>>>>    >     4843 root      20   0    3680   1160    816 S   0.0   0.0
>>>>    0:00.23
>>>>>
>>>>> This top output for `localhost` seems very different than the output
>> for
>>>>> `localhost` in your previous message.
>>>>>
>>>>>
>>>>>    > lttng-r+
>>>>>    > root@localhost:~# nrpco
>>>>>    > bash: nrpco: command not found
>>>>>    > root@localhost:~# nproc
>>>>>    > 16
>>>>>    > root@localhost:~# cat /sys/devices/system/cpu/possible
>>>>>    > 0-15
>>>>>    >
>>>>>
>>>>> You indicated the bookworm machine has 32 cores, this is showing 16. If
>>>>> you're comparing a 16 core machine to a 32 core machine, it is very
>>>>> normal that the memory usage is higher on the 32 core machine.
>>>>>
>>>>>    >
>>>>>    > Most of the process are running as asorcs user but some are running
>>>>> as root.
>>>>>
>>>>> So you have two users with instrumented applications.
>>>>>
>>>>>
>>>>> Given the discrepancies in the information provided I'm finding it a
>> bit
>>>>> hard to understand what you're looking at.
>>>>>
>>>>>
>>>>> In general, a channel's shared memory footprint can be estimated
>> with[1]:
>>>>>
>>>>>      (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>>>> (nUIDs or nPIDs)
>>>>>
>>>>> Note that the sub-buffer sizes you are using get rounded to the nearest
>>>>> larger power of 2. See [2].
>>>>>
>>>>> thanks,
>>>>> kienan
>>>>>
>>>>> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
>>>>> [2]:
>>>>>
>> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size
>>>>>
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL: <
>>>>
>> https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm
>>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 3
>>>> Date: Wed, 12 Mar 2025 10:36:28 -0400
>>>> From: Kienan Stewart <[email protected]>
>>>> To: Gour DEV <[email protected]>, [email protected]
>>>> Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
>>>> Message-ID: <[email protected]>
>>>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>>>
>>>> Hi Lakshya,
>>>>
>>>> On 3/12/25 5:03 AM, Gour DEV wrote:
>>>>> Hi, Kienan
>>>>>
>>>>> I am attaching an screen recording of the behaviour I am seeing in this
>>>>> mail. The behaviour is same irrespective of the device i use, sorry for
>>>>> miscommunication in the npocs output (I assumed it was 32), but other
>>>> than
>>>>> that all outputs are same (except the hostname as there are multiple
>>>>> devices with same lttng config but this memory cosumption is seen on
>> all
>>>>> the devices).
>>>>>
>>>>> I had few question
>>>>>
>>>>> 1. Does lltng allocated all the memory it needs and mark it as dirty in
>>>> ram
>>>>> when any process which links/uses lttng-ust runs? (here i tried with
>> one
>>>>> process but it is same for any of my process)
>>>>
>>>> I believe the shared memory for per-CPU data structures is allocated
>>>> when an instrumented application connects. There is no pre-allocation
>>>> for each possible UID on the system.
>>>>
>>>> You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to
>>>> see when the connection happens[1].
>>>>
>>>>> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
>>>>> (nUIDs or nPIDs)
>>>>>
>>>>> How do we calculate uid in the system is it all uids in the system? is
>> it
>>>>> equal to `cat /etc/passwd | wc -l` ?
>>>>
>>>> nUIDs is the number of distinct UIDs running instrumented applications.
>>>>
>>>>>
>>>>> I will put my calculations according to the above estimate based on all
>>>> the
>>>>> channel i am creating
>>>>>
>>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal
>> to
>>>>> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on
>> the
>>>>> start_lttng.py please do correct me if am wrong here.]
>>>>>
>>>>> But since there are only two users which uses lttng i think the correct
>>>>> estimate would be
>>>>> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB
>>>>
>>>> The estimate I gave is per-channel.
>>>>
>>>> small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel
>>>> per-UID
>>>> medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel
>> per-UID
>>>> large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID
>>>>
>>>> Now, you said you have 0 small channels, 6 medium channels, and 16 large
>>>> channels in your session. (Note: I see your script differs from these
>>>> stated channel counts).
>>>>
>>>> small: 0 * 1.0625 MiB = 0 MiB per-UID
>>>> medium: 6 * 17 MiB = 102 MiB per-UID
>>>> large: 16 * 272 MiB = 4352 MiB per-UID
>>>>
>>>> And if you're running instrumented applications with 2 users:
>>>>
>>>> small: 0 MiB * 2 = 0 MiB with 2 UIDs
>>>> medium: 102 MiB * 2 = 204 MiB with 2 UIDs
>>>> large: 4352 MiB * 2 = 8704 MiB with 2 UIDs
>>>>
>>>> Now this is just an estimation for the per-CPU ring buffers only, and
>>>> you numbers aren't hugely off so without analyzing your specific system
>>>> it doesn't seem to be that strange to me.
>>>>
>>>> If I take the number of channels I see in your script, it becomes:
>>>>
>>>> small: 0 MiB with 2 UIDs
>>>> medium: 136 MiB with 2 UIDs
>>>> large: 7616 MiB with 2 UIDs
>>>>
>>>> total: 7.57 GiB with 2 UIDs
>>>>
>>>>>
>>>>> Please do correct me If I am wrong calculations here.
>>>>>
>>>>> Now, there are a few things here, according to my output lttng is using
>>>> 11G
>>>>> which is much more higher than the what is configured.
>>>>>
>>>>
>>>> I have no idea what 'service start spyder' is doing. Maybe it's running
>>>> instrumented applications with an extra user that you didn't expect? I
>>>> can't help you with that aspect of your system.
>>>>
>>>> The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs
>>>> so maybe?
>>>>
>>>> I'd recommend you read your verbose sessiond log so see which
>>>> applications are connecting and with which UIDs.
>>>>
>>>>> I am attaching the lttng status and the file which is uses to create
>> the
>>>>> lttng sessions.
>>>>>
>>>>>
>>>>>
>>>>> Thank You.
>>>>>
>>>>
>>>> In any case, the information you have given to date hasn't demonstrated
>>>> to me in a tangible manner that you are seeing a difference related to
>>>> the version of LTTng being used.
>>>>
>>>> thanks,
>>>> kienan
>>>>
>>>> [1]:
>> https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Subject: Digest Footer
>>>>
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> [email protected]
>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> End of lttng-dev Digest, Vol 203, Issue 7
>>>> *****************************************
>>>>
>>>
>>
>>
>