Re: [lttng-dev] Profiling LTTng tracepoint latency on different arm platforms
Mathieu Desnoyers via lttng-dev <[email protected]>
| Newsgroups | org.lttng.lists.lttng-dev |
|---|---|
| Message-ID | <[email protected]> |
On 6/21/23 01:39, Yitschak, Yehuda wrote: >> On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote: >>> On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote: >>>> Hello, >>> >>>> >>>> >> Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimproveito >> n*platform****1*? >>>> >>>> Thanks and best regards, >>>> >>>> Anas. >>>> >>> >>> I recommend using "perf" when tracing with the sample program in a >>> loop to figure out the hot spots. With that information on the "fast" >>> and "slow" system, we might be able to figure out what differs. >>> >>> Also, comparing the kernel configurations of the two systems can help. >>> Also comparing the glibc versions of the two systems would be relevant. >>> >> >> Also make sure you benchmark the lttng "snapshot" mode [1] to make sure >> you don't run into a situation where the disk/network I/O throughput cannot >> cope with the generated event throughput, thus causing the ring buffer to >> discard events. This would therefore "speed up" tracing from the application >> perspective because discarding an event is faster than writing it to a ring >> buffer. > > You mean we should avoid the "discard" loss mode and use "overwrite" loss mode since discard mode can fake fast performance ? Yes. In addition to use "overwrite-when-buffer-full" mode, the "snapshot" session also ensures that no consumer daemon extracts the trace data (unless an explicit snapshot record is performed), which allows comparing the ring buffer producer performance with minimal noise. If you really want to benchmark the discard-when-buffer-full mode and the the consumer daemon I/O behavior, then you need to take into account event discarded counts and the actual trace data size that was written to disk. Thanks, Mathieu > >> >> Thanks, >> >> Mathieu >> >> [1] https://lttng.org/docs/v2.13/#doc-taking-a-snapshot >> >>> Thanks, >>> >>> Mathieu >>> >>> >> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> https://www.efficios.com >> >> _______________________________________________ >> lttng-dev mailing list >> [email protected] >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev