Re: [RFC PATCH v2 0/5] Add perf.data tracepoint events to trace.dat conversion

On Sun, Aug 2, 2026 at 11:08=E2=80=AFAM Tanushree Shah <[email protected]=
> wrote:
>
> This RFC patch series introduces support for converting perf.data files
> containing tracepoint events into trace.dat format, enabling seamless
> visualization and analysis using KernelShark.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Background and Motivation
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> Currently, perf and trace-cmd operate as separate tracing ecosystems with
> incompatible data formats. Users who collect tracepoint data with
> 'perf record' cannot easily visualize it in KernelShark's graphical
> timeline view or leverage trace-cmd's analysis capabilities.
>
> This creates workflow friction when users need to:
>
> - Visualize perf tracepoint data in KernelShark's interactive graphical
>   timeline
> - Share trace data between perf and trace-cmd workflows and toolchains
> - Perform architecture-independent conversion and analysis of traces
>
> This conversion bridge eliminates these barriers by enabling seamless
> data exchange between perf and trace-cmd ecosystems, allowing users to
> choose the best tool for each analysis phase.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Implementation Overview
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> The series implements the trace.dat file format specification (version 7)
> within perf's data conversion framework.
>
> **Patch 1/5: Core trace.dat Export Infrastructure**
> Introduces util/trace-dat.c and util/trace-dat.h implementing:
> - Per-CPU raw event buffer management (init, collect, free)
> - Ftrace ring buffer page construction
> - trace.dat section writers (strings, options, flyrecord sections)
>
> **Patch 2/5: Metadata Integration**
> Extends util/trace-event-read.c to write trace.dat metadata during
> perf.data
> parsing:
> - Initial format header (magic, version, endian, page size, compression)
> - Section 16: HEADER INFO (header_page + header_event)
> - Section 17: FTRACE EVENT FORMATS
> - Section 18: EVENT FORMATS (per system/event format files)
> - Section 19: KALLSYMS
> - Section 21: CMDLINES
> - Section 15: STRINGS (written last after all sections)
>
> **Patch 3/5: Conversion Backend**
> Implements util/data-convert-trace.c with trace_convert__perf2dat()
> function:
> - Processes PERF_TYPE_TRACEPOINT samples via process_sample_event()
> - Collects raw event data per-CPU using trace_dat__collect_cpu_event()
> - Writes OPTIONS sections (CPUCOUNT, TRACECLOCK, metadata offsets)
> - Writes FLYRECORD section with per-CPU ring buffer pages
>
> **Patch 4/5: User Interface**
> Extends tools/perf/builtin-data.c with --to-trace-dat option:
> - Adds command-line option for trace.dat output
> - Mutually exclusive with --to-ctf and --to-json
> - Calls trace_convert__perf2dat() to perform conversion
>
> **Patch 5/5: Shell Test**
> Adds a shell test (tools/perf/tests/shell/) covering normal tracepoint
> recordings, pipe mode and mixed tracepoint/non-tracepoint recording
> conversions and --force flag behaviour.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Current Implementation Details
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> **trace.dat Format Version:**
> The implementation currently targets trace.dat format version 7, which
> is the stable version supported by current trace-cmd releases (v3.x).
> This version is hardcoded to ensure compatibility with existing
> trace-cmd and KernelShark installations. Future enhancements could add
> version negotiation or support for newer format versions as they become
> standardized.
>
> **Compression Strategy:**
> Compression is explicitly disabled (set to NONE) in the generated
> trace.dat files.
> This design choice:
> - Simplifies the initial implementation and testing
> - Ensures maximum compatibility across trace-cmd versions
> - Avoids external compression library dependencies
>
> Future work could add support for various compression algorithms (zlib,
> zstd, lz4) with runtime selection via command-line options, significantly
> reducing file sizes for large traces.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Usage Example
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> ```bash
> *Record tracepoint events with perf*
> perf record -e sched:sched_switch -e sched:sched_wakeup -a sleep 10
>
> *Convert to trace.dat format*
> perf data convert --to-trace-dat=3Doutput.dat
>
> *Verify trace.dat structure*
> trace-cmd dump --summary output.dat
>
> *Analyze with trace-cmd*
> trace-cmd report output.dat
>
> *Visualize in KernelShark*
> kernelshark output.dat
> ```
>
> **Conversion Output:**
> ```
> [ perf data convert: Converted 'perf.data' into trace.dat format
> 'output.dat' ]
> [ perf data convert: Converted 2684 events ]
> ```
> **trace-cmd dump --summary Output:**
> ```
>  Tracing meta data in file output.dat:
>         [Initial format]
>                 7       [Version]
>                 0       [Little endian]
>                 8       [Bytes in a long]
>                 65536   [Page size, bytes]
>                 none    [Compression algorithm]
>                         [Compression version]
>         [buffer "", "local" clock, 65536 page size, 16 cpus, 1048576 byte=
s
>     flyrecord data]
>         [10 options]
>         [Saved command lines, 0 bytes]
>         [Kallsyms, 0 bytes]
>         [Ftrace format, 0 events]
>         [Header page, 206 bytes]
>         [Header event, 205 bytes]
>         [Events format, 1 systems]
>         [9 sections]
> ```
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Testing and Verification
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> The series has been extensively tested with:
> - Various tracepoint events (sched, irq, syscalls, block I/O)
> - Mixed recordings containing both tracepoint and non-tracepoint events
>   only tracepoints converted)
> - Verification with trace-cmd report and KernelShark visualization
> - Memory leak testing with Valgrind (0 bytes leaked).
> - Cross-architecture testing: v1 tested x86_64 and ppc64le. v2 adds
>   s390 (big-endian) perf.data converted on both ppc64le
>   (little-endian) and x86_64 (little-endian) hosts, in addition to
>   same-arch x86_64 (LE->LE) and ppc64le (BE->BE) conversion.
> - Pipe mode support has been tested end-to-end. (in v2)
>
> All generated trace.dat files successfully open in:
> - trace-cmd report (v3.1+)
> - KernelShark (v2.0+)
>
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Next Steps
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> We would highly appreciate reviews, comments, and feedback on:
> - The overall architectural approach and integration points
> - Compatibility considerations with trace-cmd ecosystem
> - Performance characteristics for large-scale traces
> - Additional use cases or workflow scenarios
> - Future enhancement priorities
>
> ---
> Changes in v2
>
> Addressing the Sashiko AI review findings on v1:
>
> Cross-arch correctness:
>   - Introduce to_file_u16/u32/u64 helpers (wrapping tep_read_number())
>     to write all multi-byte fields in the recorded machine's byte order;
>     apply throughout metadata sections and flyrecord page/record headers
>     (ts, commit, TIME_EXTEND, large-event data_len).
>   - Fix flyrecord record header bit layout for big-endian files.
>     The record header word bit layout differs by file endianness,
>     matching kbuffer-parse.c type_len4host()/ts4host():
>       LE: type_len in bits [4:0],   time_delta in bits [31:5]
>       BE: type_len in bits [31:27], time_delta in bits [26:0]
>
> Pipe mode:
>   - Add process_attr(), process_feature(), and process_tracing_data()
>     callbacks required for pipe mode operation.
>   - Defer CPU buffer initialisation until the first tracepoint sample,
>     after process_feature()/process_tracing_data() have populated the
>     session header. This ensures the recorded machine's CPU count is
>     used rather than the host's - critical for cross-platform analysis.
>
> Format compliance:
>   - Implement TIME_EXTEND records for timestamp deltas >27 bits to
>     prevent silent truncation and maintain chronological ordering.
>   - Fix large event encoding (>=3D29 words): use type_len=3D0 with a
>     separate 32-bit length word, avoiding collision with reserved types
>     (PADDING=3D29, TIME_EXTEND=3D30, TIME_STAMP=3D31).
>   - Add bounds check rejecting records larger than a page payload before
>     batching, preventing heap overflow in trace_dat__write_page().
>   - Fix flyrecord section_size to exclude the 16-byte section header,
>     matching trace.dat specification and trace-cmd behaviour.
>
> CLI behavior:
>   - Fix --force flag: open with O_CREAT|O_EXCL when force is not set,
>     failing with -EEXIST instead of silently overwriting existing files.
>
> Memory safety:
>   - Fix realloc overwrite of cpu_events->events and page_records on
>     failure: use temporary pointers, only commit on success.
>   - Fix use-after-free/double-free in sequential page_records realloc
>     failure: replace with malloc+memcpy+free pattern.
>   - Fix section_size computed from before section header position.
>   - Add NULL checks for get_tracing_file(), calloc() padding, and
>     trace_dat_options_offset assignment on write failure.
>   - Use goto out_free on record allocation failure to avoid leaking
>     accumulated page_records entries.
>   - Replace direct read() with do_read() in read_proc_kallsyms() to
>     handle short reads correctly.
>   - On fwrite failure, set trace_dat_write_failed and continue parsing
>     so that perf.data processing completes normally.
>
> Testing (new in v2):
>   - Add shell test covering conversion, trace-cmd dump validation,
>     sched_switch event verification, and --force flag behaviour.
>
> Documentation (new in v2):
>   - Add documentation for 'perf data convert --to-trace-dat', covering
>     usage and supported options.
>
> v1: https://lore.kernel.org/linux-perf-users/20260608125951.90425-2-tshah=
@linux.ibm.com/

Thanks again Tanushree for pursuing this. I see on sashiko.dev that
sashiko couldn't review the v2 patches because they failed to apply:
https://sashiko.dev/#/log/baseline/53604/2
The SHA of perf-tools-next that sashiko.dev was trying was
d04ef71492fad7230d474efe33d05f4c0563d409, which corresponds to "perf
machine: Check snprintf truncation for guest kallsyms path," added 4
days ago:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.gi=
t/commit/?h=3Dperf-tools-next&id=3Dd04ef71492fad7230d474efe33d05f4c0563d409
Could you rebase the patches so that we can benefit from the Sashiko review=
?

Thanks,
Ian

> Tanushree Shah (5):
>   perf/trace-dat: Add trace.dat export infrastructure
>   perf/trace-event: Write trace.dat metadata sections during parsing
>   perf data-convert: Add perf.data to trace.dat conversion backend
>   perf data: Add --to-trace-dat option for converting perf.data
>     tracepoint events into trace.dat format
>   perf test: Add test validating trace.dat generated by 'perf data
>     convert --to-trace-dat'
>
>
>  tools/perf/Documentation/perf-data.txt        |   7 +
>  tools/perf/builtin-data.c                     |  40 +-
>  ...rf_data_converter_tracepoints_trace_dat.sh | 169 ++++
>  tools/perf/util/Build                         |   2 +
>  tools/perf/util/data-convert-trace.c          | 240 +++++
>  tools/perf/util/data-convert.h                |   4 +
>  tools/perf/util/trace-dat.c                   | 876 ++++++++++++++++++
>  tools/perf/util/trace-dat.h                   | 113 +++
>  tools/perf/util/trace-event-read.c            | 307 +++++-
>  9 files changed, 1749 insertions(+), 9 deletions(-)
>  create mode 100755 tools/perf/tests/shell/test_perf_data_converter_trace=
points_trace_dat.sh
>  create mode 100644 tools/perf/util/data-convert-trace.c
>  create mode 100644 tools/perf/util/trace-dat.c
>  create mode 100644 tools/perf/util/trace-dat.h
>
> --
> 2.47.3
>