[PATCH v7 0/4] perf sched latency: Refine outputs, unit scaling, and histogram support
Aaron Tomlin <[email protected]> Sun, 2 Aug 2026 17:09:10 -0400
| Newsgroups | org.kernel.vger.linux-perf-users,org.kernel.vger.linux-kernel |
|---|---|
| Message-ID | <[email protected]> |
This patch series improves 'perf sched latency' by suppressing misleading
empty table output, extending pipe mode stream processing, introducing
dynamic unit auto-scaling for latency and runtime statistics, and adding
latency histogram visualisation alongside time-span filtering.
Patch 1 addresses an issue where 'perf sched latency' fell through and
returned success (0) when perf_session__has_traces() failed due to missing
tracepoint events in a perf.data file. This caused empty header tables and
zeroed summary statistics to be rendered. In addition,
thread__get_runtime() in map_switch_event() is guarded against potential
NULL pointer dereferences under memory allocation failures.
Patch 2 extends pipe mode stream support. Because event attributes in pipe
mode are received dynamically during event processing, session->evlist is
not populated prior to event processing.
To handle pipe input correctly:
- Missing callbacks (.attr, .tracing_data, .build_id, and .feature) are
registered in cmd_sched() so header events are parsed correctly
- The handlers array is promoted to file-scope (latency_handlers[]) and
evlist__set_tracepoints_handlers() is invoked dynamically inside
perf_sched__process_tracepoint_sample() when evsel->handler is NULL
- Trace presence checks are performed post-processing when handling
pipe data, while non-pipe files continue to abort early upfront
Patch 3 introduces dynamic auto-scaling for latency and runtime display
columns (Runtime, Avg delay, Max delay). Previously, all values were
unconditionally formatted in milliseconds (ms), making microsecond- or
second-scale latencies difficult to read. Columns are now dynamically
scaled to the most appropriate unit (ns, us, ms, s), column headers are
updated, and format specifiers are aligned character-for-character with
table headers.
Patch 4 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between
snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or
100 us equal-width linear (linear) mode
--time:
Filters trace event processing to a specified [start,stop] time
span
Changes since v6:
- Fixed O(N) hot-path traversal overhead in
perf_sched__process_tracepoint_sample() for unhandled tracepoints by
assigning a dummy ignore handler (process_sched_ignore)
- Replaced evlist__set_tracepoints_handlers() in sample processing with a
per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
late-arriving pipe events
- Updated commit message for the pipe mode patch to reflect the
single-pass handler assignment logic
- Link to v6: https://lore.kernel.org/lkml/[email protected]/
Changes since v5:
- Split pipe mode trace sample handling from Patch 1 into a standalone
patch (Namhyung Kim)
- Added a Fixes: tag to the pipe mode patch referencing commit
27295592c22e ("perf session: Share the common trace sample_check routine
as perf_session__has_traces")
- Updated commit log with before and after illustrations of table header
formatting
- Link to v5: https://lore.kernel.org/lkml/[email protected]/
Changes since v4:
- Added the missing .feature callback to sched.tool in cmd_sched()
- Fixed memory leak by reusing uncompleted work atoms when sched_in events
are skipped or lost
- Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
added a dynamic lookup via evlist__set_tracepoints_handlers() inside
perf_sched__process_tracepoint_sample() whenever a sample arrives with
an unattached handler (i.e., evsel->handler == NULL)
- Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
in pipe mode and non-pipe modes
- Added missing trailing pipe in output_lat_thread's new auto-scaling
format string
- Stripped redundant prefix strings from each row's format string to
produce a clean, tabular output
- Link to v4: https://lore.kernel.org/lkml/[email protected]/
Changes since v3:
- Registered Missing Callbacks. In perf_tool__init configuration inside
cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
Without these callbacks, pipe mode drops header attributes entirely,
preventing tracepoints from being populated in session->evlist
- Introduced an explicit post-processing pipe check. This ensures that
pipe mode aborts correctly and does not produce superfluous empty
latency tables when no trace samples are available
- Prevented potential NULL pointer dereference. Added a NULL check for
thread__get_runtime() in map_switch_event() under memory allocation
failures
- Fixed column alignment. Modified format specifiers to align
character-for-character with header column widths across all rows
- Adjusted header label padding. Updated header label padding and column
width delimiters in perf_sched__lat() to match the exact field format
specifiers printed in output_lat_thread()
- Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
CPU-specific idle thread) from global_hist buckets
- Preserved task state machine across --time bounds. Moved time-window
filtering inside add_sched_in_event() to gate latency statistics
recording without breaking wakeup state tracking across time interval
boundaries
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes since v2:
- Ensured vertical pipe separators align across all latency table columns
- Excluded "swapper" (idle thread) latency samples from global_hist
- Preserved task state machine transitions across --time boundaries
- Suppressed empty table headers, total lines, and histogram graphs when
no matching trace samples exist (e.g., when --time, --CPU, or --pids
exclude all samples), outputting "No matching trace samples found."
instead
- Linked to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Fixed integer overflow in latency_bucket() (Ian Rogers)
- Linked to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf sched: Suppress latency table output when trace samples are
missing
perf sched: Handle missing trace samples in pipe mode
perf sched latency: Auto-scale latency and runtime display units
perf sched latency: Add histogram and time interval options
tools/perf/Documentation/perf-sched.txt | 6 +
tools/perf/builtin-sched.c | 338 +++++++++++++++++++++---
2 files changed, 301 insertions(+), 43 deletions(-)
--
2.55.0