[PATCH v5 00/11] Add python, PCP support via libdtrace interfaces
Alan Maguire <[email protected]>
| Newsgroups | dev.linux.lists.dtrace |
|---|---|
| Message-ID | <[email protected]> |
Having python bindings for libdtrace interfaces to compile, run and collect information from DTrace programs is valuable because it could help integration into metric collection frameworks like PCP. This series adds python bindings for DTrace (patch 5) and provides tests for them (patch 8). The support is similar to what cmd/dtrace.c can do; compile a program, enable probes, run it and collect data. Support is also added to grab or create processes as is done by dtrace -p, -c options. Aggregation snapshot walk is also supported, with aggregation values filled out as raw values, ints, lists of function names for stack keys and dicts for quantized aggregations. For detailed description/usage see the README in patch 4 and the tests. Patch 9 then provides a consumer of python support - a Performance Co-Pilot (PCP) Performance Metric Data Agent (PMDA) which enables flexible metric collection via DTrace scripts. See the README.md in patch 9 for more details. In order to support an environment where we have multiple handles per process, some prep work is required. Patch 1 fixes some issues with multiple handle use, while patch 2 improves sharing for vmlinux BTF (and CTF generated from it if needed) which multiple handles in a process image will benefit from. The upper bound of per-process handles is now limited by the maximum number of BPF programs attachable to a uprobe which is 64 (BPF_TRACE_MAX_PROGS) which is encountered with attaches to the BEGIN probe. Patch 3 fixes an issue that surfaced during testing - we try to load BTF for builtin modules when we have modules.builtin.ranges - to avoid this we mark DTrace modules as builtin and have them share vmlinux BTF. Patch 4 provides libdtrace APIs to compute aggregation keys/values rather than having the python bindings reach too deeply into the implementation. Patch 5 is the cpython bindings themselves; the README.md describes their usage. Patch 6 packages them and patches 7 and 8 facilitate testing them. Patch 9 then builds the PCP PMDA using the python bindings while patch 10 packages them in a pcp-pmda-dtrace package. Finally patch 11 provides some basic testing in the absence of a full PCP setup. Changes since v4: - Fixed issues in libdtrace with module.builtin.ranges (patch 3) - Instead of sharing math header and doing aggr computations in the python code, provide libdtrace APIs to compute keys/values (patch 4) - Fixed issues around python/pcp package dependencies (Kris, patches 6, 10) - Fixed issues around pmdadtrace deleted/re-added stats (Kris, patch 5) - Fixed dynamic PMDA number use in pmdadtrace.python (Kris, patch 9) - Fixed session close issues for multi-session test (Kris, patch 8) - Fixed support to allow multiple defines to be passed in (patch 5) - Added tests for compile time/setopt options including defines and added an aggregation clear test to improve coverage (patch 8) - Fixed up pmdadtrace to handle compile-time options properly (patch 9) Changes since v3: - Fixed up issues identified by Kris around custom BTF path (patch 2) - Added PCP patches 8/9 Changes since v2: - Fixed some handle lifetime issues and related potential segfaults in error paths (patch 4) - Fixed up packaging to derive python package version from DTrace version (patch 5) Changes since v1: - Fixed multi-handle issues (patches 1/2) - Improved aggregation representation of stacks, added stddev and *quantize representations (patch 4) Alan Maguire (11): libdtrace: Support multiple DTrace handles per process libdtrace: share vmlinux BTF/CTF globally to support faster startup libdtrace: avoid BTF probes for ranged built-in modules libdtrace: Add aggregation key/value computation APIs python: Add cpython bindings for libdtrace dtrace.spec: add python bindings packaging runtest.sh: Prepare Python bindings for in-tree testing test: add tests for python bindings dtrace: Add a PCP PMDA to expose DTrace data as metrics dtrace.spec: Add PCP PMDA packaging pcp-pmda-dtrace test: Add a test for the PCP ManagedDTraceScript interface GNUmakefile | 4 + Makecheck | 3 + bindings/Build | 39 + bindings/python/README.md | 205 ++ bindings/python/pyproject.toml | 3 + bindings/python/setup.py | 67 + bindings/python/src/pydtrace_module.c | 2057 +++++++++++++++++ configure | 7 +- dtrace.spec | 88 +- libdtrace/dt_aggregate.c | 517 +++++ libdtrace/dt_bpf.c | 6 +- libdtrace/dt_btf.c | 100 +- libdtrace/dt_btf.h | 1 + libdtrace/dt_impl.h | 6 +- libdtrace/dt_module.c | 20 +- libdtrace/dt_open.c | 5 +- libdtrace/dt_prov_dtrace.c | 6 +- libdtrace/dtrace.h | 83 + libdtrace/libdtrace.ver | 3 + pcp/Build | 66 + pcp/Install | 37 + pcp/README.md | 343 +++ pcp/Remove | 26 + pcp/autostart/.gitkeep | 0 pcp/dtrace.conf | 12 + pcp/examples/io_times.d | 28 + pcp/examples/io_times.json | 8 + pcp/examples/irq_times.d | 51 + pcp/examples/irq_times.json | 8 + pcp/examples/packet_drop_reasons.d | 16 + pcp/examples/packet_drop_reasons.json | 8 + pcp/examples/profile_kernel.d | 33 + pcp/examples/profile_kernel.json | 11 + pcp/examples/syscall_counts.d | 8 + pcp/examples/syscall_counts.json | 8 + pcp/pmdadtrace.1 | 135 ++ pcp/pmdadtrace.python | 1664 +++++++++++++ runtest.sh | 4 + test/unittest/python/test.x | 17 + test/unittest/python/tst.aggr-actions.sh | 152 ++ test/unittest/python/tst.aggr-change.sh | 74 + test/unittest/python/tst.aggr-clear.sh | 44 + test/unittest/python/tst.aggr-keys.sh | 89 + test/unittest/python/tst.aggr-stack.sh | 62 + test/unittest/python/tst.aggr.sh | 58 + test/unittest/python/tst.compile-opts.sh | 50 + test/unittest/python/tst.exit.sh | 67 + test/unittest/python/tst.multi-session-end.sh | 89 + test/unittest/python/tst.multi-session.sh | 62 + test/unittest/python/tst.pcp.sh | 69 + test/unittest/python/tst.proc-create.sh | 71 + test/unittest/python/tst.proc-exit.sh | 57 + test/unittest/python/tst.proc-grab.sh | 84 + test/unittest/python/tst.setopt.sh | 30 + test/unittest/python/tst.work-error.sh | 36 + 55 files changed, 6778 insertions(+), 19 deletions(-) create mode 100644 bindings/Build create mode 100644 bindings/python/README.md create mode 100644 bindings/python/pyproject.toml create mode 100644 bindings/python/setup.py create mode 100644 bindings/python/src/pydtrace_module.c create mode 100644 pcp/Build create mode 100755 pcp/Install create mode 100644 pcp/README.md create mode 100755 pcp/Remove create mode 100644 pcp/autostart/.gitkeep create mode 100644 pcp/dtrace.conf create mode 100644 pcp/examples/io_times.d create mode 100644 pcp/examples/io_times.json create mode 100644 pcp/examples/irq_times.d create mode 100644 pcp/examples/irq_times.json create mode 100644 pcp/examples/packet_drop_reasons.d create mode 100644 pcp/examples/packet_drop_reasons.json create mode 100644 pcp/examples/profile_kernel.d create mode 100644 pcp/examples/profile_kernel.json create mode 100644 pcp/examples/syscall_counts.d create mode 100644 pcp/examples/syscall_counts.json create mode 100644 pcp/pmdadtrace.1 create mode 100755 pcp/pmdadtrace.python create mode 100755 test/unittest/python/test.x create mode 100755 test/unittest/python/tst.aggr-actions.sh create mode 100755 test/unittest/python/tst.aggr-change.sh create mode 100755 test/unittest/python/tst.aggr-clear.sh create mode 100755 test/unittest/python/tst.aggr-keys.sh create mode 100755 test/unittest/python/tst.aggr-stack.sh create mode 100755 test/unittest/python/tst.aggr.sh create mode 100755 test/unittest/python/tst.compile-opts.sh create mode 100755 test/unittest/python/tst.exit.sh create mode 100755 test/unittest/python/tst.multi-session-end.sh create mode 100755 test/unittest/python/tst.multi-session.sh create mode 100755 test/unittest/python/tst.pcp.sh create mode 100755 test/unittest/python/tst.proc-create.sh create mode 100755 test/unittest/python/tst.proc-exit.sh create mode 100755 test/unittest/python/tst.proc-grab.sh create mode 100755 test/unittest/python/tst.setopt.sh create mode 100755 test/unittest/python/tst.work-error.sh -- 2.43.5