[PATCH v5 00/33] drm/xe: Add structured SIGID error logging infrastructure
Michal Wajdeczko <[email protected]>
| Newsgroups | org.freedesktop.lists.intel-xe |
|---|---|
| Message-ID | <[email protected]> |
Today the driver reports faults with ad-hoc drm_err()/xe_gt_err() strings that have no stable shape. That is readable for a human, but it gives fleet tooling nothing durable to match on: the wording changes between releases, lines can be rate-limited or dropped under an error storm, and there is no consistent way to ask "which recognised fault just happened?". Introduce a signature identifier (SIGID): a small, stable integer that names one recognised Xe fault situation and serves as the primary handle for triage. A SIGID maps, through published end-user documentation, to a description and a recommended action; the driver only has to emit the right SIGID next to the usual human-readable text. Design decisions: - Software-emitted signatures only. This header enumerates just the situations the driver detects and reports itself. Signatures that originate in firmware or hardware are identified by those layers (via their own records/counters) and are logged as received -- minting a driver-side id for them would duplicate an id the reporting layer already owns. - Flat catalogue, chosen per report site. Each site emits the single most specific situation for that site, so a multi-layer failure produces a chain of reports rather than one ambiguous classification (e.g. a failed GT reset reports GT_TDR and then WEDGED). A site that matches no defined situation keeps using ordinary xe_err() / xe_gt_err() rather than forcing a wrong id. - Stable numbering. A single flat list numbered sequentially from 1, in introduction order. Values are only ever appended, never renumbered or reused. - Severity is decoupled from the SIGID and chosen at the call site via xe_ras_log_fatal() / _recoverable() / _info(); the same situation can be reported at different severities depending on the instance. - dmesg stays close to a normal xe error line by reusing xe_err() / xe_gt_err() (and their Tile/GT decoration); the only stable, machine-matchable token added is SIGID=<n>. dmesg is not an ABI -- the durable machine record is the CPER carrying the same SIGID (a planned follow-up, left as a TODO). Wire up a representative site for each software signature so the set is exercised rather than merely declared. Signed-off-by: Mallesh Koujalagi <[email protected]> Signed-off-by: Rodrigo Vivi <[email protected]> Signed-off-by: Michal Wajdeczko <[email protected]> Cc: Thomas Hellström <[email protected]> Cc: Matthew Brost <[email protected]> Cc: Aravind Iddamsetty <[email protected]> Cc: Riana Tauro <[email protected]> Cc: Raag Jadav <[email protected]> Cc: Badal Nilawar <[email protected]> v1: https://patchwork.freedesktop.org/series/171022/#rev1 v2: https://patchwork.freedesktop.org/series/171022/#rev2 component/location dmesg decorations moved away from macros HW component identifiers related to RAS/FW enum values CORRECTED severity rendered as an error more support for const pointers log 'probe blocked' as INFO fix kunit device setup more tests v3: https://patchwork.freedesktop.org/series/171022/#rev3 fix typos, args list and to_dev() corner case (Sashiko) use assoc macros in xe_any (Jani) add xe_any kunit test suite (Michal) add more xe_log test cases (Michal) v4: https://patchwork.freedesktop.org/series/171022/#rev4 more patches with SIGID usage (Michal) support for printk index (Jani) more macros and other fixes (Mallesh) no recovery buckets (all) v5: new helper xe_log_err_info (Michal) survivability changes (Riana) report fault-response as info (Rodrigo/Daniele) Mallesh Koujalagi (1): drm/xe/log: Add structured SIGID error logging infrastructure Michal Wajdeczko (32): drm/xe: Introduce xe_any helpers drm/xe/log: Introduce structured component/location identifiers drm/xe/log: Add component/location decorations to dmesg drm/xe/log: Add SIGID log helpers for severity drm/xe/log: Add SIGID log helpers for location drm/xe/log: Add SIGID log helpers for location & severity drm/xe/log: Add SIGID log helpers for components drm/xe/log: Add SIGID log helpers for component & severity drm/xe/log: Add SIGID log helpers for errno-only drm/xe/log: Index all SIGID printk messages drm/xe/log: Add hardware error signatures drm/xe/log: Extend components list with hardware items drm/xe/ras: Check RAS and LOG component definitions drm/xe/kunit: Setup driver data in the test device drm/xe/tests: Add Kunit tests for xe_log drm/xe/tests: Add kunit tests for xe_any drm/xe: Report 'probe blocked' status using SIGID drm/xe: Report all probe errors using SIGID drm/xe/survivability: Report 'boot status' using SIGID drm/xe/survivability: Report sysfs failure in one place drm/xe/survivability: Report 'boot status' on sysfs failure drm/xe/survivability: Report 'Boot Mode enabled' status using SIGID drm/xe/survivability: Report 'Runtime Mode enabled' status using SIGID drm/xe: Report 'device wedged' errors using SIGID drm/xe/pcode: Report 'Mailbox failed' error using SIGID drm/xe/pcode: Report 'timeout, retrying' error using SIGID drm/xe/pcode: Report 'initialization timedout' error using SIGID drm/xe/guc: Report 'GuC mmio' errors using SIGID drm/xe/gt: Report 'reset failed' errors using SIGID drm/xe/gt: Report 'Fault response' pagefault error using SIGID drm/xe/gt: Report 'Queue full' pagefault error using SIGID drm/xe/pci: Report 'cannot re-enable' error using SIGID Documentation/gpu/xe/index.rst | 1 + Documentation/gpu/xe/xe_sigid.rst | 14 + drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/abi/xe_log_abi.h | 199 +++++++ drivers/gpu/drm/xe/abi/xe_sigid_abi.h | 172 ++++++ drivers/gpu/drm/xe/tests/Makefile | 1 + drivers/gpu/drm/xe/tests/xe_any_kunit.c | 213 ++++++++ drivers/gpu/drm/xe/tests/xe_kunit_helpers.c | 4 + drivers/gpu/drm/xe/tests/xe_log_kunit.c | 553 ++++++++++++++++++++ drivers/gpu/drm/xe/xe_any.h | 137 +++++ drivers/gpu/drm/xe/xe_device.c | 16 +- drivers/gpu/drm/xe/xe_gt.c | 7 +- drivers/gpu/drm/xe/xe_guc.c | 18 +- drivers/gpu/drm/xe/xe_log.c | 235 +++++++++ drivers/gpu/drm/xe/xe_log.h | 194 +++++++ drivers/gpu/drm/xe/xe_pagefault.c | 8 +- drivers/gpu/drm/xe/xe_pci.c | 32 +- drivers/gpu/drm/xe/xe_pci_error.c | 7 +- drivers/gpu/drm/xe/xe_pcode.c | 14 +- drivers/gpu/drm/xe/xe_ras.c | 11 + drivers/gpu/drm/xe/xe_survivability_mode.c | 85 +-- 21 files changed, 1854 insertions(+), 68 deletions(-) create mode 100644 Documentation/gpu/xe/xe_sigid.rst create mode 100644 drivers/gpu/drm/xe/abi/xe_log_abi.h create mode 100644 drivers/gpu/drm/xe/abi/xe_sigid_abi.h create mode 100644 drivers/gpu/drm/xe/tests/xe_any_kunit.c create mode 100644 drivers/gpu/drm/xe/tests/xe_log_kunit.c create mode 100644 drivers/gpu/drm/xe/xe_any.h create mode 100644 drivers/gpu/drm/xe/xe_log.c create mode 100644 drivers/gpu/drm/xe/xe_log.h -- 2.47.1