Re: [RFC PATCH 0/3] drm/amdgpu: Render-node second-level trap handler

On Thu, Aug 20, 2026 at 3:40 AM Srinivasan Shanmugam
<[email protected]> wrote:
>
> When a GPU shader hits an error or exception, the hardware calls a trap
> handler. AMD GPUs support two levels: a first-level handler that runs
> inside the kernel (via CWSR), and an optional second-level handler that
> runs in userspace. The second-level handler lets a runtime or debugger
> catch shader exceptions without any kernel changes.
>
> KFD already provides this for compute workloads. Render-node user queues
> use the same first-level CWSR trap infrastructure but had no way to
> install a second-level handler. This series adds that support.
>
> The second-level handler is a per-process (per-VM) setting. All shader
> queues belonging to the same process share the same hardware VMID, so
> one SET_L2_TRAP call covers every GFX and compute queue for that
> process. This setting belongs in the VM ioctl (DRM_AMDGPU_VM), not in
> the CWSR ioctl, because it is not specific to the first-level handler
> mechanism.
>
> UAPI backward compatibility
> ---------------------------
> The drm_amdgpu_vm_in struct gains a 32-byte union for the new op data.
> Existing ops (RESERVE/UNRESERVE_VMID) only use the first 8 bytes and
> are unaffected. The DRM framework automatically zero-fills the new fields
> for older userspace programs. No existing Mesa or ROCr code is broken.
>
> Implementation
> --------------
> On discrete GPUs the CWSR scratch buffer (TMA) may live in VRAM, which
> is accessed through MMIO on some CPU platforms. Direct CPU pointer writes
> are unsafe there. The driver uses struct iosys_map for all TMA writes,
> which picks the right accessor automatically (regular memory or MMIO).
>
> Installing a new TBA/TMA requires two separate memory writes, which
> creates a race: a shader that traps between the two writes would see a
> mismatched TBA/TMA pair and could crash. To prevent this, the driver
> stops all user queues, flushes the GPU TLB, writes both values, then
> restarts the queues.
>
> GEM_VA UNMAP and CLEAR operations must always succeed — they cannot
> return an error. If userspace removes a TBA or TMA buffer without first
> calling CLEAR_L2_TRAP, the driver detects the overlap, runs the same
> stop-flush-clear sequence, and lets the unmap proceed silently. Queues
> that lose their trap handler this way are marked invalid and not
> restarted — a queue cannot safely run without a valid trap handler.
>
> TTMP register layout
> --------------------
> When a shader wave enters the second-level trap handler, these hardware
> registers carry the relevant state:
>
>   ttmp0/1    faulting shader PC (set by hardware)
>   ttmp14/15  second-level TMA address (set by first-level handler)
>   ttmp2/3    free for the handler to use as temporaries
>   ttmp6[30]  wave stopped flag (set by second-level handler/debugger)
>   ttmp6[29]  saved halt flag (set by second-level handler/debugger)
>   ttmp11[23] debug mode enabled (set by first-level handler)
>

I think it would be good to add some kernel doc for this.  We should
document the GPR usage per GFXIP version and explain how the trap
handler works in general.  For reference, the trap handler is per
GPUVM, not per queue.  As such, the trap handler is allocated by the
kernel driver and mapped into every GPUVM which needs a trap handler
at the same GPU VA.  This means the trap handler needs to be unified
for gfx and compute and needs to handle CWSR for compute.  If an
application wants to register its own trap handler, you can register
one with the primary trap handler.  This works by setting a flag in
the secondary trap handler and a GPU VA to jump to for the secondary
handler.  The application would call the trap handler IOCTL to
register the secondary trap handler and specify the GPU VA of the trap
handler it wants to register.  The IOCTL would then modify the primary
trap handler to enable the secondary handler.

Before we enable trap handlers for gfx, we need to modify the current
trap handlers in the kernel driver to handle gfx.

Alex

> Only compilation tested.
>
> Based on: amd-unified-interface branch
> Base commit: 0f354d789fdc ("drm/amdgpu: Drop vm_manager PASID to VM
>              mapping")
>
> Cc: Alex Deucher <[email protected]>
> Cc: Christian König <[email protected]>
> Cc: Lancelot Six <[email protected]>
> Cc: Felix Kuehling <[email protected]>
> Cc: James Zhu <[email protected]>
> Cc: Lijo Lazar <[email protected]>
> Cc: Pierre-Eric Pelloux-Prayer <[email protected]>
> Cc: Timur Kristóf <[email protected]>
> Cc: Samuel Pitoiset <[email protected]>
> Cc: Natalie Vock <[email protected]>
>
> Srinivasan Shanmugam (3):
>   drm/amdgpu/uapi: Add second-level trap handler ops to VM ioctl
>   drm/amdgpu: Add VM ioctl handlers for second-level trap handler
>   drm/amdgpu: Disable L2 trap handler when its VA range is unmapped
>
>  drivers/gpu/drm/amd/amdgpu/amdgpu_cwsr.c  | 341 ++++++++++++++++++----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_cwsr.h  |  54 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c   |  24 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c   |   5 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c |  15 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h |   1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  26 ++
>  include/uapi/drm/amdgpu_drm.h             |  80 ++++-
>  8 files changed, 469 insertions(+), 77 deletions(-)
>
>
> base-commit: 0f354d789fdcd03147eb77ab2e3ecdd5a67a1fad
> --
> 2.34.1
>