RE: [RFC PATCH 1/3] drm/amdgpu/uapi: Add second-level trap handler ops to VM ioctl

AMD General

> -----Original Message-----
> From: Natalie Vock <[email protected]>
> Sent: Thursday, August 20, 2026 3:29 PM
> To: SHANMUGAM, SRINIVASAN <[email protected]>;
> Koenig, Christian <[email protected]>; Deucher, Alexander
> <[email protected]>
> Cc: [email protected]; Kuehling, Felix <[email protected]>;
> Zhu, James <[email protected]>; Lazar, Lijo <[email protected]>; Six,
> Lancelot <[email protected]>; Pelloux-Prayer, Pierre-Eric <Pierre-
> [email protected]>; Timur Kristóf <[email protected]>; Samuel
> Pitoiset <[email protected]>
> Subject: Re: [RFC PATCH 1/3] drm/amdgpu/uapi: Add second-level trap handler
> ops to VM ioctl
>
> Hi,
>
> first of all: Thanks for working on this! It's great seeing trap handler support come
> together.
>
> On 8/20/26 09:01, Srinivasan Shanmugam wrote:
> > When a GPU shader hits an exception, memory fault, or debug
> > breakpoint, the hardware jumps to the first-level trap handler. The
> > first-level handler (managed by the kernel via CWSR) checks the TMA
> > buffer for a second-level handler address. If one is installed, it
> > forwards the trap to that userspace handler, allowing the runtime or
> > debugger to handle shader exceptions without modifying the kernel trap handler.
> >
> > KFD already supports this for compute workloads. Render-node user
> > queues had no equivalent mechanism. Add it.
> >
> > The second-level handler is a per-VM setting — it applies to all
> > shader waves executing under that VMID regardless of queue type. GFX
> > and compute queues from the same process share the same VMID, so one
> > SET_L2_TRAP call covers all queue types for that process. This
> > configuration is not CWSR-specific; CWSR is only the first-level
> > handler mechanism. The correct home for this setting is the VM ioctl
> > (DRM_AMDGPU_VM), following the same pattern as
> > AMDGPU_VM_OP_RESERVE_VMID.
> >
> > Add two new VM ioctl operations:
> >    AMDGPU_VM_OP_SET_L2_TRAP   (op = 3) — install second-level handler
> >    AMDGPU_VM_OP_CLEAR_L2_TRAP (op = 4) — remove second-level
> handler
> >
> > Extend drm_amdgpu_vm_in with a 32-byte union for op-specific data. The
> > l2trap member carries the GPU virtual addresses and sizes of the TBA
> > (handler code) and TMA (handler scratch memory).
>
> This should be a BO handle and offset+size, instead. The BOs associated with the
> TBA/TMA must be tracked as used by every submission from the VM that has this
> trap handler installed, otherwise you introduce a ton of race conditions. Off the top
> of my head, here are a few:
> 1. The GEM VA ioctl can spuriously fail to actually update page tables.
>     This is okay and intentional, and BOs with outdated page tables will
>     be updated on the next submit if they're used by the submission. If
>     the TBA/TMA BOs aren't marked in the set of used buffers, the PTs may
>     end up never being updated and subsequent accesses will fault.
> 2. The TBA/TMA may be evicted/moved around concurrently with executing
>     submissions if these submissions didn't add their fences to the
>     TBA/TMA resv, which would likely randomly corrupt things or hang.
>
> A simpler solution could be requiring the TBA/TMA buffers to be
> VM_ALWAYS_VALID, in which case synchronization to all submissions in the VM
> is taken care of automagically. This prevents exporting the TBA/TMA to an fd, but I
> don't expect anyone would want to do this.

Hi Natalie,

Thanks for the feedback.

We will update the UAPI in v2 to pass BO handles alongside
the GPU VA for TBA/TMA. This ensures the kernel can properly
track and pin the buffers.

Thanks,
Srini

>
> Regards,
> Natalie