Re: gfx1100: KFD leaks process VRAM after CP fault during AQL queue abort; device unusable until reboot

On 2026-08-25 15:39, michael wizardsound.com wrote:
> [Some people who received this message don't often get email [email protected]. Learn why this is important athttps://aka.ms/LearnAboutSenderIdentification ]
>
> Hi AMD,
>
> I submitted some findings and I wanted to send this over.  I hope this helps.
>
> On a 7900 XTX in a Proxmox host (kernel 6.14.8-2-pve, ROCm userspaces 7.2.x/7.14 in an LXC sharing the host kernel), a userspace bug (rocm-systems#10021; root cause filed as rocm-systems#10713, runtime fix proposed in rocm-systems PR #10714) can place a malformed packet on a user AQL queue. The CP raises gfx_v11_0_bad_op_irq ("Illegal opcode in command stream") followed by an HSA_STATUS_ERROR_INVALID_PACKET_FORMAT queue abort. After the process exits, its /sys/class/kfd/kfd/proc/<pid> entry persists and ~25.7 GB VRAM stays pinned. amdgpu mode1 reset succeeds but does not release it; unbind succeeds but re-probe fails with EEXIST on mem_info_preempt_used (stale sysfs child from the wedged teardown), and subsequent per-device remove/bind attempts leave D-state tasks in device_release_driver_internal / device_driver_attach. Only a reboot recovers the device. Clean SIGSEGV deaths of the same workload never leak; it is specific to the CP-fault teardown. Full logs, timelines, and a userspace reproducer chain available; patches and standalone reproducers for the userspace side are athttps://github.com/OhgurLabs/rocm-hipgraph-fixes.

The most common cause of this type of bug is a reference counting 
problem in struct kfd_process. In this case it appears to be in in the 
event handling code for SOC15_INTSRC_CP_BAD_OPCODE.  There used to be a 
missing kfd_unref_process in kfd_dqm_suspend_bad_queue_mes in 
kfd_device_queue_manager.c. This was fixed as part of this commit over a 
year ago:

commit 8544374c0f82edb285779f21b149826fe2c2977c
Author:     Xiaogang Chen<[email protected]>
AuthorDate: Mon Jan 13 17:35:59 2025 -0600
Commit:     Alex Deucher<[email protected]>
CommitDate: Wed Feb 12 21:02:55 2025 -0500

     drm/amdkfd: Have kfd driver use same PASID values from graphic driver

     Current kfd driver has its own PASID value for a kfd process and uses it to
     locate vm at interrupt handler or mapping between kfd process and vm. That
     design is not working when a physical gpu device has multiple spatial
     partitions, ex: adev in CPX mode. This patch has kfd driver use same pasid
     values that graphic driver generated which is per vm per pasid.

     These pasid values are passed to fw/hardware. We do not need change interrupt
     handler though more pasid values are used. Also, pasid values at log are
     replaced by user process pid; pasid values are not exposed to user. Users see
     their process pids that have meaning in user space.

     Signed-off-by: Xiaogang Chen<[email protected]>
     Reviewed-by: Felix Kuehling<[email protected]>
     Signed-off-by: Alex Deucher<[email protected]>

I guess you are using the driver that comes with your 6.14.8-2-pve 
kernel, not the DKMS package that shipped with ROCm. This fix landed in 
Linux 6.15.

Regards,
   Felix


>
> Michael Levesque, MBA