Re: [PATCH] drm/amdgpu: drain SDMA before process eviction during GPU reset

On 22-Jul-26 12:01 AM, Sunday Clement wrote:
> Host kernel panics (Oops in free_swap_cache) when GPU reset is
> triggered under multi-process load. The crash is caused by SDMA engines
> writing to memory while munmap clears page table entries during process
> eviction.
> 
> The race condition:
> amdgpu_device_halt_activities() loops over all devices in the reset
> list and calls amdgpu_amdkfd_pre_reset() on each device. The first call
> to amdgpu_amdkfd_pre_reset() triggers global process eviction across
> all GPUs via kgd2kfd_suspend_process(), which starts munmap clearing
> PTEs. However, SDMA engines on other devices in the reset list still
> have in-flight DMA transactions that have not been drained.
> 
> Fix by draining ALL SDMA rings on ALL devices in the reset list BEFORE
> calling amdgpu_amdkfd_pre_reset() on any device. Also flush the HDP
> write cache to ensure SDMA writes reach system memory before PTEs are
> cleared. This guarantees no in-flight DMAs exist when process eviction
> begins.
> 

The processes are signaled during pre_reset. Without 
suspending/notifying any process, how do you expect rings to drain?

Thanks,
Lijo

> Signed-off-by: Sunday Clement <[email protected]>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 +++++++++++++++++++++-
>   1 file changed, 30 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 472e96ae884e..0cd3dc4cffb2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -5340,7 +5340,36 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev,
>   					  bool need_emergency_restart)
>   {
>   	struct amdgpu_device *tmp_adev = NULL;
> -	int i;
> +	int i, r;
> +	struct amdgpu_ring *ring;
> +
> +    /*
> +     * Drain all SDMA on all devices before any KFD pre_reset.
> +     *
> +     * amdgpu_amdkfd_pre_reset() triggers global process eviction across
> +     * all GPUs, which starts munmap clearing PTEs. We must ensure no
> +     * in-flight SDMA DMAs on any device when eviction begins, otherwise
> +     * SDMA writes race with munmap, corrupting PTEs.
> +     */
> +	list_for_each_entry(tmp_adev, device_list, reset_list) {
> +		for (i = 0; i < tmp_adev->sdma.num_instances; i++) {
> +			ring = &tmp_adev->sdma.instance[i].ring;
> +			if (ring->sched.timeout == MAX_SCHEDULE_TIMEOUT)
> +				continue;
> +
> +			r = amdgpu_fence_wait_empty(ring);
> +			if (r)
> +				dev_warn(tmp_adev->dev,
> +					"SDMA%d ring drain timeout before reset, proceeding\n", i);
> +		}
> +
> +		/*
> +		 * Flush HDP write cache to ensure SDMA writes reach system
> +		 * memory before munmap. Prevents cached writes from arriving
> +		 * after pages are freed.
> +		 */
> +		amdgpu_device_flush_hdp(tmp_adev, NULL);
> +	}
>   
>   	/* block all schedulers and reset given job's ring */
>   	list_for_each_entry(tmp_adev, device_list, reset_list) {