RE: [PATCH 1/2] drm/amdgpu: Remove separate guilty compute userq reset

AMD General


Regards,
      Prike

> -----Original Message-----
> From: Alex Deucher <[email protected]>
> Sent: Tuesday, August 4, 2026 10:18 PM
> To: Liang, Prike <[email protected]>
> Cc: [email protected]; Zhang, Jesse(Jie) <[email protected]>;
> Liu, Shaoyun <[email protected]>; Deucher, Alexander
> <[email protected]>; Koenig, Christian <[email protected]>
> Subject: Re: [PATCH 1/2] drm/amdgpu: Remove separate guilty compute userq reset
>
> On Tue, Aug 4, 2026 at 4:16 AM Liang, Prike <[email protected]<mailto:[email protected]>> wrote:
> >
> > AMD General
> >
> > Regards,
> >       Prike
> >
> > > -----Original Message-----
> > > From: Alex Deucher <[email protected]<mailto:[email protected]>>
> > > Sent: Monday, August 3, 2026 9:40 PM
> > > To: Liang, Prike <[email protected]<mailto:[email protected]>>
> > > Cc: [email protected]<mailto:[email protected]>; Zhang, Jesse(Jie)
> > > <[email protected]<mailto:[email protected]>>; Liu, Shaoyun <[email protected]<mailto:[email protected]>>; Deucher,
> > > Alexander <[email protected]<mailto:[email protected]>>; Koenig, Christian
> > > <[email protected]<mailto:[email protected]>>
> > > Subject: Re: [PATCH 1/2] drm/amdgpu: Remove separate guilty compute
> > > userq reset
> > >
> > > On Mon, Aug 3, 2026 at 9:36 AM Liang, Prike <[email protected]<mailto:[email protected]>> wrote:
> > > >
> > > > AMD General
> > > >
> > > > Regards,
> > > >       Prike
> > > >
> > > > > -----Original Message-----
> > > > > From: Alex Deucher <[email protected]<mailto:[email protected]>>
> > > > > Sent: Monday, August 3, 2026 9:23 PM
> > > > > To: Liang, Prike <[email protected]<mailto:[email protected]>>
> > > > > Cc: [email protected]<mailto:[email protected]>; Zhang, Jesse(Jie)
> > > > > <[email protected]<mailto:[email protected]>>; Liu, Shaoyun <[email protected]<mailto:[email protected]>>;
> > > > > Deucher, Alexander <[email protected]<mailto:[email protected]>>; Koenig,
> > > > > Christian <[email protected]<mailto:[email protected]>>
> > > > > Subject: Re: [PATCH 1/2] drm/amdgpu: Remove separate guilty
> > > > > compute userq reset
> > > > >
> > > > > On Mon, Aug 3, 2026 at 5:25 AM Liang, Prike <[email protected]<mailto:[email protected]>>
> wrote:
> > > > > >
> > > > > > AMD General
> > > > > >
> > > > > > As for the hung userq, it should be identified by the MES
> > > > > > reset API with the
> > > > > hang_detect_only setting. However, it is unlikely to miss
> > > > > detecting the invalid opcode hang case, especially given that
> > > > > the userq invalid opcode IGT test has not been implemented yet.
> > > > > >
> > > > > > Hi @Liu, Shaoyun, are you aware of any known userq hang
> > > > > > scenarios that cannot
> > > > > be identified by MES API hang_detect_only? If not, could you
> > > > > please help review the following patch, which unifies the userq
> > > > > reset path for hung
> > > queues?
> > > > > >
> > > > >
> > > > > We added it in the first place to deal with those cases.  There
> > > > > can be queues which are not hung, but will never complete and
> > > > > hence never signal their fence.  E.g., you can have a queue that
> > > > > is waiting on a memory location that MES can preempt, but due to
> > > > > a bug elsewhere that memory location will never change so the fence will
> never signal.
> > > >
> > > > Thank you for the input. However, for fake timeout cases such as
> > > > the long shader
> > > scenario, we should identify the fake hang cases by checking whether
> > > the guilty queue appears in the hang list, or whether the queue rptr
> > > is still updating? If so, preempt the queue rather than resetting it?
> > > >
> > >
> > > We have to assume that if we end up in the queue reset path that the queue is
> hung.
> > > The fences have to signal in finite time.  If we preempt the queue
> > > that won't signal the fence so we'll just end up in the queue reset
> > > path again.  Preemption of a queue that won't make progress only makes sense
> if fences are not involved.
> >
> > The most cases relevant to userq fence timeout and reset worker likely to be
> scheduled when a userq fence polling period expires. If a long running shader is
> detected, the driver may need to try preempting the queue within a few retry cycles.
> If the userq fence remains unsignaled after the retries are exhausted, the driver can
> either return -ETIME to userspace for further handling or fall back to resetting the
> queue directly. Meanwhile, If the preemption succeeds and the queue completes its
> work during the subsequent restore process, no further reset is necessary for the
> guilty queue?
> >
>
> If an application wants to run long running jobs they shouldn't use protected fences
> in the first place.  If they don't use protected fences, then it should behave like KFD
> queues.  If there is some operation that needs to happen the queues will get
> preempted and then will continue later.  DMA fences need to signal in finite time so
> we can't just keep pushing them off.  Other kernel paging operations may depend on
> them signalling.

Regarding the protected fences, do you mean the userq fence that triggers the hang detection work queue?

If so, the current implementation calls amdgpu_userq_fence_init() at fence emit time (i.e., at each queue submission), which both initializes the dma_fence structure and starts the hang detection timer for that submission. Once the timeout expires, amdgpu_userq_hang_detect_work() fires and triggers a queue reset regardless of whether the queue is actually hung or simply long-running. But there's a core problem for the kernel to distinguish a hung queue from a slow one at fence emit time, since both look identical an unsignaled fence with an active queue.

However, we can defer that determination to the first timeout expiry in amdgpu_userq_hang_detect_work(). At that point we have more information to work with:

•       Check for real hang: query rptr and rptr has not updated since the fence was emitted, the queue is genuinely stuck (frozen waves, infinite loop). This is a real hang, reset immediately and forcing completing the fences.
•       Check for forward progress: if rptr has updated but the fence is still unsignaled, the queue is making progress, so it is long running, not hung. In this case:
o       Extend the timeout (reschedule hang_detect_work with a longer interval)
o       Track retry count if the queue still has not completed after N extended cycles, escalate: Preempt the queue, force completing the fence and return -ETIME to userspace then let the application decide: resubmit, split the workload, or drop.
•       Avoid unnecessary resets: if preemption succeeds and the queue resumes and completes within the extended window, cancel the reset entirely since the queue was slow, not broken.

Do you think this is the right way for handling long-running tasks to resolve the userq timeout issue, or does it align with your idea?

Thanks,
Prike

Avoid unnecessary resets: if preemption succeeds and the queue resumes and completes within the extended window, cancel the reset entirely — the queue was slow, not broken.>
> Alex
>
> > In the longer term, we may need to introduce a more robust mechanism to
> distinguish slow queue cases from genuine hangs, handling them appropriately via
> extending time slice for completing the queue submission, userspace driven
> resubmission, or dropping the work through a queue reset. If it makes sense and
> right way to do, then I will work on implementing the solution for such slow/long
> queue cases.
> >
> > Thanks,
> > Prike
> >
> > > Alex
> > >
> > > > >
> > > > > Alex
> > > > >
> > > > > > Regards,
> > > > > >       Prike
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Liang, Prike
> > > > > > > Sent: Thursday, July 23, 2026 2:38 PM
> > > > > > > To: [email protected]<mailto:[email protected]>; Zhang, Jesse(Jie)
> > > > > > > <[email protected]<mailto:[email protected]>>
> > > > > > > Cc: Deucher, Alexander <[email protected]<mailto:[email protected]>>; Koenig,
> > > > > > > Christian <[email protected]<mailto:[email protected]>>
> > > > > > > Subject: RE: [PATCH 1/2] drm/amdgpu: Remove separate guilty
> > > > > > > compute userq reset
> > > > > > >
> > > > > > > I checked each different userq hang cases, and the guilty
> > > > > > > userq can be identified by the MES firmware and report correctly.
> > > > > > > @Zhang,
> > > > > > > Jesse(Jie) could you further check as well at you side?
> > > > > > >
> > > > > > > If there're some hang queues miss identified by MES
> > > > > > > firmware, then the correct thing is to further debug from
> > > > > > > MES firmware side rather than have such strange reset
> > > > > > > sequence and this reset workaround should be
> > > > > cleaned sooner or later.
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > >       Prike
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Liang, Prike <[email protected]<mailto:[email protected]>>
> > > > > > > > Sent: Wednesday, July 15, 2026 2:31 PM
> > > > > > > > To: [email protected]<mailto:[email protected]>
> > > > > > > > Cc: Deucher, Alexander <[email protected]<mailto:[email protected]>>;
> > > > > > > > Koenig, Christian <[email protected]<mailto:[email protected]>>; Liang, Prike
> > > > > > > > <[email protected]<mailto:[email protected]>>
> > > > > > > > Subject: [PATCH 1/2] drm/amdgpu: Remove separate guilty
> > > > > > > > compute userq reset
> > > > > > > >
> > > > > > > > amdgpu_mes_detect_and_reset_hung_queues() already detects
> > > > > > > > the guilty compute user queue and resets it through
> > > > > > > > mes_userq_reset_queue(). The additional reset via
> > > > > > > > mes_userq_reset() is unnecessary, so remove it to unify
> > > > > > > > the compute userq
> > > > > reset.
> > > > > > > >
> > > > > > > > Signed-off-by: Prike Liang <[email protected]<mailto:[email protected]>>
> > > > > > > > ---
> > > > > > > >  drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c    | 5 -----
> > > > > > > >  drivers/gpu/drm/amd/amdgpu/mes_userqueue.c | 2 --
> > > > > > > >  2 files changed, 7 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> > > > > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> > > > > > > > index 1e275c2e7dd3..4f2d5ff2f7be 100644
> > > > > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> > > > > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> > > > > > > > @@ -2315,11 +2315,6 @@ int
> > > > > > > > amdgpu_gfx_reset_mes_compute(struct
> > > > > > > > amdgpu_device *adev,
> > > > > > > >             deferred_end[n_deferred].fence = guilty_fence;
> > > > > > > >             n_deferred++;
> > > > > > > >     }
> > > > > > > > -   if (uq) {
> > > > > > > > -           r = mes_userq_reset(uq);
> > > > > > > > -           if (r)
> > > > > > > > -                   goto out;
> > > > > > > > -   }
> > > > > > > >     for (i = 0; i < num_hung; i++) {
> > > > > > > >             struct amdgpu_ring *hr = NULL;
> > > > > > > >             struct amdgpu_fence *hf = NULL; diff --git
> > > > > > > > a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
> > > > > > > > b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
> > > > > > > > index b6bfa3974839..fab21d4275f3 100644
> > > > > > > > --- a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
> > > > > > > > +++ b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
> > > > > > > > @@ -226,8 +226,6 @@ int mes_userq_reset_queue(struct
> > > > > > > > amdgpu_device *adev,
> > > > > > > >
> > > > > > > >     xa_for_each(&adev->userq_doorbell_xa, uq_id, uq) {
> > > > > > > >             if (uq->queue_type == queue_type) {
> > > > > > > > -                   if (uq == guilty_uq)
> > > > > > > > -                           continue;
> > > > > > > >                     if (uq->doorbell_index == db) {
> > > > > > > >                             uq->state = AMDGPU_USERQ_STATE_HUNG;
> > > > > > > >                             if (use_mmio)
> > > > > > > > --
> > > > > > > > 2.34.1
> > > > > >