Re: [RFC PATCH 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

On 8/19/26 20:53, Denis Pisarev wrote:
> Hi,
> 
> This RFC follows up on bugzilla 219492 ("failed to write reg 28b4 wait
> reg 28c6" after hibernation) and on the resume_gpu_stable proposal
> rejected in January:
> 
>   https://lore.kernel.org/amd-gfx/[email protected]/
> 
> The rejection argued that (a) kiq sched.ready is managed correctly -
> set false in suspend, set true again only after ring tests pass on
> resume - and (b) the proposed flag would have disabled the KIQ flush
> path permanently. Both points are correct, and this RFC takes a
> different route: nothing is disabled proactively; the driver reacts to
> observed failures.
> 
> MEASURED FAILURE MODE (Cezanne iGPU, Ryzen 7 PRO 5850U, 7.1.8, KDE
> Wayland, no ROCm):
> 
> - S4 resume -> "failed to write reg 28b4/28c6" + "1a6f4/1a706" pairs
>   begin in the same second as the thaw and continue at 80-140/hour
>   for 9+ hours until reboot. kiq sched.ready is true throughout (the
>   ring test at resume passed).
> - Two user-visible consequences: each failed flush burns the ~5 s KIQ
>   retry window (desktop-wide sluggishness), and the invalidation is
>   then silently dropped (stale TLBs - correctness).
> 
> TRIGGER ISOLATION (single-variable tests, one hibernate cycle each):
> 
> - stock (GFXOFF allowed across S4):    ~70-140 errors / 30 min of use
> - GFXOFF held off across S4:                    0 errors (30 min)
> - GFXOFF re-enabled 40 min after the clean
>   resume:                                      0 errors (15 min;
>   gating resumption not independently verified)
> 
> So the wedge forms in the S4 resume window while GFXOFF is allowed,
> and is consistent with the existing semaphore workaround comment in
> gmc_v9_0.c about losing invalidate-acknowledge state across
> power-gating cycles. A boot-time amdgpu.pg_mask=0 does not even
> initialize on this part (probe -ETIMEDOUT), so runtime debugfs
> control was used.
> 
> THE PATCH (1/1):
> 
> - amdgpu_gmc_fw_reg_write_reg_wait() returns -ETIME on failure and
>   counts consecutive failures
> - gmc_v9_0_flush_gpu_tlb() uses its existing MMIO path as fallback
>   when the KIQ submit fails (invalidations no longer dropped), and
>   after 3 consecutive failures skips the KIQ path entirely until the
>   counter resets - so wedged systems stop paying 5 s per flush
> - counter resets on every success and in gmc_v9_0_hw_fini(); each
>   suspend/resume cycle re-arms KIQ. No proactive disabling anywhere.
> - gmc_v10/v11/v12 call sites unchanged and compile-safe (int return
>   used as statement); extending the fallback to them is trivial once
>   the approach is agreed.
> 
> This mirrors what the sibling PASID path already does (KIQ submit,
> -ETIME on timeout, MMIO variant as fallback), applied to the per-VMID
> path.
> 
> OPEN QUESTIONS:
> 
> 1. Root cause in the S4 window: RLC/ME not processing KIQ invalidates
>    while GFXOFF is allowed? Happy to run tracing (amdgpu.vm_debug,
>    VM_INVALIDATE ack registers) on the affected hardware on request.
> 2. Should the latch be per-xcc-inst?
> 3. Should amdgpu_gmc_flush_gpu_tlb_pasid()'s KIQ branch share the
>    latch?

That still doesn't make any sense at all.

First of all GFXOFF completely turns of the CP/ME/KIQ, so sending *anything* to the KIQ including the ring test would fail.

Then second fallback is simply not an acceptable option, it is a *must* have to have a working KIQ.

Regards,
Christian.

> 
> Patch 1/1 follows. Comments very welcome - especially on whether the
> failure-driven fallback is the right shape vs. fixing the S4 resume
> ordering directly.
> 
> Denis Pisarev
> 
> Denis Pisarev (1):
>   drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
>     unresponsive
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h     |  2 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 15 +++++++++++----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  4 +++-
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 18 ++++++++++++++----
>  4 files changed, 30 insertions(+), 9 deletions(-)
> 
> --
> 2.55.0
>