Re: [RFC PATCH 0/3] drm/xe: diagnostics and workaround for GuC TLB invalidation ack stalls on ARL

On Mon, 2026-08-03 at 23:14 -0300, Tales A. Mendonça wrote:
> Hi,
> 
> This series is a follow-up to the TLB invalidation ack stall I have
> been debugging on ARL, tracked in:
> 
>   https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
> 
> Summary of the issue: with GuC 70.53.0 on ARL (reproduced on 7d51 and
> 7dd1 machines here, plus an independent Arc Pro 130T report on the
> issue above), TLB invalidation acks intermittently stall for ~2.3s.
> The H2G request is consumed from the CTB immediately and the G2H CTB
> is empty the whole time - the firmware simply does not send the ack
> until much later. The fence timeout fires at 2.25s and the ack lands
> tens of ms after it. Userspace blocked on the invalidation
> (compositor
> buffer unmaps etc.) hitches for the full window.

Firstly, thanks for the patch!

I haven't looked in to all the details of the sighting you were
debugging, but we have had similar issues that were fixed in a later
GuC version. I think around 70.60.0? It might be worth trying on
something later than that to see if that helps... (+Daniele)

> 
> Patch 1 adds xe_devcoredump_gt() so this kind of hang - which has no

Is there a reason we don't just re-use the main xe_devcoredump()?

> exec queue or job to blame - leaves a devcoredump with the GuC log
> and
> CT state behind (Matt suggested capturing devcoredumps when we
> discussed the issue; devcoredumps from both machines are attached to
> the issue above).
> 
> Patch 2 logs when the ack for a timed out invalidation finally
> arrives. This is what established that the acks are late rather than
> lost.
> 
> Patch 3 is the RFC part: a delayed work that pokes the GuC (status
> register read, CT flush, doorbell ring) every 250ms while an ack is
> overdue. On my machines this converts the guaranteed 2.3s stall into

I'm a little worried we're just papering over something here that needs
to be addressed in GuC, particularly around GT going to sleep or
something around the time we're expecting a response, so the pings on
registers might be prematurely waking things up which is something we'd
want to happen in GuC, not the KMD.

Thanks,
Stuart

> a
> sub-500ms hiccup for the majority of occurrences; a minority of
> severe
> episodes ignore 8-9 consecutive doorbells, which points at the GuC
> firmware being internally blocked for the whole window. Full data on
> the issue. I am happy to rework the approach (different delay,
> tying it to the G2H handler, dropping the status read, etc.) - mainly
> I would like the firmware side investigated, since no host-side poke
> can fix the severe cases.
> 
> Based on drm-tip. Tested for several days on both ARL machines under
> desktop and VM-heavy workloads.
> 
> Thanks,
> Tales
> 
> Tales A. Mendonça (3):
>   drm/xe: Capture devcoredump on TLB invalidation timeout
>   drm/xe: Log when a timed out TLB invalidation ack finally arrives
>   drm/xe: Kick GuC while TLB invalidation acks are overdue
> 
>  drivers/gpu/drm/xe/xe_devcoredump.c     |  68 ++++++++++++
>  drivers/gpu/drm/xe/xe_devcoredump.h     |   6 ++
>  drivers/gpu/drm/xe/xe_tlb_inval.c       | 131
> +++++++++++++++++++++++-
>  drivers/gpu/drm/xe/xe_tlb_inval_types.h |  42 ++++++++
>  4 files changed, 243 insertions(+), 4 deletions(-)
>