Re: [RFC PATCH 0/3] drm/xe: diagnostics and workaround for GuC TLB invalidation ack stalls on ARL

On Tue, Aug 04, 2026 at 03:02:44PM -0600, Summers, Stuart wrote:
> On Mon, 2026-08-03 at 23:14 -0300, Tales A. Mendonça wrote:
> > Hi,
> > 
> > This series is a follow-up to the TLB invalidation ack stall I have
> > been debugging on ARL, tracked in:
> > 
> >   https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
> > 
> > Summary of the issue: with GuC 70.53.0 on ARL (reproduced on 7d51 and
> > 7dd1 machines here, plus an independent Arc Pro 130T report on the
> > issue above), TLB invalidation acks intermittently stall for ~2.3s.
> > The H2G request is consumed from the CTB immediately and the G2H CTB
> > is empty the whole time - the firmware simply does not send the ack
> > until much later. The fence timeout fires at 2.25s and the ack lands
> > tens of ms after it. Userspace blocked on the invalidation
> > (compositor
> > buffer unmaps etc.) hitches for the full window.
> 
> Firstly, thanks for the patch!
> 
> I haven't looked in to all the details of the sighting you were
> debugging, but we have had similar issues that were fixed in a later
> GuC version. I think around 70.60.0? It might be worth trying on
> something later than that to see if that helps... (+Daniele)
> 

I think this would require an AR on our end to make a new firmware
version available.

The upstream repo only has 70.53.0 available for ARL [1] (iirc, ARL
aliases to MTL for firmware). (+Julia too).

Presumably, the GuC changelogs should indicate whether an issue related
this has been fixed. If so, we need to update all GuC versions across
both i915 and Xe.

[1] https://gitlab.com/kernel-firmware/linux-firmware/-/blob/main/i915/mtl_guc_70.bin?ref_type=heads

> > 
> > Patch 1 adds xe_devcoredump_gt() so this kind of hang - which has no
> 
> Is there a reason we don't just re-use the main xe_devcoredump()?
>

This is my suggestion: the main devcoredump infrastructure is job-based,
so it cannot be used for hangs that are not associated with a job.

In my opinion, this is a gap on our end. Introducing something like
`xe_devcoredump_gt()`, which can be used for non-job-based hangs (e.g.,
TLB invalidation timeouts like those addressed in this series, or more
generally any GuC protocol hang), makes sense to me.

I haven't looked at the patch yet, but at a high level, adding
`xe_devcoredump_gt()` seems like a reasonable approach.

> > exec queue or job to blame - leaves a devcoredump with the GuC log
> > and
> > CT state behind (Matt suggested capturing devcoredumps when we
> > discussed the issue; devcoredumps from both machines are attached to
> > the issue above).
> > 
> > Patch 2 logs when the ack for a timed out invalidation finally
> > arrives. This is what established that the acks are late rather than
> > lost.
> > 
> > Patch 3 is the RFC part: a delayed work that pokes the GuC (status
> > register read, CT flush, doorbell ring) every 250ms while an ack is
> > overdue. On my machines this converts the guaranteed 2.3s stall into
> 
> I'm a little worried we're just papering over something here that needs
> to be addressed in GuC, particularly around GT going to sleep or
> something around the time we're expecting a response, so the pings on
> registers might be prematurely waking things up which is something we'd
> want to happen in GuC, not the KMD.
> 

In general, I agree with this. We should avoid papering over the issue
and instead fix it properly in the GuC. That said, this workaround
provides a pretty strong data point, since it appears to get the TLB
invalidation unstuck.

Matt

> Thanks,
> Stuart
> 
> > a
> > sub-500ms hiccup for the majority of occurrences; a minority of
> > severe
> > episodes ignore 8-9 consecutive doorbells, which points at the GuC
> > firmware being internally blocked for the whole window. Full data on
> > the issue. I am happy to rework the approach (different delay,
> > tying it to the G2H handler, dropping the status read, etc.) - mainly
> > I would like the firmware side investigated, since no host-side poke
> > can fix the severe cases.
> > 
> > Based on drm-tip. Tested for several days on both ARL machines under
> > desktop and VM-heavy workloads.
> > 
> > Thanks,
> > Tales
> > 
> > Tales A. Mendonça (3):
> >   drm/xe: Capture devcoredump on TLB invalidation timeout
> >   drm/xe: Log when a timed out TLB invalidation ack finally arrives
> >   drm/xe: Kick GuC while TLB invalidation acks are overdue
> > 
> >  drivers/gpu/drm/xe/xe_devcoredump.c     |  68 ++++++++++++
> >  drivers/gpu/drm/xe/xe_devcoredump.h     |   6 ++
> >  drivers/gpu/drm/xe/xe_tlb_inval.c       | 131
> > +++++++++++++++++++++++-
> >  drivers/gpu/drm/xe/xe_tlb_inval_types.h |  42 ++++++++
> >  4 files changed, 243 insertions(+), 4 deletions(-)
> > 
>