Re: [RFC PATCH 0/3] drm/xe: diagnostics and workaround for GuC TLB invalidation ack stalls on ARL

> That would definitely help, because if the issue does not happen on i915
> it likely means that we're missing a WA or something like that in Xe.

Two data points on that, pulling in different directions:

* My second ARL machine (7dd1) has been running i915 for 7 days now,
  with the same GuC 70.53.0, and there is not a single TLB invalidation
  timeout or GuC error in its logs. Caveat: i915's TLB invalidation
  timeout is longer than xe's 2.25s, so short stalls could be silent
  there.

* However, Matt just pointed at i915 MTL issue 14469, which looks like
  the same problem on i915 - so it may not be xe-specific after all.

If it helps I can also boot i915 on the primary machine (7d51), where I
can compare against days of xe statistics on identical workloads.

> Just a bit of a terminology update here, to make sure we're on the same
> page: we usually refer to the notification you're sending to the GuC as
> an H2G interrupt and not a doorbell.

Thanks for the correction - I will use H2G interrupt from here on and
fix the terminology in v2.

> It feels like when the
> issue occurs something is stuck in HW rather than GuC FW and triggering
> the interrupt causes the HW to get unstuck.

That fits a pattern I can now see clearly with more data. Since
enabling the bigger GuC logs (~1.5 days, 35 stalls): 12 stalls were
unstuck by one of the H2G interrupts within 0.3-1.5s, but 23 ignored
8-9 consecutive interrupts and ran to the end. And in those severe
cases the request-to-ack time is nearly constant: 2.28-2.34s, every
single time. It does not look like congestion - it looks like a fixed
internal timeout expiring somewhere and releasing things.

Related: an A/B experiment I ran earlier (holding forcewake across the
whole GT, C6 residency pinned at 0ms for the whole window) still hit 9
timeouts in a row, so GT-level RC6 avoidance alone does not prevent it.

> Also, would you be able to capture the GuC logs when the issue occurs?

Done. I rebuilt with the debug-sized log buffers (8M event data / 1M
crash dump / 1M state capture) and xe.guc_log_level=3, and the series'
patch 1 (devcoredump on TLB invalidation timeout) captures the GuC log
at the exact moment the timeout fires. I attached three devcoredumps
(4-8.7MB each, containing the full GuC log around severe stalls that
ignored 8-9 H2G interrupts) to the gitlab issue:

  https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678

I have more captures if useful (9 so far).

> I have pushed the latest GuC FW for MTL here in case you want to give it
> a go:

Downloaded and staged - I will switch to 70.72.1 via
xe.guc_firmware_path today and report back with a few days of data
(this machine currently reproduces 20-35 stalls/day under my normal
workload, so the signal should be quick).

Thanks,
Tales

Em ter., 4 de ago. de 2026 às 20:50, Daniele Ceraolo Spurio
<[email protected]> escreveu:
>
>
>
> On 8/4/2026 4:00 PM, Tales A. Mendonça wrote:
> >> Are we seeing this on i915?
> > I have not tested i915 on the affected machines yet - the
> > instrumentation that measured the stalls (late-ack logging, kick
> > results) is xe-only, so I have no comparable i915 data. I can boot one
> > of the ARL machines with i915 for a few days and watch for TLB
> > invalidation timeouts there, if that data helps.
>
> That would definitely help, because if the issue does not happen on i915
> it likely means that we're missing a WA or something like that in Xe.
>
> >
> > More generally: both machines here reproduce reliably (~1 stall/hour
> > on a desktop workload, much more under memory pressure), so I am happy
> > to test anything on them - including any GuC build the firmware team
> > would like data on.
> >
> > On Stuart's masking concern: fully agreed, that is why patch 3 is
> > marked RFC. Patches 1-2 are pure diagnostics and stand on their own; I
> > am fine holding patch 3 until the firmware side has been looked at.
> > The data point it adds is that a doorbell ring unblocks the ack in the
> > majority of episodes, while the severe ones ignore 8-9 consecutive
> > rings - hopefully that narrows where to look inside the GuC.
>
> Just a bit of a terminology update here, to make sure we're on the same
> page: we usually refer to the notification you're sending to the GuC as
> an H2G interrupt and not a doorbell. I'm making this clarification
> because the GuC supports a separate per-context notification mechanism
> that is referred to as doorbell and which we currently do not implement
> in neither i915 nor Xe.
>
> When receiving the H2G interrupt, the only thing that the GuC does is
> look into the CTB and process anything in there; however, you've said
> that the contents of the H2G CTB are processed immediately, so the
> follow up interrupt should result in the GuC just bailing out and doing
> nothing because there is no data to process. It feels like when the
> issue occurs something is stuck in HW rather than GuC FW and triggering
> the interrupt causes the HW to get unstuck.
>
> I have pushed the latest GuC FW for MTL here in case you want to give it
> a go:
> https://gitlab.com/dceraolo/drm-firmware/-/blob/f783b931555be057dafc2400b7eb4d445c953fec/i915/mtl_guc_70.72.1.bin
> . You can override the GuC firmware used by the driver via the
> xe.guc_firmware_path modparam; the path is relative to /lib/firmware/
> and the firmware needs to be in initramfs for the driver to find it at
> boot. Note that we haven't tested this image on MTL, so it might have
> unexpected results.
>
> Also, would you be able to capture the GuC logs when the issue occurs?
> The default guc log size is relatively small, so you'd have to capture
> right when the issue happens. However, you can make them bigger by
> building the kernel with CONFIG_DRM_XE_DEBUG or by simply modifying the
> xe_guc_log.h file to pick the bigger size by default. If you go with the
> latter, please also set xe.guc_log_level=3 on the command line (this is
> automatically added by the kconfig).
>
> Thanks,
> Daniele
>
> >
> > I will send a v2 addressing Matt's review comments (the
> > __xe_devcoredump unification and the fixes on patch 2).
> >
> > Thanks,
> > Tales
> >
> > Em ter., 4 de ago. de 2026 às 19:08, Daniele Ceraolo Spurio
> > <[email protected]> escreveu:
> >>
> >>
> >> On 8/4/2026 2:33 PM, Summers, Stuart wrote:
> >>> On Tue, 2026-08-04 at 14:27 -0700, Matthew Brost wrote:
> >>>> On Tue, Aug 04, 2026 at 03:02:44PM -0600, Summers, Stuart wrote:
> >>>>> On Mon, 2026-08-03 at 23:14 -0300, Tales A. Mendonça wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> This series is a follow-up to the TLB invalidation ack stall I
> >>>>>> have
> >>>>>> been debugging on ARL, tracked in:
> >>>>>>
> >>>>>>     https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
> >>>>>>
> >>>>>> Summary of the issue: with GuC 70.53.0 on ARL (reproduced on 7d51
> >>>>>> and
> >>>>>> 7dd1 machines here, plus an independent Arc Pro 130T report on
> >>>>>> the
> >>>>>> issue above), TLB invalidation acks intermittently stall for
> >>>>>> ~2.3s.
> >>>>>> The H2G request is consumed from the CTB immediately and the G2H
> >>>>>> CTB
> >>>>>> is empty the whole time - the firmware simply does not send the
> >>>>>> ack
> >>>>>> until much later. The fence timeout fires at 2.25s and the ack
> >>>>>> lands
> >>>>>> tens of ms after it. Userspace blocked on the invalidation
> >>>>>> (compositor
> >>>>>> buffer unmaps etc.) hitches for the full window.
> >>>>> Firstly, thanks for the patch!
> >>>>>
> >>>>> I haven't looked in to all the details of the sighting you were
> >>>>> debugging, but we have had similar issues that were fixed in a
> >>>>> later
> >>>>> GuC version. I think around 70.60.0? It might be worth trying on
> >>>>> something later than that to see if that helps... (+Daniele)
> >>>>>
> >>>> I think this would require an AR on our end to make a new firmware
> >>>> version available.
> >>>>
> >>>> The upstream repo only has 70.53.0 available for ARL [1] (iirc, ARL
> >>>> aliases to MTL for firmware). (+Julia too).
> >>>>
> >>>> Presumably, the GuC changelogs should indicate whether an issue
> >>>> related
> >>>> this has been fixed. If so, we need to update all GuC versions across
> >>>> both i915 and Xe.
> >>> Right... I guess I'd still like to see if we can test this in GuC (or
> >>> get confirmation we can't for some reason) before committing something.
> >>> My worry is we will prevent bug reports like this by working around it
> >>> and miss critical bugs that need to be fixed in the right component.
> >>>
> >>>> [1]
> >>>> https://gitlab.com/kernel-firmware/linux-firmware/-/blob/main/i915/mtl_guc_70.bin?ref_type=heads
> >>>>
> >>>>>> Patch 1 adds xe_devcoredump_gt() so this kind of hang - which has
> >>>>>> no
> >>>>> Is there a reason we don't just re-use the main xe_devcoredump()?
> >>>>>
> >>>> This is my suggestion: the main devcoredump infrastructure is job-
> >>>> based,
> >>>> so it cannot be used for hangs that are not associated with a job.
> >>>>
> >>>> In my opinion, this is a gap on our end. Introducing something like
> >>>> `xe_devcoredump_gt()`, which can be used for non-job-based hangs
> >>>> (e.g.,
> >>>> TLB invalidation timeouts like those addressed in this series, or
> >>>> more
> >>>> generally any GuC protocol hang), makes sense to me.
> >>> Ok makes sense. We can do that here. It would be nice to have a more
> >>> inclusive implementation that lets us call this from anywhere so we
> >>> aren't duplicating things around for different use cases. But not a
> >>> blocker here.
> >>>
> >>>> I haven't looked at the patch yet, but at a high level, adding
> >>>> `xe_devcoredump_gt()` seems like a reasonable approach.
> >>>>
> >>>>>> exec queue or job to blame - leaves a devcoredump with the GuC
> >>>>>> log
> >>>>>> and
> >>>>>> CT state behind (Matt suggested capturing devcoredumps when we
> >>>>>> discussed the issue; devcoredumps from both machines are attached
> >>>>>> to
> >>>>>> the issue above).
> >>>>>>
> >>>>>> Patch 2 logs when the ack for a timed out invalidation finally
> >>>>>> arrives. This is what established that the acks are late rather
> >>>>>> than
> >>>>>> lost.
> >>>>>>
> >>>>>> Patch 3 is the RFC part: a delayed work that pokes the GuC
> >>>>>> (status
> >>>>>> register read, CT flush, doorbell ring) every 250ms while an ack
> >>>>>> is
> >>>>>> overdue. On my machines this converts the guaranteed 2.3s stall
> >>>>>> into
> >>>>> I'm a little worried we're just papering over something here that
> >>>>> needs
> >>>>> to be addressed in GuC, particularly around GT going to sleep or
> >>>>> something around the time we're expecting a response, so the pings
> >>>>> on
> >>>>> registers might be prematurely waking things up which is something
> >>>>> we'd
> >>>>> want to happen in GuC, not the KMD.
> >>>>>
> >>>> In general, I agree with this. We should avoid papering over the
> >>>> issue
> >>>> and instead fix it properly in the GuC. That said, this workaround
> >>>> provides a pretty strong data point, since it appears to get the TLB
> >>>> invalidation unstuck.
> >>> So if we hit this issue I guess we're already going to have some
> >>> performance degredation and the workaround makes that better. I need to
> >>> look at the implementation, but we could be potentially introducing
> >>> performance penalties in other areas doing these pings.
> >>>
> >>> Again, I'd like to see if we can fix this in the right place before
> >>> implementing a workaround for it. Hopefully Daniele or Julia can give
> >>> some direction there.
> >> Are we seeing this on i915 at all? Given that Xe does not officially
> >> support MTL/ARL and is missing several critical WAs for those platforms,
> >> the approach so far has been to only update the GuC FW if it is required
> >> for i915.
> >> Looking at the GuC release notes, there have been a couple of
> >> TLB-related fixes after 70.53, but they're both marked as only affecting
> >> PVC and Xe2+ platforms, so no fixes seem to be available for ARL (or at
> >> least they're not listed in the release notes).
> >>
> >> Daniele
> >>
> >>> Thanks,
> >>> Stuart
> >>>
> >>>> Matt
> >>>>
> >>>>> Thanks,
> >>>>> Stuart
> >>>>>
> >>>>>> a
> >>>>>> sub-500ms hiccup for the majority of occurrences; a minority of
> >>>>>> severe
> >>>>>> episodes ignore 8-9 consecutive doorbells, which points at the
> >>>>>> GuC
> >>>>>> firmware being internally blocked for the whole window. Full data
> >>>>>> on
> >>>>>> the issue. I am happy to rework the approach (different delay,
> >>>>>> tying it to the G2H handler, dropping the status read, etc.) -
> >>>>>> mainly
> >>>>>> I would like the firmware side investigated, since no host-side
> >>>>>> poke
> >>>>>> can fix the severe cases.
> >>>>>>
> >>>>>> Based on drm-tip. Tested for several days on both ARL machines
> >>>>>> under
> >>>>>> desktop and VM-heavy workloads.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Tales
> >>>>>>
> >>>>>> Tales A. Mendonça (3):
> >>>>>>     drm/xe: Capture devcoredump on TLB invalidation timeout
> >>>>>>     drm/xe: Log when a timed out TLB invalidation ack finally
> >>>>>> arrives
> >>>>>>     drm/xe: Kick GuC while TLB invalidation acks are overdue
> >>>>>>
> >>>>>>    drivers/gpu/drm/xe/xe_devcoredump.c     |  68 ++++++++++++
> >>>>>>    drivers/gpu/drm/xe/xe_devcoredump.h     |   6 ++
> >>>>>>    drivers/gpu/drm/xe/xe_tlb_inval.c       | 131
> >>>>>> +++++++++++++++++++++++-
> >>>>>>    drivers/gpu/drm/xe/xe_tlb_inval_types.h |  42 ++++++++
> >>>>>>    4 files changed, 243 insertions(+), 4 deletions(-)
> >>>>>>
> >
>


-- 
Com os cumprimentos,

Tales A. Mendonça
talesam.org
communitybig.org