Re: xe: GT0 GuC scheduler wedge on Meteor Lake (7d55 ) — "Schedule disable failed to respond", recovers or hard -locks nondeterministically

Hi,

On Wed, 2026-08-05 at 23:03 +0000, Vivi, Rodrigo wrote:
> Yes, please send it to gitlab following
> https://drm.pages.freedesktop.org/intel-docs/how-to-file-i915-bugs.html

Filed: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/8870

Two things changed since the original post, both in the issue:

The matched GuC does not fix it. The post asked whether running 70.53.0
against a driver requesting 70.44.1 could affect the scheduler path, and
said we were about to soak the matched build. We did, and it has since
wedged twice on 70.44.1 - 2026-08-06 05:38 and 2026-08-08 09:37 - with a
byte-identical fingerprint both times: same guc_id, same flags=0x0, same
systemd-logind. So the version mismatch is ruled out. The rate looks
somewhat lower on the matched build, but on a corpus where the identical
signature has survived 0.2 min and 305 min we are not claiming that as an
improvement.

Two more survival data points, 61.7 min and 33.9 min, both on the matched
build. The 08-06 one is the cleanest example of the nondeterminism we are
reporting: the devcoredump aged out on its normal ~1h TTL, systemd units
were still executing a minute after that, and then the log simply stops -
no shutdown target, no panic.

Attached to the issue: three per-boot kernel logs and the devcoredumps for
both matched-GuC wedges. The logs have one unrelated OOM event's process
table removed; the issue body says so and offers the unfiltered originals.

Not in that batch: a drm.debug=0xe capture. Every wedge so far was caught
under a normal cmdline. Those parameters are armed now, and the box
reproduces roughly once a day, so I will attach one to the issue as soon as
I have it - no need to ask.

Thanks for pointing me at the tracker, and apologies for the delay - I only
saw your reply today.

Orestis Palampougioukis

On Wed, Aug 5, 2026 at 11:03 PM Vivi, Rodrigo <[email protected]>
wrote:

> On Tue, 2026-07-28 at 12:26 +0300, Orestis Palampougioukis wrote:
>
> Subject: drm/xe: MTL GT0 "Schedule disable failed to respond" wedge,
> sometimes fatal
>
> Hi,
>
> Reporting an xe GT0 wedge on Meteor Lake that we have been tracking on
> one machine for about four weeks. Happy to move this to the GitLab
> tracker if you prefer;
>
>
> Yes, please send it to gitlab following
>
> https://drm.pages.freedesktop.org/intel-docs/how-to-file-i915-bugs.html
>
>  sending here first since it needs no account.
>
>
>
> But this email had been lost in the freedesktop spam filter... I just
> noticed in my
> inbox...
>
>
> A device coredump is available (349K gzipped) plus ten more from earlier
> instances. I have not attached it to a list post - say the word and I
> will send it directly or attach it to a tracker issue.
>
>
> SUMMARY
> -------
>
> The xe GT0 render queue wedges with
>
>   GT0: Schedule disable failed to respond, guc_id=N
>
> followed by a GT reset that reports "reset done" but does not restore a
> working GPU. Measured rate: one wedge per ~29 hours, 16 wedges across
> 31 boots and 466 hours of uptime.
>
> The part that seems worth reporting beyond "MTL GuC hangs": the same
> signature has two very different outcomes and we cannot predict which.
> Sometimes rendering dies but the kernel keeps running normally for
> hours; sometimes the machine hard-locks within seconds. Same kernel,
> same firmware, same guilty process, same guc_id.
>
>
> HARDWARE / SOFTWARE
> -------------------
>
>   Machine    Lenovo ThinkPad P16s Gen 3 (21KS000MGM)
>              BIOS R2DET38W 1.23
>   CPU        Intel Core Ultra 7 165H (Meteor Lake)
>   iGPU       8086:7d55 rev 08, subsystem 17aa:50ea
>              GT0 main IP 12.71.4, GT1 media IP 13.0.8
>   Driver     xe, via i915.force_probe=!7d55 xe.force_probe=7d55
>   Kernel     6.17.0-35-generic, Ubuntu 24.04.4 LTS
>              also reproduces on 7.0.0-14-generic, at a higher rate
>   GuC        70.53.0 - note the driver asks for 70.44.1, see below
>   Mesa       25.2.8, libdrm 2.4.125
>   Session    native Wayland (GNOME/mutter), 3 displays all on card0
>   Note       fully muxless box; the discrete NVIDIA GPU has no display
>              connectors and its driver is blacklisted
>
>
> KERNEL LOG SIGNATURE
> --------------------
>
> The entire event, inside 11 ms:
>
>   xe 0000:00:02.0: [drm] GT0: Schedule disable failed to respond, guc_id=22
>   xe 0000:00:02.0: [drm] Xe device coredump has been created
>   xe 0000:00:02.0: [drm] GT0: trying reset from
> guc_exec_queue_timedout_job [xe]
>   xe 0000:00:02.0: [drm] GT0: reset queued
>   xe 0000:00:02.0: [drm] GT0: reset started
>   xe 0000:00:02.0: [drm] GT0: reset done
>   xe 0000:00:02.0: [drm] GT0: Timedout job: seqno=3090483,
>       lrc_seqno=3090483, guc_id=22, flags=0x0 in systemd-logind [1507]
>
> Coredump header:
>
>   Reason: Schedule disable failed to respond, guc_id=22, ret=0, guc_read=0
>   kernel: 6.17.0-35-generic   module: xe
>   Process: systemd-logind [1507]
>   PCI ID: 0x7d55   PCI revision: 0x08
>   GuC firmware: i915/mtl_guc_70.bin
>   GuC version: 70.53.0 (wanted 70.44.1)
>
> Note ret=0, guc_read=0 - the GuC is not returning an error, it is not
> answering at all.
>
> There is a second, rarer shape on the same box which storms instead of
> firing once (e.g. 30 of each plus 21 reset cycles in one burst):
>
>   GT0: VM job timed out on non-killed execqueue
>   GT0: Timedout job: seqno=N, lrc_seqno=N, guc_id=N, flags=0x4 in
>       no process [-1]
>
> flags=0x4 and "in no process [-1]", versus flags=0x0 and a named
> process in the common shape. Unclear to us whether these are one bug
> or two.
>
>
> THE NONDETERMINISTIC OUTCOME - the main thing we are reporting
> -------------------------------------------------------------
>
> Measured as (last journal entry) minus (the GT wedge line), every wedge
> since 07-18:
>
>   date          xe-hang lines   survived after wedge
>   ------------  -------------   --------------------
>   07-18 20:22    179 (storm)        0.2 min
>   07-19 01:30      3                2.5 min
>   07-19 21:10      3              304.9 min
>   07-20 07:23      3              124.3 min
>   07-25 01:48      5               10.7 min
>   07-25 15:39      3               70.0 min
>   07-26 21:23   2449 (storm)        0.0 min
>   07-28 11:27      3                0.2 min
>
> In the long-survival cases the box is fully usable except that nothing
> renders: the journal keeps flowing, timers and cron keep firing, the
> devcoredump even ages out on its normal ~1h TTL. In the short cases
> everything stops. On 07-28 the last log line is 15 s after the wedge,
> and a 30-second systemd timer that fired at 11:27:31 never fired again.
>
> Three orders of magnitude of variation on an identical fingerprint. We
> could not find a covariate. Ruled out on our corpus:
>
>   - not the hang-line count: a 3-line wedge produced both 0.2 min and
>     304.9 min
>   - not thermal: the fatal boot peaked at 310k package-throttle events,
>     while boots that survived 14.8h and 53.1h peaked at 1.25M and 959k
>   - not memory: 62 GB RAM, ~46 GB available, zero swap used, no OOM
>     anywhere
>   - not CPU: no MCE / machine-check / panic in any boot, pstore empty
>     every time
>   - not the client: see below
>
>
> THE GUILTY CONTEXT IS systemd-logind IN 8 OF 8 WEDGES
> -----------------------------------------------------
>
> systemd-logind is named in the "Timedout job" line of every wedge we
> have full logs for. Chrome appeared in early instances and was the
> obvious suspect, but we have run Chrome fully GPU-starved
> (--disable-gpu --disable-gpu-compositing, VA-API decode off) since
> 2026-07-09 and the wedges continued at the same rate with Chrome absent
> from the guilty list entirely. Xwayland appears in some storms as an
> additional owner.
>
> We read logind as an always-present session client rather than a
> culprit, but it is the one constant, and it is a light client - which is
> why "heavy GPU load causes this" never fit our data.
>
>
> FIRMWARE VERSION - possible relevance
> -------------------------------------
>
> The coredump reports:
>
>   GuC version: 70.53.0 (wanted 70.44.1)
>
> We are running a GuC nine minor versions ahead of what this driver
> requests. That was not deliberate - we were fixing a too-old-GuC
> warning and took the newest blob on the linux-firmware branch. We are
> about to soak the matched 70.44.1 build. If a driver/GuC mismatch in
> this direction is known to affect the scheduler path, that would be
> useful to know before we spend days on the soak.
>
>
> WHY xe AND NOT i915
> -------------------
>
> MAINTAINERS scopes the i915 entry to "Meteor Lake, DG2 and older" and
> the xe entry to "Lunar Lake and newer", so at first glance an MTL report
> belongs to i915. We think the code boundary is the right one here: this
> machine runs xe on MTL (the distro default cmdline is
> i915.force_probe=!7d55 xe.force_probe=7d55), the wedge is in
> drivers/gpu/drm/xe/xe_guc_submit.c, and that path is covered by the xe
> entry's F: glob with no platform qualifier. If MTL-on-xe is out of
> scope, please say so and we will refile - but the config is what Ubuntu
> 24.04 ships by default, so it is not exotic.
>
>
> PRIOR ART WE CHECKED - why we do not think this is a duplicate
> -------------------------------------------------------------
>
> drm/i915#14469 tracks Meteor Lake GuC hangs attributed to the GuC
> failing to respond when the GPU wakes from RC6, with the signature
> "GT0: GUC: TLB invalidation response timed out" plus "GPU HANG: ecode".
> Plausibly the same underlying behaviour, but:
>
>   - it is filed against i915; we are on xe, and xe has no
>     enable_rc6-style parameter (37 module parameters, none for RC6 or
>     GT idle), so its documented workarounds do not apply
>   - we have zero instances of that signature. Grepping every boot back
>     to 07-01 for "TLB invalidation response/fence timed out" returns 0
>     hits. Ours is exclusively "Schedule disable failed to respond" on
>     the exec-queue scheduler path
>   - the BIOS workaround suggested there ("disable IGFX RenderStandby")
>     is a Dasharo/coreboot option. This Lenovo firmware exposes 104
>     attributes via thinklmi and none is RenderStandby or any IGFX/RC6
>     toggle
>
> For what it is worth, RC6 is active here - the following reports
> name=gt0-rc and idle_status=gt-c6:
>
>   /sys/class/drm/card0/device/tile0/gt0/gtidle/
>
> So if #14469's RC6 mechanism generalises to xe, this box is exposed to
> it; we just cannot confirm it from our logs and cannot test the
> workaround.
>
>
> WHAT WE HAVE TRIED (all negative, listed so you can skip them)
> -------------------------------------------------------------
>
>   GuC 70.36.0 -> 70.53.0        wedges continued
>
>   kernel 6.17.0-35 -> 7.0.0-14  worse: 2 wedges in 13 min; reverted
>
>   fall back to i915             no driver binds at all on 6.17; falls
>                                 through to simpledrm, no acceleration
>                                 and no external displays
>
>   Chrome fully GPU-starved      wedges continued at the same rate;
>                                 Chrome left the guilty-context list
>                                 entirely
>
>   Chrome render onto the dGPU   GT still wedged, guilty context still
>   (PRIME offload, x11 ozone)    logind; abandoned for unrelated reasons
>
>
> WHAT WE CAN PROVIDE
> -------------------
>
>   - the 07-28 devcoredump, plus ten more from 07-20 to 07-26 (a udev
>     rule snapshots them before the ~1h TTL expires)
>   - full journalctl -k for any of the 31 boots
>   - we can run instrumented builds or debug parameters on request. This
>     box reproduces roughly once a day under normal desktop use.
>
> Thanks,
> Orestis Palampougioukis
>
>
>