RE: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround

AMD General

I am suspecting that clock gating could be disabled so the utilization is stuck at 100%.
When the utilization is 100%, any chance you can check on your system with debugfs: /sys/kernel/debug/dri/x/amdgpu_pm_info?
The output: "Clock Gating Flags Mask:…” indicates if clock gating is disabled or not.
Thanks.



From: Kevin Hake <[email protected]>
Sent: Tuesday, August 25, 2026 3:23 PM
To: Feng, Kenneth <[email protected]>
Cc: [email protected]; Deucher, Alexander <[email protected]>
Subject: Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround

The KDE session is actually still live on tty1 during that test, I've just switched to TTY3. I'm away for a few days so can't confirm if I have logs of exactly what sent those, but it may be regular background system activity like maybe Wayland cleaning up context(?)

But the NOP is just the smallest reproducer I could come up with - I originally caught the behavior just watching regular system activity.
When I saw sclk toggle from 0 to high and stay there with no other jobs dispatched, iirc it was always <10 jobs showing up on that log line. My first attempt to trigger it myself was a simple open gl task, but that didn't work - iirc setting up the context, running it, and destructing dispatched a larger bunch of tasks (20-something). It wasn't until I held the context between "pokes" that I could get the manual clock toggling behavior.

Maybe worth noting the Python version of the "poke" reproducer (used in the demo) is using a GL stack to dispatch an empty compute shader, whereas the c poke is the more barebones PM4 packet with NOP,  but either one acts as a toggle.

In the wild, the pattern seems to be:
GPU idles, sclk->0
Lone task comes in and completes with nothing behind it
sclk->max
Later, another task comes in
sclk->0 (my speculation is queuing or completing a task triggers a test of % of recent idle clocks, and since it's been sitting at max clock with nothing to do, it sees 100% idle clocks and immediately throttles back down to 0)

- Kevin





On Mon, Aug 24, 2026, 9:04 PM Feng, Kenneth <[email protected]<mailto:[email protected]>> wrote:

AMD General

Thanks Kevin.
At the beginning of the video, right after gpu-monitor.py is executed, there are 13 jobs dispatched.
Are they also the NOP packets?
Thanks.
[cid:[email protected]]

From: Kevin Hake <[email protected]<mailto:[email protected]>>
Sent: Tuesday, August 25, 2026 3:22 AM
To: Feng, Kenneth <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>; Deucher, Alexander <[email protected]<mailto:[email protected]>>
Subject: Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround

You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hi Kenneth,
I think that's the bug - it is reporting 100% utilization but the GPU is idle - I know it's idle because I traced job submissions and dispatch, and separately drm_sched_job_done, debugfs pending fences, the kernel's drm-engine-* nanoseconds - everything points to nothing actually crunching, at least nothing real from userspace. Not to mention I'm in TTY with minimal system activity. You can see the behavior in this demo: https://asciinema.org/a/Qc5CQBW6ycb89ldd

I'm wondering if there's a bug where utilization is calculated something something like, "how many idle clocks have there been since we last checked?". If the gpu has been asleep with sclk == 0, then it sees 0 idle clocks in the past n seconds (no idle clocks?? we must be really busy!), and sets utilization to 100. But I'm just speculating, what I can confirm is that with sclk at 0, a single NOP to the gpu toggles utilization to 100% and sclk to max. It seems odd numbers of very small work trigger the toggle, When larger groups of work come in it behaves more normally and doesn't "stick" high.

Thanks,
Kevin Hake
+1 716 432 7106
+52 81 1300 5479


On Mon, Aug 24, 2026 at 8:10 AM Feng, Kenneth <[email protected]<mailto:[email protected]>> wrote:

AMD General

What’s the GPU utilization(observed by sudo cat /sys/kernel/debug/dri/x/amdgpu_pm_info) when the sclk is highest?
The policy is that if the utilization is 100%, the clock has to stay highest.
Could you please elaborate what ‘a tiny job’ is to trigger the issue?
Thanks.


From: amd-gfx <[email protected]<mailto:[email protected]>> On Behalf Of Kevin Hake
Sent: Sunday, August 23, 2026 11:05 PM
To: [email protected]<mailto:[email protected]>
Cc: Deucher, Alexander <[email protected]<mailto:[email protected]>>
Subject: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround

Some people who received this message don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Hi all,

Follow-up to drm/amd issue #3549 ("AMD GPU usage peaks with system in idle —
Radeon RX 7600 XT"), where I posted a root-cause analysis and reproducer
a few weeks ago (note 3586398). The thread's been quiet, so I'm summarizing
here in case it's useful to whoever owns PMFW/swsmu for SMU 13.0.7.

Behavior (RX 7600 XT / Navi 33, kernels 6.x–7.1, likely any DE (tho I'm on Plasma w/Wayland) or bare TTY):

  - With the gfx job stream verifiably empty, a single tiny job submitted
    while sclk is at 0 (deep sleep) latches sclk at max (~2990 MHz, ~115 W)
    indefinitely. A second completed gfx job clears it. So at true idle the
    card toggles between 0 and max on every stray job, and can sit at max
    for an hour with the screen off.
  - Light sustained load (video, a compositor, radeontop) masks it, which
    is why the earlier "monitoring tools toggle gfxoff" hypothesis looked
    plausible. It reproduces on demand from a TTY with nothing polling.
  - Umio's finding that disabling GFXOFF via ppfeaturemask stops the spikes
    is consistent with this model: without GFXOFF there is no "single job
    from deep sleep" transition to mis-handle.

How I measured:

  - Job stream: a private ftrace instance on amdgpu_cs_ioctl /
    amdgpu_sched_run_job / drm_sched_job_done, cross-checked against fdinfo
    drm-engine counters and debugfs amdgpu_fence_info, so I could prove the
    kernel submitted nothing while clocks moved.
  - Clock: sampling the firmware's gpu_metrics (v1.3) at 1 kHz, matched to
    the driver's 1 ms refresh of that table, to catch the ramp edges.
  - Stimulus: a minimal compute dispatch with a held context (no setup/
    teardown jobs), and later a one-packet PM4 NOP via libdrm — either one
    toggles the latch.

Everything points at the firmware's clock selection rather than the driver.
Since PMFW can't be patched from outside, I published a small userspace
workaround: a C daemon over libdrm/render node that submits one PM4 NOP
when it sees sclk latched (same binary is the manual reproducer):

  https://github.com/KevinHake/amdgpu-unstick

Scripts, logs, and an asciinema demo are attached to the issue. If a
narrower repro would help (specific firmware versions, mesa-free path,
other kernels), tell me what you need and I'll run it.

Thanks,
Kevin Hake