Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround

The KDE session is actually still live on tty1 during that test, I've just
switched to TTY3. I'm away for a few days so can't confirm if I have logs
of exactly what sent those, but it may be regular background system
activity like maybe Wayland cleaning up context(?)

But the NOP is just the smallest reproducer I could come up with - I
originally caught the behavior just watching regular system activity.
When I saw sclk toggle from 0 to high and stay there with no other jobs
dispatched, iirc it was always <10 jobs showing up on that log line. My
first attempt to trigger it myself was a simple open gl task, but that
didn't work - iirc setting up the context, running it, and destructing
dispatched a larger bunch of tasks (20-something). It wasn't until I held
the context between "pokes" that I could get the manual clock toggling
behavior.

Maybe worth noting the Python version of the "poke" reproducer (used in the
demo) is using a GL stack to dispatch an empty compute shader, whereas the
c poke is the more barebones PM4 packet with NOP,  but either one acts as a
toggle.

In the wild, the pattern seems to be:
GPU idles, sclk->0
Lone task comes in and completes with nothing behind it
sclk->max
Later, another task comes in
sclk->0 (my speculation is queuing or completing a task triggers a test of
% of recent idle clocks, and since it's been sitting at max clock with
nothing to do, it sees 100% idle clocks and immediately throttles back down
to 0)

- Kevin








On Mon, Aug 24, 2026, 9:04 PM Feng, Kenneth <[email protected]> wrote:

> AMD General
>
> Thanks Kevin.
>
> At the beginning of the video, right after gpu-monitor.py is executed,
> there are 13 jobs dispatched.
>
> Are they also the NOP packets?
>
> Thanks.
>
>
>
> *From:* Kevin Hake <[email protected]>
> *Sent:* Tuesday, August 25, 2026 3:22 AM
> *To:* Feng, Kenneth <[email protected]>
> *Cc:* [email protected]; Deucher, Alexander <
> [email protected]>
> *Subject:* Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis,
> userspace workaround
>
>
>
> You don't often get email from [email protected]. Learn why this is important
> <https://aka.ms/LearnAboutSenderIdentification>
>
> Hi Kenneth,
>
> I think that's the bug - it is reporting 100% utilization but the GPU is
> idle - I know it's idle because I traced job submissions and dispatch, and
> separately drm_sched_job_done, debugfs pending fences, the kernel's
> drm-engine-* nanoseconds - everything points to nothing actually crunching,
> at least nothing real from userspace. Not to mention I'm in TTY with
> minimal system activity. You can see the behavior in this demo:
> https://asciinema.org/a/Qc5CQBW6ycb89ldd
>
>
> I'm wondering if there's a bug where utilization is calculated something
> something like, "how many idle clocks have there been since we last
> checked?". If the gpu has been asleep with sclk == 0, then it sees 0 idle
> clocks in the past n seconds (no idle clocks?? we must be really busy!),
> and sets utilization to 100. But I'm just speculating, what I can confirm
> is that with sclk at 0, a single NOP to the gpu toggles utilization to 100%
> and sclk to max. It seems odd numbers of very small work trigger the
> toggle, When larger groups of work come in it behaves more normally and
> doesn't "stick" high.
>
>
> Thanks,
>
> Kevin Hake
>
> +1 716 432 7106
>
> +52 81 1300 5479
>
>
>
>
>
> On Mon, Aug 24, 2026 at 8:10 AM Feng, Kenneth <[email protected]>
> wrote:
>
> AMD General
>
>
>
> What’s the GPU utilization(observed by sudo cat
> /sys/kernel/debug/dri/x/amdgpu_pm_info) when the sclk is highest?
>
> The policy is that if the utilization is 100%, the clock has to stay
> highest.
>
> Could you please elaborate what ‘a tiny job’ is to trigger the issue?
>
> Thanks.
>
>
>
>
>
> *From:* amd-gfx <[email protected]> *On Behalf Of *Kevin
> Hake
> *Sent:* Sunday, August 23, 2026 11:05 PM
> *To:* [email protected]
> *Cc:* Deucher, Alexander <[email protected]>
> *Subject:* [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis,
> userspace workaround
>
>
>
> Some people who received this message don't often get email from
> [email protected]. Learn why this is important
> <https://aka.ms/LearnAboutSenderIdentification>
>
>
> Hi all,
>
> Follow-up to drm/amd issue #3549 ("AMD GPU usage peaks with system in idle
> —
> Radeon RX 7600 XT"), where I posted a root-cause analysis and reproducer
> a few weeks ago (note 3586398). The thread's been quiet, so I'm summarizing
> here in case it's useful to whoever owns PMFW/swsmu for SMU 13.0.7.
>
> Behavior (RX 7600 XT / Navi 33, kernels 6.x–7.1, likely any DE (tho I'm on
> Plasma w/Wayland) or bare TTY):
>
>   - With the gfx job stream verifiably empty, a single tiny job submitted
>     while sclk is at 0 (deep sleep) latches sclk at max (~2990 MHz, ~115 W)
>     indefinitely. A second completed gfx job clears it. So at true idle the
>     card toggles between 0 and max on every stray job, and can sit at max
>     for an hour with the screen off.
>   - Light sustained load (video, a compositor, radeontop) masks it, which
>     is why the earlier "monitoring tools toggle gfxoff" hypothesis looked
>     plausible. It reproduces on demand from a TTY with nothing polling.
>   - Umio's finding that disabling GFXOFF via ppfeaturemask stops the spikes
>     is consistent with this model: without GFXOFF there is no "single job
>     from deep sleep" transition to mis-handle.
>
> How I measured:
>
>   - Job stream: a private ftrace instance on amdgpu_cs_ioctl /
>     amdgpu_sched_run_job / drm_sched_job_done, cross-checked against fdinfo
>     drm-engine counters and debugfs amdgpu_fence_info, so I could prove the
>     kernel submitted nothing while clocks moved.
>   - Clock: sampling the firmware's gpu_metrics (v1.3) at 1 kHz, matched to
>     the driver's 1 ms refresh of that table, to catch the ramp edges.
>   - Stimulus: a minimal compute dispatch with a held context (no setup/
>     teardown jobs), and later a one-packet PM4 NOP via libdrm — either one
>     toggles the latch.
>
> Everything points at the firmware's clock selection rather than the driver.
> Since PMFW can't be patched from outside, I published a small userspace
> workaround: a C daemon over libdrm/render node that submits one PM4 NOP
> when it sees sclk latched (same binary is the manual reproducer):
>
>   https://github.com/KevinHake/amdgpu-unstick
>
> Scripts, logs, and an asciinema demo are attached to the issue. If a
> narrower repro would help (specific firmware versions, mesa-free path,
> other kernels), tell me what you need and I'll run it.
>
> Thanks,
> Kevin Hake
>
>