[REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)
Martin King <[email protected]>
| Newsgroups | dev.linux.lists.regressions,org.kernel.vger.linux-kernel,org.kernel.vger.linux-rt-users,org.kernel.vger.stable |
|---|---|
| Message-ID | <CAEB5A_91hob8ddOhW=PrO1=O7GrFmSxY3r1-_Ard6x8KHHuJGA@mail.gmail.com> |
Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
non-PREEMPT_RT desktop running a DAW.
Symptom
-------
Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
2-3 minutes of use) while the machine is otherwise idle and responsive.
Mechanism (as measured, not inferred)
-------------------------------------
REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
worker threads. During a stall:
- the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
- exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
and is enqueued but never scheduled
- its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:
reaper/livefx tid=43910 run=0.0ms wait=3406.6ms slices=2
(during a 3403ms dropout)
- meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
thread does meaningful work (largest run was 7.2ms across 3.4s)
- the machine stays responsive throughout; unrelated SCHED_OTHER work
proceeds at 84-94% of normal rate with 0-1ms runqueue wait
- all 16 cores keep taking timer interrupts at normal rates
So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
cores idle. With push disabled nothing hands it to an idle CPU.
Evidence
--------------------------------
kernel RT_PUSH_IPI playing dropouts >=25ms xruns
7.1.8 off (default) 19 min 8 (191ms - 3471ms) bursts
23/265/434 per min
7.1.3 on 39 min 0 0
7.1.8 on 42 min 0 0
The third row differs from the first ONLY in the sched feature: Identical
kernel, toolchain, GPU driver, project and audio configuration. At the
observed broken rate ~13 dropouts were expected in that window; zero were
seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
cycles at quantum 256 / 48kHz).
Workaround
----------
echo RT_PUSH_IPI > /sys/kernel/debug/sched/features
Applied at boot via a systemd unit.
System
------
AMD Ryzen 7 5800X (8c/16t), Arch Linux
kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
CONFIG_PREEMPT_RT not set