[REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

Martin King <[email protected]>
Newsgroups dev.linux.lists.regressions,org.kernel.vger.linux-kernel,org.kernel.vger.linux-rt-users,org.kernel.vger.stable
Message-ID <CAEB5A_91hob8ddOhW=PrO1=O7GrFmSxY3r1-_Ard6x8KHHuJGA@mail.gmail.com>
Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
non-PREEMPT_RT desktop running a DAW.

Symptom
-------
Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
2-3 minutes of use) while the machine is otherwise idle and responsive.

Mechanism (as measured, not inferred)
-------------------------------------
REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
worker threads. During a stall:

  - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
  - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
    and is enqueued but never scheduled
  - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:

        reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
        (during a 3403ms dropout)

  - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
    thread does meaningful work (largest run was 7.2ms across 3.4s)
  - the machine stays responsive throughout; unrelated SCHED_OTHER work
    proceeds at 84-94% of normal rate with 0-1ms runqueue wait
  - all 16 cores keep taking timer interrupts at normal rates

So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
cores idle. With push disabled nothing hands it to an idle CPU.

Evidence
--------------------------------
  kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
  7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
23/265/434 per min
  7.1.3   on             39 min   0                     0
  7.1.8   on             42 min   0                     0

The third row differs from the first ONLY in the sched feature: Identical
kernel, toolchain, GPU driver, project and audio configuration. At the
observed broken rate ~13 dropouts were expected in that window; zero were
seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
cycles at quantum 256 / 48kHz).

Workaround
----------
  echo RT_PUSH_IPI > /sys/kernel/debug/sched/features

Applied at boot via a systemd unit.

System
------
  AMD Ryzen 7 5800X (8c/16t), Arch Linux
  kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
  PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
  CONFIG_PREEMPT_RT not set
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.