[PATCH] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT
Liang Hao <[email protected]>
| Newsgroups | dev.linux.lists.linux-rt-devel,org.kernel.vger.linux-doc,org.kernel.vger.linux-kernel |
|---|---|
| Message-ID | <[email protected]> |
Documentation/timers/hrtimers.rst described the high-resolution timer subsystem but did not cover the PREEMPT_RT expiry-mode semantics. On a PREEMPT_RT kernel a timer that is not explicitly marked HRTIMER_MODE_HARD is forced into softirq expiry and its callback runs on the per-CPU ktimers/%u thread at SCHED_FIFO priority 1, regardless of the priority of the task that armed it -- a SCHED_FIFO task's priority does not extend into its timer callback. This is a recurring source of hard-to-diagnose latency for RT/DL authors who assume the opposite. Add a dedicated "Expiry modes and PREEMPT_RT" section that documents: - the distinction between HRTIMER_MODE_HARD and HRTIMER_MODE_SOFT; - the fact that unmarked timers are forced into softirq expiry on PREEMPT_RT (__hrtimer_setup()); - the role of the per-CPU ktimers/%u thread and its fixed SCHED_FIFO priority 1 (sched_set_fifo_low()); - the absence of priority inheritance between the arming task and the timer callback on RT; - the sleeper exception, where hrtimer_setup_sleeper() automatically marks RT/DL-armed timers HRTIMER_MODE_HARD (__hrtimer_setup_sleeper()); - the critical distinction between the *requested* expiry mode (passed by the caller) and the *effective* execution context (chosen by the kernel and stored in timer->is_soft). The hrtimer_start tracepoint logs the *requested* mode, not the effective one. On PREEMPT_RT a timer armed with the default mode (e.g. ABS or REL without HARD/SOFT flags) is implicitly forced into softirq expiry, so the effective soft nature must be inferred from the *absence* of explicit mode flags in the trace rather than read directly. Documentation only; no code or behaviour change. Signed-off-by: Liang Hao <[email protected]> --- Documentation/timers/hrtimers.rst | 66 +++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/Documentation/timers/hrtimers.rst b/Documentation/timers/hrtimers.rst index f88ff8bae89c..ff42377668c1 100644 --- a/Documentation/timers/hrtimers.rst +++ b/Documentation/timers/hrtimers.rst @@ -171,3 +171,69 @@ hrtimers-based high-resolution clock implementation, so the hrtimers code got a healthy amount of testing and use in practice. Thomas Gleixner, Ingo Molnar + + +Expiry modes and PREEMPT_RT +--------------------------- + +Each hrtimer carries an expiry mode that determines the execution context +of its callback: + + * ``HRTIMER_MODE_HARD`` -- the callback runs in hard interrupt context. + It must be hardirq-safe (no sleeping locks, no allocations, no + scheduling). + * ``HRTIMER_MODE_SOFT`` -- the callback runs in softirq context and may + use operations that are not hardirq-safe. + * Default (neither flag) -- the mode is selected by the subsystem or + the kernel configuration. + +On ``CONFIG_PREEMPT_RT`` the choice is not optional for most timers: +any timer not explicitly marked ``HRTIMER_MODE_HARD`` is forced into +softirq expiry (see ``__hrtimer_setup()``). Instead of executing in +hardirq context or within the context of the task that armed it, the +callback runs on the per-CPU ``ktimers/%u`` thread. + +The ``ktimersd`` thread operates at ``SCHED_FIFO`` priority 1 (established +via ``sched_set_fifo_low()``). This priority is fixed and **does not +inherit the priority of the task that armed the timer**. The effective +execution context is determined internally by the hrtimer subsystem at +setup time and is reflected in ``timer->is_soft``; it is not directly +visible as a mode flag at arming time. + +Consequently, even if a timer is armed by a ``SCHED_FIFO`` task with +priority 99, its callback will execute only when the ``ktimersd`` thread +(priority 1) is selected to run. + +This design ensures that timer processing does not interfere with +higher-priority real-time workloads, while still providing bounded +latency relative to ``SCHED_OTHER`` tasks. However, it also means that +latency-sensitive processing must not rely on implicit priority +inheritance through the timer arming path. + +A notable exception is the sleeper path: a timer set up via +``hrtimer_setup_sleeper()`` (used by ``clock_nanosleep()`` and similar) +that is armed by an RT or DEADLINE task is automatically marked +``HRTIMER_MODE_HARD``, so its wakeup runs in hardirq context and does +not go through ``ktimersd`` (see ``__hrtimer_setup_sleeper()``). + +**Guidelines for RT authors:** + +- If the timer callback contains logic that must execute at the priority + of the owning RT task, the timer must be declared as + ``HRTIMER_MODE_HARD``. Ensure the callback adheres to hardirq context + constraints. +- Alternatively, move the latency-sensitive logic out of the timer + callback and into a dedicated, properly prioritized kthread which is + woken by the timer. + +**Debugging context:** The ``hrtimer_start`` tracepoint logs the +*requested* expiry mode passed by the caller (e.g. ``ABS``, ``REL``, +``ABS|HARD``), not the effective mode chosen by the kernel. On +PREEMPT_RT, any timer armed with the default mode (i.e. the trace shows +``ABS`` or ``REL`` without ``|SOFT`` or ``|HARD``) is implicitly forced +into softirq expiry. When such a timer is armed by an RT or DEADLINE +task, the consequence -- the callback running at ``ktimersd`` priority +rather than the arming task's -- is the case to watch. The effective +soft nature of such timers is inferred from the *absence* of an explicit +mode flag in the trace, combined with the ``CONFIG_PREEMPT_RT`` +configuration. -- 2.50.1 (Apple Git-155)