Re: rcu timeouts

This time correctly spelling the RCU list...

							Thanx, Paul

On Mon, Jul 27, 2026 at 11:36:59AM -0700, Paul E. McKenney wrote:
> On Mon, Jul 27, 2026 at 11:28:00AM +0000, GROTEPASS Frans - Contractor wrote:
> > Classified as: {OPEN}
> > 
> > Hi Paul,
> > 
> > I watched an interesting talk you gave about the RCU errors. Thanks for providing these insights. We are busy with i.MX8 development and ran into this issue with a kernel version using RT_PREEMPT enabled. With RT_PREEMPT enabled we noticed that rcu_preempt timeouts were generated and there were also kernel messages pointing to the mmc. I assembled the localized logs for you. What we used:
> > 
> > i.MX8sx8x running linux-fslc-6.12.x-fslc
> > 
> > We found a work around not needing the RT_PREEMPT, so the error is not blocking for us anymore, but if you might need extra support with this one, I can collect a sample for you with more logging in my own time.
> 
> Hello, Frans,
> 
> I added [email protected] on CC in case the other RCU maintainers have
> thoughts or suggestions.
> 
> Starting with serial_run_0012.txt, please note these lines:
> 
> [   77.010227] rcu:     3-...!: (1 GPs behind) idle=ca5c/1/0x4000000000000000 softirq=0/0 fqs=156 rcuc=21000 jiffies(starved)
> [   77.010235] rcu:     (t=21000 jiffies g=12173 q=94 ncpus=4)
> [   77.010240] rcu: rcu_preempt kthread timer wakeup didn't happen for 20351 jiffies! g12173 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x1000
> [   77.010245] rcu:     Possible timer handling issue on cpu=0 timer-softirq=2461
> 
> The "rcuc=21000 jiffies(starved)" indicates that RCU's rcuc kthread
> (which is used in place of RCU_SOFTIRQ in PREEMPT_RT) is not getting
> any CPU time.  Do you have a CPU-bound real-time process that is
> preventing this rcuc kthread from running?
> 
> In addition, RCU relies on its grace-period kthread (in this case named
> "rcu_preempt") being awakened every few jiffies so that it can check
> the state of idle CPUs.  The above lines say that this kthread was not
> awakened, which by itself would cause this CPU stall warning.
> I suggest looking carefully at your PREEMPT_RT timer setup.  Or perhaps
> whatever is preventing the rcuc kthread from running is also preventing
> the timers from running.
> 
> In serial_run_0026.txt, there is this:
> 
> [   75.643219] rcu:     0-....: (1 GPs behind) idle=4044/1/0x4000000000000000 softirq=0/0 fqs=4750 rcuc=21004 jiffies(starved)
> 
> The "rcuc=21004 jiffies(starved)" indicates that RCU's rcuc kthread
> (which is used in place of RCU_SOFTIRQ in PREEMPT_RT) is not getting
> any CPU time.  Which might (or might not) be related to the "mmc2:
> Timeout waiting for hardware interrupt" messages.
> 
> Similar messages appear in serial_run_0112.txt, serial_run_0184.txt,
> and serial_run_0330.txt.
> 
> In serial_run_0116.txt:
> 
> [   68.051976] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:^M
> [   68.051987] rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-3): P29/3:b..l^M
> [   68.052001] rcu:     (detected by 2, t=21002 jiffies, g=8581, q=6949 ncpus=4)^M
> [   68.052008] task:ksoftirqd/1     state:R  running task     stack:0     pid:29    tgid:29    ppid:2      flags:0x00000008^M
> [   68.052017] Call trace:^M
> [   68.052019]  __switch_to+0xe8/0x148^M
> [   68.052033]  __schedule+0x2bc/0x874^M
> [   68.052039]  preempt_schedule+0x48/0x60^M
> [   68.052047]  try_to_wake_up+0x128/0x44c^M
> [   68.052054]  wake_up_process+0x18/0x24^M
> [   68.052059]  hrtimer_wakeup+0x20/0x38^M
> [   68.052065]  __hrtimer_run_queues+0x138/0x1fc^M
> [   68.052071]  hrtimer_run_softirq+0x7c/0xdc^M
> [   68.052077]  handle_softirqs.isra.0+0xe8/0x1f0^M
> [   68.052083]  run_ksoftirqd+0x38/0x84^M
> [   68.052089]  smpboot_thread_fn+0x204/0x228^M
> [   68.052096]  kthread+0x110/0x114^M
> [   68.052105]  ret_from_fork+0x10/0x20^M
> 
> This could happen if your workload produced a flood of hrtimers.  Or if
> one of your hrtimer handlers had a longer-than-average loop.
> 
> In serial_run_0121.txt, there are lots of these:
> 
> [   64.721625] tlv320aic32x4 15-0018: ASoC: error at soc_component_read_no_lock on tlv320aic32x4.15-0018 for register: [0x0000005f] -110
> 
> In addition, there is the same message as above:
> 
> [   64.922583] rcu: INFO: rcu_preempt self-detected stall on CPU
> [   64.922593] rcu:     0-....: (1 GPs behind) idle=98d4/1/0x4000000000000000 softirq=0/0 fqs=4374 rcuc=21004 jiffies(starved)
> 
> Again, the "rcuc=21004 jiffies(starved)" indicates that the rcuc kthreads
> are not getting any CPU time.
> 
> I hope that this helps.
> 
> 							Thanx, Paul
> 
> > Best regards,
> > 
> > Frans
> > 
> > 
> > ------------------------------------------------------------------------------------------------------------
> > Disclaimer:
> > 
> > If you are not the intended recipient of this email, please notify the sender and
> > delete it.
> > Any unauthorized copying, disclosure or distribution of this email or its
> > attachment(s) is forbidden.
> > Thales Nederland BV will not accept liability for any damage caused by this email or
> > its attachment(s).
> > Thales Nederland BV is seated in Hengelo and is registered at the Chamber of
> > Commerce under number 06061578.
> > ------------------------------------------------------------------------------------------------------------
> > 
> 
>