Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
刘杨 <[email protected]>
| Newsgroups | dev.linux.lists.xenomai |
|---|---|
| Message-ID | <ed7c43ef85386e7d9b7bfcd2895307dc2948cb81.016f3d46.7d4b.4b85.be4a.b6c524f3b5d7@feishu.cn> |
Hi,
We captured the following on the serial console during an RCU stall. CPU 4 was interrupted in the idle/nohz path and spun on the timer lock inside the thermal handler:
[ 2205.999617] Sending NMI from CPU 8 to CPUs 4:
[ 2205.999619] NMI backtrace for cpu 4
[ 2205.999620] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G U W OE 6.12.90-13-rkernel-cip24-x64-industry #metaOS
[ 2205.999621] Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 2205.999622] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
[ 2205.999622] IRQ stage: Linux
[ 2205.999623] RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x380
[ 2205.999625] Code: 2c 24 08 0f 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 5c 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
[ 2205.999626] RSP: 0018:ffffb065802dce38 EFLAGS: 00000002
[ 2205.999627] RAX: 0000000000000001 RBX: ffff9334dfaaa428 RCX: 0000000000000000
[ 2205.999627] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9334dfa53580
[ 2205.999628] RBP: ffffb065802dce58 R08: 0000000000000001 R09: 0000000000000000
[ 2205.999629] R10: 0000000000000000 R11: ffffb065802dcff8 R12: ffff9334dfa53580
[ 2205.999629] R13: 0000000000000000 R14: 0000000029f00004 R15: ffff9334dfa53580
[ 2205.999630] FS: 0000000000000000(0000) GS:ffff9334dfa00000(0000) knlGS:0000000000000000
[ 2205.999631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2205.999631] CR2: 00007f56540d1b68 CR3: 00000001637a8005 CR4: 0000000000772ef0
[ 2205.999632] PKRU: 55555554
[ 2205.999633] Call Trace:
[ 2205.999633] <IRQ>
[ 2205.999634] do_raw_spin_lock+0xbe/0xd0
[ 2205.999636] _raw_spin_lock_irqsave+0x59/0x70
[ 2205.999638] ? lock_timer_base+0x7b/0xa0
[ 2205.999641] lock_timer_base+0x7b/0xa0
[ 2205.999643] add_timer_on+0x70/0x140
[ 2205.999646] __queue_delayed_work+0xac/0x110
[ 2205.999648] queue_delayed_work_on+0x94/0xb0
[ 2205.999650] therm_throt_process+0x164/0x180
[ 2205.999653] intel_thermal_interrupt+0x12b/0x350
[ 2205.999655] __sysvec_thermal+0x46/0x180
[ 2205.999658] sysvec_thermal+0x9f/0xc0
[ 2205.999660] </IRQ>
[ 2205.999660] <TASK>
[ 2205.999661] asm_sysvec_thermal+0x1f/0x30
[ 2205.999663] RIP: 0010:_raw_spin_unlock+0x0/0x50
[ 2205.999665] Code: 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 41 bc 01 00 00 00 e8 32 04 2c ff 58 48 8d 65 f0 44 89 e0 5b 41 5c 5d c3 0f 1f 44 00 00 <f3> 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 c7 18
[ 2205.999666] RSP: 0018:ffffb065801c3e00 EFLAGS: 00000246
[ 2205.999666] RAX: 0000000000000001 RBX: 0000000100161a47 RCX: 0000000000000001
[ 2205.999667] RDX: 0000000000000004 RSI: ffffffff90217c1e RDI: ffff9334dfa54840
[ 2205.999668] RBP: ffffb065801c3e58 R08: 0000000000000001 R09: 0000000000000000
[ 2205.999668] R10: 0000000000000001 R11: 0000000000000c2c R12: ffff9334dfa53580
[ 2205.999669] R13: ffff9334dfa54840 R14: 00000197236c7580 R15: ffffb065801c3e7f
[ 2205.999672] ? __get_next_timer_interrupt+0x132/0x380
[ 2205.999676] timer_base_try_to_set_idle+0x17/0x30
[ 2205.999678] tick_nohz_stop_tick+0x70/0x370
[ 2205.999681] tick_nohz_idle_stop_tick+0x72/0xd0
[ 2205.999683] do_idle+0xb8/0x150
[ 2205.999685] cpu_startup_entry+0x30/0x40
[ 2205.999688] start_secondary+0x10e/0x130
[ 2205.999690] common_startup_64+0x13e/0x141
[ 2205.999694] </TASK>
We addressed this with a local kernel patch, and the issue no longer reproduces on our side after applying it.
Could you please review whether this change is correct and acceptable upstream?
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 1b9264e5b..bbd20cfe2 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -803,7 +803,7 @@ DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR, sysvec_deferred_error);
# endif
# ifdef CONFIG_X86_THERMAL_VECTOR
-DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR, sysvec_thermal);
+DECLARE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR, sysvec_thermal);
# else
# define fred_sysvec_thermal NULL
# endif
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index aecdb2d02..c432c22ed 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -605,7 +605,8 @@ static void smp_thermal_vector(void)
smp_processor_id());
}
-DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
+DEFINE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,
+ sysvec_thermal)
{
trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
inc_irq_stat(irq_thermal_count);
diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
index 8c9d587bf..c7ade9c2c 100644
--- a/arch/x86/kernel/irq_pipeline.c
+++ b/arch/x86/kernel/irq_pipeline.c
@@ -164,6 +164,11 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
regs);
break;
+#ifdef CONFIG_X86_THERMAL_VECTOR
+ case THERMAL_APIC_VECTOR:
+ run_sysvec_on_irqstack_cond(__sysvec_thermal, regs);
+ break;
+#endif
default:
printk_once(KERN_ERR "irq_pipeline: unexpected event"
" on vector #%.2x (irq=%u)", vector, irq);
Thanks
liuyang
> From: "刘杨"<[email protected]>
> Date: Fri, Jul 31, 2026, 20:29
> Subject: Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> To: "Jan Kiszka"<[email protected]>
> Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> Hi
> I used the following changes
>
> sudo sysctl -w kernel.panic_on_rcu_stall=1
> sudo sysctl -w kernel.ftrace_dump_on_oops=1
> sudo sysctl -w kernel.panic=0
> sudo trace-cmd start \
> -e 'cobalt*' \
> -e 'sched:*' \
> -e 'irq:*' \
> -e 'irq_vectors:*' \
> -e 'signal:*' \
> -e 'rcu:rcu_stall_warning'
> echo 'snapshot' | sudo tee /sys/kernel/debug/tracing/events/rcu/rcu_stall_warning/trigger
> echo 'traceoff' | sudo tee /sys/kernel/debug/tracing/events/rcu/rcu_stall_warning/trigger
>
> echo 10 > /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout
>
> Kernel config add
> CONFIG_DEBUG_KERNEL=y
> CONFIG_PROVE_LOCKING=y
> CONFIG_PROVE_RAW_LOCK_NESTING=y
> CONFIG_DEBUG_PREEMPT=y
>
> The log obtained is as follows:
>
> fiveages@W2-0000:~$ [18611.389806] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> [18611.395936] rcu: 6-...0: (2 ticks this GP) idle=9d34/0/0x3 softirq=7383494/7383494 fqs=1876
> [18611.404400] rcu: 11-...0: (1 ticks this GP) idle=3a5c/1/0x4000000000000002 softirq=7395092/7395092 fqs=1876
> [18611.414250] rcu: (detected by 1, t=15004 jiffies, g=14237693, q=73 ncpus=12)
> [18611.423416] rcu: rcu_preempt kthread starved for 7500 jiffies! g14237693 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=0
> [18611.434054] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
> [18611.443210] rcu: RCU grace-period kthread stack dump:
> [18611.448322] rcu: Stack dump where RCU GP kthread last ran:
> [18611.454839] Kernel panic - not syncing: RCU Stall
> [18611.454840] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G S U W OE 6.18.29-13-rkernel-x64-industry #metaOS PREEMPT(full)
> [18611.454842] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> [18611.454843] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
> [18611.454843] IRQ stage: Linux
> [18611.454843] Call Trace:
> [18611.454844] <IRQ>
> [18611.454844] dump_stack_lvl+0x2d/0xe0
> [18611.454847] dump_stack+0x14/0x20
> [18611.454848] vpanic+0x387/0x390
> [18611.454849] panic+0x5b/0x60
> [18611.454851] ? wake_up_process+0x19/0x20
> [18611.454853] rcu_sched_clock_irq+0x17f6/0x1800
> [18611.454855] ? get_jiffies_update+0x4b/0xe0
> [18611.454858] ? get_jiffies_update+0x56/0xe0
> [18611.454860] update_process_times+0x84/0xf0
> [18611.454862] tick_nohz_handler+0xa5/0x160
> [18611.454863] ? tick_do_update_jiffies64+0x150/0x150
> [18611.454864] __hrtimer_run_queues+0x1c6/0x390
> [18611.454867] hrtimer_interrupt+0x132/0x290
> [18611.454870] proxy_irq_handler+0x21/0x30
> [18611.454871] handle_synthetic_irq+0xac/0x250
> [18611.454873] do_irq_inband+0x1d/0x20
> [18611.454874] arch_do_IRQ_pipelined+0xd8/0x840
> [18611.454875] </IRQ>
> [18611.454876] <TASK>
> [18611.454876] sync_current_irq_stage+0x1b9/0x1d0
> [18611.454878] sync_irq_stage+0xbc/0xd0
> [18611.454879] synchronize_pipeline+0x62/0x90
> [18611.454880] handle_irq_pipelined_finish+0x88/0x190
> [18611.454882] arch_pipeline_entry+0xe3/0x130
> [18611.454884] sysvec_apic_timer_interrupt+0x12/0x20
> [18611.454885] asm_sysvec_apic_timer_interrupt+0x1f/0x30
> [18611.454886] RIP: 0010:cpu_idle_poll.isra.0+0x69/0x210
> [18611.454888] Code: ff e8 8b f4 ff ff 48 8b 7d 08 e8 e2 e1 ff ff e8 7d af 25 ff 65 48 8b 1d 85 0d fb 00 48 8b 03 a8 10 74 0b eb 1c f3 90 48 8b 03 <a8> 10 75 13 8b 05 21 59 cc 00 85 c0 75 ed e8 24 f1 ff ff 85 c0 75
> [18611.454889] RSP: 0000:ffffa8da40193ed8 EFLAGS: 00000202
> [18611.454890] RAX: 0000000000080040 RBX: ffff8d1900d34980 RCX: 000000000000001f
> [18611.454890] RDX: ffffffff9dd1e7aa RSI: ffffffff9f0f769a RDI: ffffffff9dd1e7aa
> [18611.454891] RBP: ffffa8da40193ee0 R08: 0000000000000000 R09: 0000000000000000
> [18611.454891] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d1900d34980
> [18611.454892] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
> [18611.454893] ? do_idle+0x7a/0x170
> [18611.454895] ? do_idle+0x7a/0x170
> [18611.454897] ? cpu_idle_poll.isra.0+0x53/0x210
> [18611.454898] do_idle+0x7a/0x170
> [18611.454899] cpu_startup_entry+0x30/0x40
> [18611.454901] start_secondary+0x109/0x150
> [18611.454902] common_startup_64+0x13e/0x141
> [18611.454905] </TASK>
> [18612.496574] Shutting down cpus with NMI
> [18612.739167] Dumping ftrace buffer:
> [18612.739169] (ftrace buffer empty)
> [18612.739171] Kernel Offset: 0x1ca00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> [18612.847320] ---[ end Kernel panic - not syncing: RCU Stall ]---
>
> > From: "刘杨"<[email protected]>
> > Date: Fri, Jul 31, 2026, 14:14
> > Subject: Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> > To: "Jan Kiszka"<[email protected]>
> > Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> > Yes, I have only installed the kernel with Dovetail and Cobalt patches. No user-space programs linked against Cobalt are running.
> >
> > Liuyang
> >
> > > From: "Jan Kiszka"<[email protected]>
> > > Date: Fri, Jul 31, 2026, 14:10
> > > Subject: Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> > > To: "刘杨"<[email protected]>
> > > Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> > > On 31.07.26 08:01, 刘杨 wrote:
> > > > Hello Jan,
> > > > No Xenomai user‑space programs are running on the customer’s system.
> > > >
> > >
> > > So, you are getting a stall in the absence of any Xenomai load, but
> > > while running a Xenomai-enabled kernel?
> > >
> > > Another thing you can try is turning on debugging switches of the
> > > kernel, specifically
> > >
> > > https://gitlab.com/cip-project/cip-kernel/cip-kernel-config/-/blob/master/cfg/lock-debugging.cfg?ref_type=heads
> > >
> > > Jan
> > >
> > > >> From: "Jan Kiszka"<[email protected]>
> > > >> Date: Fri, Jul 31, 2026, 13:44
> > > >> Subject: Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> > > >> To: "刘杨"<[email protected]>
> > > >> Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> > > >> On 31.07.26 06:25, 刘杨 wrote:
> > > >>> Hello,
> > > >>> I am using Linux kernel 6.18 with Xenomai 3.3.1. I have set i915.modeset=0 in GRUB to prevent the i915 driver from loading.
> > > >>> Since trace‑cmd logs cannot be dumped when the kernel hangs, I configured the following sysctl parameters:
> > > >>> plaintext
> > > >>>
> > > >>> sudo sysctl -w kernel.panic_on_rcu_stall=1
> > > >>> sudo sysctl -w kernel.ftrace_dump_on_oops=1
> > > >>> sudo sysctl -w kernel.panic=0
> > > >>>
> > > >>> The full serial log is attached. Could you please help analyze whether the log contains sufficient information for troubleshooting? Thank you.
> > > >>
> > > >> I'm missing cobalt events other cores and an annotation inside the trace
> > > >> where the error was noticed. That's why I proposed a different recording
> > > >> method.
> > > >>
> > > >> Also, as the RCU stall detection may be delayed too much to still see
> > > >> normal operation in the trace, you might have to tune the timeout
> > > >> (https://docs.kernel.org/RCU/stallwarn.html#fine-tuning-the-rcu-cpu-stall-detector)
> > > >> until we find an earlier problem indicator.
> > > >>
> > > >> Jan
> > > >>
> > > >> --
> > > >> Siemens AG, Foundational Technologies
> > > >> Linux Expert Center
> > > >>
> > >
> > >
> > > --
> > > Siemens AG, Foundational Technologies
> > > Linux Expert Center
> > >