Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H

Hi,
Thanks for the feedback.
We agree an oob thermal handler is unlikely. The board can still run hot (we have seen package temperatures around 100°C and many TRM interrupts). We have not changed the cooling yet; we will also address this on the hardware side later.
We only adjusted the software thermal handling path for now. With that change, the system has been under test for over 72 hours without any freeze or RCU stall.
Do you think a software fix for the thermal handling path is still
necessary, or is solving the overheating on the hardware side enough?

Best regards,
liuyang

> From: "Jan Kiszka"<[email protected]>
> Date:  Tue, Aug 11, 2026, 15:36
> Subject:  Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> To: "Florian Bezdeka"<[email protected]>, "刘杨"<[email protected]>, "Philippe Gerum"<[email protected]>
> Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> On 11.08.26 09:19, Florian Bezdeka wrote:
> > On Tue, 2026-08-11 at 10:17 +0800, 刘杨 wrote:
> >> Hi,
> >> We captured the following on the serial console during an RCU stall. CPU 4 was interrupted in the idle/nohz path and spun on the timer lock inside the thermal handler:
> >>
> >> [ 2205.999617] Sending NMI from CPU 8 to CPUs 4:
> >> [ 2205.999619] NMI backtrace for cpu 4
> >> [ 2205.999620] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G     U  W  OE      6.12.90-13-rkernel-cip24-x64-industry #metaOS
> >> [ 2205.999621] Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> >> [ 2205.999622] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
> >> [ 2205.999622] IRQ stage: Linux
> >> [ 2205.999623] RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x380
> >> [ 2205.999625] Code: 2c 24 08 0f 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 5c 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
> >> [ 2205.999626] RSP: 0018:ffffb065802dce38 EFLAGS: 00000002
> >> [ 2205.999627] RAX: 0000000000000001 RBX: ffff9334dfaaa428 RCX: 0000000000000000
> >> [ 2205.999627] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9334dfa53580
> >> [ 2205.999628] RBP: ffffb065802dce58 R08: 0000000000000001 R09: 0000000000000000
> >> [ 2205.999629] R10: 0000000000000000 R11: ffffb065802dcff8 R12: ffff9334dfa53580
> >> [ 2205.999629] R13: 0000000000000000 R14: 0000000029f00004 R15: ffff9334dfa53580
> >> [ 2205.999630] FS:  0000000000000000(0000) GS:ffff9334dfa00000(0000) knlGS:0000000000000000
> >> [ 2205.999631] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> [ 2205.999631] CR2: 00007f56540d1b68 CR3: 00000001637a8005 CR4: 0000000000772ef0
> >> [ 2205.999632] PKRU: 55555554
> >> [ 2205.999633] Call Trace:
> >> [ 2205.999633]  <IRQ>
> >> [ 2205.999634]  do_raw_spin_lock+0xbe/0xd0
> >> [ 2205.999636]  _raw_spin_lock_irqsave+0x59/0x70
> >> [ 2205.999638]  ? lock_timer_base+0x7b/0xa0
> >> [ 2205.999641]  lock_timer_base+0x7b/0xa0
> >> [ 2205.999643]  add_timer_on+0x70/0x140
> >> [ 2205.999646]  __queue_delayed_work+0xac/0x110
> >> [ 2205.999648]  queue_delayed_work_on+0x94/0xb0
> >> [ 2205.999650]  therm_throt_process+0x164/0x180
> >> [ 2205.999653]  intel_thermal_interrupt+0x12b/0x350
> >> [ 2205.999655]  __sysvec_thermal+0x46/0x180
> >> [ 2205.999658]  sysvec_thermal+0x9f/0xc0
> >> [ 2205.999660]  </IRQ>
> >> [ 2205.999660]  <TASK>
> >> [ 2205.999661]  asm_sysvec_thermal+0x1f/0x30
> >> [ 2205.999663] RIP: 0010:_raw_spin_unlock+0x0/0x50
> >> [ 2205.999665] Code: 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 41 bc 01 00 00 00 e8 32 04 2c ff 58 48 8d 65 f0 44 89 e0 5b 41 5c 5d c3 0f 1f 44 00 00 <f3> 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 c7 18
> >> [ 2205.999666] RSP: 0018:ffffb065801c3e00 EFLAGS: 00000246
> >> [ 2205.999666] RAX: 0000000000000001 RBX: 0000000100161a47 RCX: 0000000000000001
> >> [ 2205.999667] RDX: 0000000000000004 RSI: ffffffff90217c1e RDI: ffff9334dfa54840
> >> [ 2205.999668] RBP: ffffb065801c3e58 R08: 0000000000000001 R09: 0000000000000000
> >> [ 2205.999668] R10: 0000000000000001 R11: 0000000000000c2c R12: ffff9334dfa53580
> >> [ 2205.999669] R13: ffff9334dfa54840 R14: 00000197236c7580 R15: ffffb065801c3e7f
> >> [ 2205.999672]  ? __get_next_timer_interrupt+0x132/0x380
> >> [ 2205.999676]  timer_base_try_to_set_idle+0x17/0x30
> >> [ 2205.999678]  tick_nohz_stop_tick+0x70/0x370
> >> [ 2205.999681]  tick_nohz_idle_stop_tick+0x72/0xd0
> >> [ 2205.999683]  do_idle+0xb8/0x150
> >> [ 2205.999685]  cpu_startup_entry+0x30/0x40
> >> [ 2205.999688]  start_secondary+0x10e/0x130
> >> [ 2205.999690]  common_startup_64+0x13e/0x141
> >> [ 2205.999694]  </TASK>
> >>
> >> We addressed this with a local kernel patch, and the issue no longer reproduces on our side after applying it.
> >> Could you please review whether this change is correct and acceptable upstream?
> >>
> >> diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
> >> index 1b9264e5b..bbd20cfe2 100644
> >> --- a/arch/x86/include/asm/idtentry.h
> >> +++ b/arch/x86/include/asm/idtentry.h
> >> @@ -803,7 +803,7 @@ DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,              sysvec_deferred_error);
> >>  # endif
> >>
> >>  # ifdef CONFIG_X86_THERMAL_VECTOR
> >> -DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,           sysvec_thermal);
> >> +DECLARE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,         sysvec_thermal);
> >>  # else
> >>  # define fred_sysvec_thermal                           NULL
> >>  # endif
> >> diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
> >> index aecdb2d02..c432c22ed 100644
> >> --- a/arch/x86/kernel/irq.c
> >> +++ b/arch/x86/kernel/irq.c
> >> @@ -605,7 +605,8 @@ static void smp_thermal_vector(void)
> >>                        smp_processor_id());
> >>  }
> >>
> >> -DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
> >> +DEFINE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,
> >> +                                 sysvec_thermal)
> >>  {
> >>         trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
> >>         inc_irq_stat(irq_thermal_count);
> >> diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
> >> index 8c9d587bf..c7ade9c2c 100644
> >> --- a/arch/x86/kernel/irq_pipeline.c
> >> +++ b/arch/x86/kernel/irq_pipeline.c
> >> @@ -164,6 +164,11 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
> >>                 run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
> >>                                         regs);
> >>                 break;
> >> +#ifdef CONFIG_X86_THERMAL_VECTOR
> >> +       case THERMAL_APIC_VECTOR:
> >> +               run_sysvec_on_irqstack_cond(__sysvec_thermal, regs);
> >> +               break;
> >> +#endif
> >>         default:
> >>                 printk_once(KERN_ERR "irq_pipeline: unexpected event"
> >>                         " on vector #%.2x (irq=%u)", vector, irq);
> >>
> > 
> > Yes, the THERMAL_APIC_VECTOR implementation has a pipelining problem.
> > 
> > Main question: Should the event handling take place in-band (Linux) or
> > out-of-band (cobalt/evl)?
> > 
> > You selected the first option, which looks sane to me. You now delay the
> > handling to the in-band stage.
> 
> It's rather unlikely that there will be any oob handler for the thermal
> thing.
> 
> At the same time, the occurrence of a thermal event is very likely
> already a problem of its own (which does not mean we should not fix its
> handling). Is your system overheated? Will it possibly die soon when you
> continue like that? At least, it may go into thermal throttling, and
> that will not make your RT workload happier.
> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center
>