Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H

On Tue, 2026-08-11 at 12:12 +0200, Jan Kiszka wrote:
> On 11.08.26 11:14, 刘杨 wrote:
> > Hi,
> > Thanks for the feedback.
> > We agree an oob thermal handler is unlikely. The board can still run hot (we have seen package temperatures around 100°C and many TRM interrupts). We have not changed the cooling yet; we will also address this on the hardware side later.
> > We only adjusted the software thermal handling path for now. With that change, the system has been under test for over 72 hours without any freeze or RCU stall.
> > Do you think a software fix for the thermal handling path is still
> > necessary, or is solving the overheating on the hardware side enough?
> > 
> 
> If a thermal interrupt is coming, if "normal" or not, we should handle
> it correctly, rather than causing even more problems this way. So, yes,
> we need your patch.

No question, we should fix the software part of it. Maybe I didn't
express that clearly.

The reason for the CPU stalls is most likely the broken acknowledgment
of the thermal IRQ / vector. apic_eio() in sysvec_thermal is a no-op
when pipelining is enabled, so the vector is never "cleared" at the IRQ
chip (APIC), stalling all vectors with lower priority.

Delaying the handling to the inband stage - as you proposed - fixes the
issue.

Could you please send out a formal patch for the 7.1-dovetail-rebase
branch? Backporting will be done afterwards.

Thanks!

Florian

> 
> Jan
> 
> > Best regards,
> > liuyang
> > 
> > > From: "Jan Kiszka"<[email protected]>
> > > Date:  Tue, Aug 11, 2026, 15:36
> > > Subject:  Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
> > > To: "Florian Bezdeka"<[email protected]>, "刘杨"<[email protected]>, "Philippe Gerum"<[email protected]>
> > > Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
> > > On 11.08.26 09:19, Florian Bezdeka wrote:
> > > > On Tue, 2026-08-11 at 10:17 +0800, 刘杨 wrote:
> > > > > Hi,
> > > > > We captured the following on the serial console during an RCU stall. CPU 4 was interrupted in the idle/nohz path and spun on the timer lock inside the thermal handler:
> > > > > 
> > > > > [ 2205.999617] Sending NMI from CPU 8 to CPUs 4:
> > > > > [ 2205.999619] NMI backtrace for cpu 4
> > > > > [ 2205.999620] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G     U  W  OE      6.12.90-13-rkernel-cip24-x64-industry #metaOS
> > > > > [ 2205.999621] Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> > > > > [ 2205.999622] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
> > > > > [ 2205.999622] IRQ stage: Linux
> > > > > [ 2205.999623] RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x380
> > > > > [ 2205.999625] Code: 2c 24 08 0f 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 5c 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
> > > > > [ 2205.999626] RSP: 0018:ffffb065802dce38 EFLAGS: 00000002
> > > > > [ 2205.999627] RAX: 0000000000000001 RBX: ffff9334dfaaa428 RCX: 0000000000000000
> > > > > [ 2205.999627] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9334dfa53580
> > > > > [ 2205.999628] RBP: ffffb065802dce58 R08: 0000000000000001 R09: 0000000000000000
> > > > > [ 2205.999629] R10: 0000000000000000 R11: ffffb065802dcff8 R12: ffff9334dfa53580
> > > > > [ 2205.999629] R13: 0000000000000000 R14: 0000000029f00004 R15: ffff9334dfa53580
> > > > > [ 2205.999630] FS:  0000000000000000(0000) GS:ffff9334dfa00000(0000) knlGS:0000000000000000
> > > > > [ 2205.999631] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 2205.999631] CR2: 00007f56540d1b68 CR3: 00000001637a8005 CR4: 0000000000772ef0
> > > > > [ 2205.999632] PKRU: 55555554
> > > > > [ 2205.999633] Call Trace:
> > > > > [ 2205.999633]  <IRQ>
> > > > > [ 2205.999634]  do_raw_spin_lock+0xbe/0xd0
> > > > > [ 2205.999636]  _raw_spin_lock_irqsave+0x59/0x70
> > > > > [ 2205.999638]  ? lock_timer_base+0x7b/0xa0
> > > > > [ 2205.999641]  lock_timer_base+0x7b/0xa0
> > > > > [ 2205.999643]  add_timer_on+0x70/0x140
> > > > > [ 2205.999646]  __queue_delayed_work+0xac/0x110
> > > > > [ 2205.999648]  queue_delayed_work_on+0x94/0xb0
> > > > > [ 2205.999650]  therm_throt_process+0x164/0x180
> > > > > [ 2205.999653]  intel_thermal_interrupt+0x12b/0x350
> > > > > [ 2205.999655]  __sysvec_thermal+0x46/0x180
> > > > > [ 2205.999658]  sysvec_thermal+0x9f/0xc0
> > > > > [ 2205.999660]  </IRQ>
> > > > > [ 2205.999660]  <TASK>
> > > > > [ 2205.999661]  asm_sysvec_thermal+0x1f/0x30
> > > > > [ 2205.999663] RIP: 0010:_raw_spin_unlock+0x0/0x50
> > > > > [ 2205.999665] Code: 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 41 bc 01 00 00 00 e8 32 04 2c ff 58 48 8d 65 f0 44 89 e0 5b 41 5c 5d c3 0f 1f 44 00 00 <f3> 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 c7 18
> > > > > [ 2205.999666] RSP: 0018:ffffb065801c3e00 EFLAGS: 00000246
> > > > > [ 2205.999666] RAX: 0000000000000001 RBX: 0000000100161a47 RCX: 0000000000000001
> > > > > [ 2205.999667] RDX: 0000000000000004 RSI: ffffffff90217c1e RDI: ffff9334dfa54840
> > > > > [ 2205.999668] RBP: ffffb065801c3e58 R08: 0000000000000001 R09: 0000000000000000
> > > > > [ 2205.999668] R10: 0000000000000001 R11: 0000000000000c2c R12: ffff9334dfa53580
> > > > > [ 2205.999669] R13: ffff9334dfa54840 R14: 00000197236c7580 R15: ffffb065801c3e7f
> > > > > [ 2205.999672]  ? __get_next_timer_interrupt+0x132/0x380
> > > > > [ 2205.999676]  timer_base_try_to_set_idle+0x17/0x30
> > > > > [ 2205.999678]  tick_nohz_stop_tick+0x70/0x370
> > > > > [ 2205.999681]  tick_nohz_idle_stop_tick+0x72/0xd0
> > > > > [ 2205.999683]  do_idle+0xb8/0x150
> > > > > [ 2205.999685]  cpu_startup_entry+0x30/0x40
> > > > > [ 2205.999688]  start_secondary+0x10e/0x130
> > > > > [ 2205.999690]  common_startup_64+0x13e/0x141
> > > > > [ 2205.999694]  </TASK>
> > > > > 
> > > > > We addressed this with a local kernel patch, and the issue no longer reproduces on our side after applying it.
> > > > > Could you please review whether this change is correct and acceptable upstream?
> > > > > 
> > > > > diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
> > > > > index 1b9264e5b..bbd20cfe2 100644
> > > > > --- a/arch/x86/include/asm/idtentry.h
> > > > > +++ b/arch/x86/include/asm/idtentry.h
> > > > > @@ -803,7 +803,7 @@ DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,              sysvec_deferred_error);
> > > > >  # endif
> > > > > 
> > > > >  # ifdef CONFIG_X86_THERMAL_VECTOR
> > > > > -DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,           sysvec_thermal);
> > > > > +DECLARE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,         sysvec_thermal);
> > > > >  # else
> > > > >  # define fred_sysvec_thermal                           NULL
> > > > >  # endif
> > > > > diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
> > > > > index aecdb2d02..c432c22ed 100644
> > > > > --- a/arch/x86/kernel/irq.c
> > > > > +++ b/arch/x86/kernel/irq.c
> > > > > @@ -605,7 +605,8 @@ static void smp_thermal_vector(void)
> > > > >                        smp_processor_id());
> > > > >  }
> > > > > 
> > > > > -DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
> > > > > +DEFINE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,
> > > > > +                                 sysvec_thermal)
> > > > >  {
> > > > >         trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
> > > > >         inc_irq_stat(irq_thermal_count);
> > > > > diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
> > > > > index 8c9d587bf..c7ade9c2c 100644
> > > > > --- a/arch/x86/kernel/irq_pipeline.c
> > > > > +++ b/arch/x86/kernel/irq_pipeline.c
> > > > > @@ -164,6 +164,11 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
> > > > >                 run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
> > > > >                                         regs);
> > > > >                 break;
> > > > > +#ifdef CONFIG_X86_THERMAL_VECTOR
> > > > > +       case THERMAL_APIC_VECTOR:
> > > > > +               run_sysvec_on_irqstack_cond(__sysvec_thermal, regs);
> > > > > +               break;
> > > > > +#endif
> > > > >         default:
> > > > >                 printk_once(KERN_ERR "irq_pipeline: unexpected event"
> > > > >                         " on vector #%.2x (irq=%u)", vector, irq);
> > > > > 
> > > >  
> > > > Yes, the THERMAL_APIC_VECTOR implementation has a pipelining problem.
> > > >  
> > > > Main question: Should the event handling take place in-band (Linux) or
> > > > out-of-band (cobalt/evl)?
> > > >  
> > > > You selected the first option, which looks sane to me. You now delay the
> > > > handling to the in-band stage.
> > >  
> > > It's rather unlikely that there will be any oob handler for the thermal
> > > thing.
> > >  
> > > At the same time, the occurrence of a thermal event is very likely
> > > already a problem of its own (which does not mean we should not fix its
> > > handling). Is your system overheated? Will it possibly die soon when you
> > > continue like that? At least, it may go into thermal throttling, and
> > > that will not make your RT workload happier.
> > >  
> > > Jan
> > >  
> > > -- 
> > > Siemens AG, Foundational Technologies
> > > Linux Expert Center
> > >  
> 
> 
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center