Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H

On 11.08.26 11:14, 刘杨 wrote:
> Hi,
> Thanks for the feedback.
> We agree an oob thermal handler is unlikely. The board can still run hot (we have seen package temperatures around 100°C and many TRM interrupts). We have not changed the cooling yet; we will also address this on the hardware side later.
> We only adjusted the software thermal handling path for now. With that change, the system has been under test for over 72 hours without any freeze or RCU stall.
> Do you think a software fix for the thermal handling path is still
> necessary, or is solving the overheating on the hardware side enough?
> 

If a thermal interrupt is coming, if "normal" or not, we should handle
it correctly, rather than causing even more problems this way. So, yes,
we need your patch.

Jan

> Best regards,
> liuyang
> 
>> From: "Jan Kiszka"<[email protected]>
>> Date:  Tue, Aug 11, 2026, 15:36
>> Subject:  Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H
>> To: "Florian Bezdeka"<[email protected]>, "刘杨"<[email protected]>, "Philippe Gerum"<[email protected]>
>> Cc: "xenomai"<[email protected]>, "宋健玮"<[email protected]>, "李小宁"<[email protected]>, "马君"<[email protected]>
>> On 11.08.26 09:19, Florian Bezdeka wrote:
>>> On Tue, 2026-08-11 at 10:17 +0800, 刘杨 wrote:
>>>> Hi,
>>>> We captured the following on the serial console during an RCU stall. CPU 4 was interrupted in the idle/nohz path and spun on the timer lock inside the thermal handler:
>>>>
>>>> [ 2205.999617] Sending NMI from CPU 8 to CPUs 4:
>>>> [ 2205.999619] NMI backtrace for cpu 4
>>>> [ 2205.999620] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G     U  W  OE      6.12.90-13-rkernel-cip24-x64-industry #metaOS
>>>> [ 2205.999621] Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
>>>> [ 2205.999622] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
>>>> [ 2205.999622] IRQ stage: Linux
>>>> [ 2205.999623] RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x380
>>>> [ 2205.999625] Code: 2c 24 08 0f 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 5c 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
>>>> [ 2205.999626] RSP: 0018:ffffb065802dce38 EFLAGS: 00000002
>>>> [ 2205.999627] RAX: 0000000000000001 RBX: ffff9334dfaaa428 RCX: 0000000000000000
>>>> [ 2205.999627] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9334dfa53580
>>>> [ 2205.999628] RBP: ffffb065802dce58 R08: 0000000000000001 R09: 0000000000000000
>>>> [ 2205.999629] R10: 0000000000000000 R11: ffffb065802dcff8 R12: ffff9334dfa53580
>>>> [ 2205.999629] R13: 0000000000000000 R14: 0000000029f00004 R15: ffff9334dfa53580
>>>> [ 2205.999630] FS:  0000000000000000(0000) GS:ffff9334dfa00000(0000) knlGS:0000000000000000
>>>> [ 2205.999631] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [ 2205.999631] CR2: 00007f56540d1b68 CR3: 00000001637a8005 CR4: 0000000000772ef0
>>>> [ 2205.999632] PKRU: 55555554
>>>> [ 2205.999633] Call Trace:
>>>> [ 2205.999633]  <IRQ>
>>>> [ 2205.999634]  do_raw_spin_lock+0xbe/0xd0
>>>> [ 2205.999636]  _raw_spin_lock_irqsave+0x59/0x70
>>>> [ 2205.999638]  ? lock_timer_base+0x7b/0xa0
>>>> [ 2205.999641]  lock_timer_base+0x7b/0xa0
>>>> [ 2205.999643]  add_timer_on+0x70/0x140
>>>> [ 2205.999646]  __queue_delayed_work+0xac/0x110
>>>> [ 2205.999648]  queue_delayed_work_on+0x94/0xb0
>>>> [ 2205.999650]  therm_throt_process+0x164/0x180
>>>> [ 2205.999653]  intel_thermal_interrupt+0x12b/0x350
>>>> [ 2205.999655]  __sysvec_thermal+0x46/0x180
>>>> [ 2205.999658]  sysvec_thermal+0x9f/0xc0
>>>> [ 2205.999660]  </IRQ>
>>>> [ 2205.999660]  <TASK>
>>>> [ 2205.999661]  asm_sysvec_thermal+0x1f/0x30
>>>> [ 2205.999663] RIP: 0010:_raw_spin_unlock+0x0/0x50
>>>> [ 2205.999665] Code: 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 41 bc 01 00 00 00 e8 32 04 2c ff 58 48 8d 65 f0 44 89 e0 5b 41 5c 5d c3 0f 1f 44 00 00 <f3> 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 c7 18
>>>> [ 2205.999666] RSP: 0018:ffffb065801c3e00 EFLAGS: 00000246
>>>> [ 2205.999666] RAX: 0000000000000001 RBX: 0000000100161a47 RCX: 0000000000000001
>>>> [ 2205.999667] RDX: 0000000000000004 RSI: ffffffff90217c1e RDI: ffff9334dfa54840
>>>> [ 2205.999668] RBP: ffffb065801c3e58 R08: 0000000000000001 R09: 0000000000000000
>>>> [ 2205.999668] R10: 0000000000000001 R11: 0000000000000c2c R12: ffff9334dfa53580
>>>> [ 2205.999669] R13: ffff9334dfa54840 R14: 00000197236c7580 R15: ffffb065801c3e7f
>>>> [ 2205.999672]  ? __get_next_timer_interrupt+0x132/0x380
>>>> [ 2205.999676]  timer_base_try_to_set_idle+0x17/0x30
>>>> [ 2205.999678]  tick_nohz_stop_tick+0x70/0x370
>>>> [ 2205.999681]  tick_nohz_idle_stop_tick+0x72/0xd0
>>>> [ 2205.999683]  do_idle+0xb8/0x150
>>>> [ 2205.999685]  cpu_startup_entry+0x30/0x40
>>>> [ 2205.999688]  start_secondary+0x10e/0x130
>>>> [ 2205.999690]  common_startup_64+0x13e/0x141
>>>> [ 2205.999694]  </TASK>
>>>>
>>>> We addressed this with a local kernel patch, and the issue no longer reproduces on our side after applying it.
>>>> Could you please review whether this change is correct and acceptable upstream?
>>>>
>>>> diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
>>>> index 1b9264e5b..bbd20cfe2 100644
>>>> --- a/arch/x86/include/asm/idtentry.h
>>>> +++ b/arch/x86/include/asm/idtentry.h
>>>> @@ -803,7 +803,7 @@ DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,              sysvec_deferred_error);
>>>>  # endif
>>>>
>>>>  # ifdef CONFIG_X86_THERMAL_VECTOR
>>>> -DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,           sysvec_thermal);
>>>> +DECLARE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,         sysvec_thermal);
>>>>  # else
>>>>  # define fred_sysvec_thermal                           NULL
>>>>  # endif
>>>> diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
>>>> index aecdb2d02..c432c22ed 100644
>>>> --- a/arch/x86/kernel/irq.c
>>>> +++ b/arch/x86/kernel/irq.c
>>>> @@ -605,7 +605,8 @@ static void smp_thermal_vector(void)
>>>>                        smp_processor_id());
>>>>  }
>>>>
>>>> -DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
>>>> +DEFINE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,
>>>> +                                 sysvec_thermal)
>>>>  {
>>>>         trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
>>>>         inc_irq_stat(irq_thermal_count);
>>>> diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
>>>> index 8c9d587bf..c7ade9c2c 100644
>>>> --- a/arch/x86/kernel/irq_pipeline.c
>>>> +++ b/arch/x86/kernel/irq_pipeline.c
>>>> @@ -164,6 +164,11 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
>>>>                 run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
>>>>                                         regs);
>>>>                 break;
>>>> +#ifdef CONFIG_X86_THERMAL_VECTOR
>>>> +       case THERMAL_APIC_VECTOR:
>>>> +               run_sysvec_on_irqstack_cond(__sysvec_thermal, regs);
>>>> +               break;
>>>> +#endif
>>>>         default:
>>>>                 printk_once(KERN_ERR "irq_pipeline: unexpected event"
>>>>                         " on vector #%.2x (irq=%u)", vector, irq);
>>>>
>>>  
>>> Yes, the THERMAL_APIC_VECTOR implementation has a pipelining problem.
>>>  
>>> Main question: Should the event handling take place in-band (Linux) or
>>> out-of-band (cobalt/evl)?
>>>  
>>> You selected the first option, which looks sane to me. You now delay the
>>> handling to the in-band stage.
>>  
>> It's rather unlikely that there will be any oob handler for the thermal
>> thing.
>>  
>> At the same time, the occurrence of a thermal event is very likely
>> already a problem of its own (which does not mean we should not fix its
>> handling). Is your system overheated? Will it possibly die soon when you
>> continue like that? At least, it may go into thermal throttling, and
>> that will not make your RT workload happier.
>>  
>> Jan
>>  
>> -- 
>> Siemens AG, Foundational Technologies
>> Linux Expert Center
>>  


-- 
Siemens AG, Foundational Technologies
Linux Expert Center