Re: Random system freezes with Linux 6.12 + Xenomai 3.3.1 on Intel Ultra 5 125H

On 11.08.26 09:19, Florian Bezdeka wrote:
> On Tue, 2026-08-11 at 10:17 +0800, 刘杨 wrote:
>> Hi,
>> We captured the following on the serial console during an RCU stall. CPU 4 was interrupted in the idle/nohz path and spun on the timer lock inside the thermal handler:
>>
>> [ 2205.999617] Sending NMI from CPU 8 to CPUs 4:
>> [ 2205.999619] NMI backtrace for cpu 4
>> [ 2205.999620] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G     U  W  OE      6.12.90-13-rkernel-cip24-x64-industry #metaOS
>> [ 2205.999621] Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
>> [ 2205.999622] Hardware name: Sinsegye SP8000-C340/SP8000-C340, BIOS SP80000020260708B 07/08/2026
>> [ 2205.999622] IRQ stage: Linux
>> [ 2205.999623] RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x380
>> [ 2205.999625] Code: 2c 24 08 0f 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 5c 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
>> [ 2205.999626] RSP: 0018:ffffb065802dce38 EFLAGS: 00000002
>> [ 2205.999627] RAX: 0000000000000001 RBX: ffff9334dfaaa428 RCX: 0000000000000000
>> [ 2205.999627] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9334dfa53580
>> [ 2205.999628] RBP: ffffb065802dce58 R08: 0000000000000001 R09: 0000000000000000
>> [ 2205.999629] R10: 0000000000000000 R11: ffffb065802dcff8 R12: ffff9334dfa53580
>> [ 2205.999629] R13: 0000000000000000 R14: 0000000029f00004 R15: ffff9334dfa53580
>> [ 2205.999630] FS:  0000000000000000(0000) GS:ffff9334dfa00000(0000) knlGS:0000000000000000
>> [ 2205.999631] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 2205.999631] CR2: 00007f56540d1b68 CR3: 00000001637a8005 CR4: 0000000000772ef0
>> [ 2205.999632] PKRU: 55555554
>> [ 2205.999633] Call Trace:
>> [ 2205.999633]  <IRQ>
>> [ 2205.999634]  do_raw_spin_lock+0xbe/0xd0
>> [ 2205.999636]  _raw_spin_lock_irqsave+0x59/0x70
>> [ 2205.999638]  ? lock_timer_base+0x7b/0xa0
>> [ 2205.999641]  lock_timer_base+0x7b/0xa0
>> [ 2205.999643]  add_timer_on+0x70/0x140
>> [ 2205.999646]  __queue_delayed_work+0xac/0x110
>> [ 2205.999648]  queue_delayed_work_on+0x94/0xb0
>> [ 2205.999650]  therm_throt_process+0x164/0x180
>> [ 2205.999653]  intel_thermal_interrupt+0x12b/0x350
>> [ 2205.999655]  __sysvec_thermal+0x46/0x180
>> [ 2205.999658]  sysvec_thermal+0x9f/0xc0
>> [ 2205.999660]  </IRQ>
>> [ 2205.999660]  <TASK>
>> [ 2205.999661]  asm_sysvec_thermal+0x1f/0x30
>> [ 2205.999663] RIP: 0010:_raw_spin_unlock+0x0/0x50
>> [ 2205.999665] Code: 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 41 bc 01 00 00 00 e8 32 04 2c ff 58 48 8d 65 f0 44 89 e0 5b 41 5c 5d c3 0f 1f 44 00 00 <f3> 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 c7 18
>> [ 2205.999666] RSP: 0018:ffffb065801c3e00 EFLAGS: 00000246
>> [ 2205.999666] RAX: 0000000000000001 RBX: 0000000100161a47 RCX: 0000000000000001
>> [ 2205.999667] RDX: 0000000000000004 RSI: ffffffff90217c1e RDI: ffff9334dfa54840
>> [ 2205.999668] RBP: ffffb065801c3e58 R08: 0000000000000001 R09: 0000000000000000
>> [ 2205.999668] R10: 0000000000000001 R11: 0000000000000c2c R12: ffff9334dfa53580
>> [ 2205.999669] R13: ffff9334dfa54840 R14: 00000197236c7580 R15: ffffb065801c3e7f
>> [ 2205.999672]  ? __get_next_timer_interrupt+0x132/0x380
>> [ 2205.999676]  timer_base_try_to_set_idle+0x17/0x30
>> [ 2205.999678]  tick_nohz_stop_tick+0x70/0x370
>> [ 2205.999681]  tick_nohz_idle_stop_tick+0x72/0xd0
>> [ 2205.999683]  do_idle+0xb8/0x150
>> [ 2205.999685]  cpu_startup_entry+0x30/0x40
>> [ 2205.999688]  start_secondary+0x10e/0x130
>> [ 2205.999690]  common_startup_64+0x13e/0x141
>> [ 2205.999694]  </TASK>
>>
>> We addressed this with a local kernel patch, and the issue no longer reproduces on our side after applying it.
>> Could you please review whether this change is correct and acceptable upstream?
>>
>> diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
>> index 1b9264e5b..bbd20cfe2 100644
>> --- a/arch/x86/include/asm/idtentry.h
>> +++ b/arch/x86/include/asm/idtentry.h
>> @@ -803,7 +803,7 @@ DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,              sysvec_deferred_error);
>>  # endif
>>
>>  # ifdef CONFIG_X86_THERMAL_VECTOR
>> -DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,           sysvec_thermal);
>> +DECLARE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,         sysvec_thermal);
>>  # else
>>  # define fred_sysvec_thermal                           NULL
>>  # endif
>> diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
>> index aecdb2d02..c432c22ed 100644
>> --- a/arch/x86/kernel/irq.c
>> +++ b/arch/x86/kernel/irq.c
>> @@ -605,7 +605,8 @@ static void smp_thermal_vector(void)
>>                        smp_processor_id());
>>  }
>>
>> -DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
>> +DEFINE_IDTENTRY_SYSVEC_PIPELINED(THERMAL_APIC_VECTOR,
>> +                                 sysvec_thermal)
>>  {
>>         trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
>>         inc_irq_stat(irq_thermal_count);
>> diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
>> index 8c9d587bf..c7ade9c2c 100644
>> --- a/arch/x86/kernel/irq_pipeline.c
>> +++ b/arch/x86/kernel/irq_pipeline.c
>> @@ -164,6 +164,11 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
>>                 run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
>>                                         regs);
>>                 break;
>> +#ifdef CONFIG_X86_THERMAL_VECTOR
>> +       case THERMAL_APIC_VECTOR:
>> +               run_sysvec_on_irqstack_cond(__sysvec_thermal, regs);
>> +               break;
>> +#endif
>>         default:
>>                 printk_once(KERN_ERR "irq_pipeline: unexpected event"
>>                         " on vector #%.2x (irq=%u)", vector, irq);
>>
> 
> Yes, the THERMAL_APIC_VECTOR implementation has a pipelining problem.
> 
> Main question: Should the event handling take place in-band (Linux) or
> out-of-band (cobalt/evl)?
> 
> You selected the first option, which looks sane to me. You now delay the
> handling to the in-band stage.

It's rather unlikely that there will be any oob handler for the thermal
thing.

At the same time, the occurrence of a thermal event is very likely
already a problem of its own (which does not mean we should not fix its
handling). Is your system overheated? Will it possibly die soon when you
continue like that? At least, it may go into thermal throttling, and
that will not make your RT workload happier.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center