[REGRESSION] Stuck CPUs after kexec() on PowerNV

On a POWER9/PowerNV dual CPU host, when launching a pSeries VM via QEMU which exposes more than around 8 guest CPU cores, there is a rapidly increasing probability of a stuck CPU after kexec() runs, which is highly sensitive to the number of cores available in the guest VM.  If an attempt is made to pass through all cores on a 144 core system, multiple stuck CPUs are effectively guaranteed along with an associated boot failure of the guest.

Here is a typical guest dmesg after kexec(), note the CPU 8 stuck message:

[    0.000000] random: crng init done
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] Activating Kernel Userspace Access Prevention
[    0.000000] Activating Kernel Userspace Execution Prevention
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000002600000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000002600000-0x0000000040000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000008000000000 with 1.00 GiB pages
[    0.000000] lpar: Using radix MMU under hypervisor
[    0.000000] Linux version 6.12.95+deb13-powerpc64le ([email protected]) (powerpc64le-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP Debian 6.12.95-1 (2026-07-04)
[    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
[    0.000000] Found initrd at 0xc0000000064f0000:0xc000000007d7979c
[    0.000000] Hardware name: IBM pSeries (emulated by qemu) POWER9 (architected) 0x4e1203 0xf000005 of:SLOF,HEAD hv:linux,kvm pSeries
[    0.000000] printk: legacy bootconsole [udbg0] enabled
[    0.000000] Partition configured for 43 cpus.
[    0.000000] CPU maps initialized for 1 thread per core
[    0.000000]  (thread shift is 0)
[    0.000000] Allocated 1624 bytes for 43 pacas
[    0.000000] numa: Partition configured for 1 NUMA nodes.
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x8000000000
[    0.000000] dcache_bsize      = 0x80
[    0.000000] icache_bsize      = 0x80
[    0.000000] cpu_features      = 0x0001c06b8f4f9183
[    0.000000]   possible        = 0x001ffbebcf5fb187
[    0.000000]   always          = 0x0000000380008181
[    0.000000] cpu_user_features = 0xdc0065c2 0xaef00000
[    0.000000] mmu_features      = 0x1c007641
[    0.000000] firmware_features = 0x00000885455a445f
[    0.000000] vmalloc start     = 0xc008000000000000
[    0.000000] IO start          = 0xc00a000000000000
[    0.000000] vmemmap start     = 0xc00c000000000000
[    0.000000] -----------------------------------------------------
[    0.000000] NODE_DATA(0) allocated [mem 0x7fffa28f00-0x7fffa310ff]
[    0.000000] rfi-flush: fallback displacement flush available
[    0.000000] rfi-flush: ori type flush available
[    0.000000] rfi-flush: mttrig type flush available
[    0.000000] rfi-flush: patched 12 locations (ori+mttrig type flush)
[    0.000000] count-cache-flush: hardware flush enabled.
[    0.000000] link-stack-flush: software flush enabled.
[    0.000000] entry-flush: patched 61 locations (ori+mttrig type flush)
[    0.000000] uaccess-flush: patched 1 locations (ori+mttrig type flush)
[    0.000000] stf-barrier: eieio barrier available
[    0.000000] stf-barrier: patched 61 entry locations (eieio barrier)
[    0.000000] stf-barrier: patched 12 exit locations (eieio barrier)
[    0.000000] PPC64 nvram contains 65536 bytes
[    0.000000] barrier-nospec: using ORI speculation barrier
[    0.000000] barrier-nospec: patched 176 locations
[    0.000000] Top of RAM: 0x8000000000, Total RAM: 0x8000000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000007fffffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000007fffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000007fffffffff]
[    0.000000] percpu: Embedded 39 pages/cpu s120472 r0 d39272 u159744
[    0.000000] pcpu-alloc: s120472 r0 d39272 u159744 alloc=39*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42
[    0.000000] Kernel command line: root=/dev/nfs rootfstype=nfs nfsroot=192.168.2.2:/mnt/udata/nfsroots/ppc64el/astrid,v3,hard,nointr,tcp ip=192.168.2.36::192.168.2.1:255.255.255.0:embed-o-matic:enp0s0:off ast.modeset=0 console=tty0 console=hvc0,115200n8 panic=10 rw
[    0.000000] Unknown kernel command line parameters "nfsroot=192.168.2.2:/mnt/udata/nfsroots/ppc64el/astrid,v3,hard,nointr,tcp ip=192.168.2.36::192.168.2.1:255.255.255.0:embed-o-matic:enp0s0:off", will be passed to user space.
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 172032 bytes
[    0.000000] printk: log_buf_len min size: 131072 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 126384(96%)
[    0.000000] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 134217728
[    0.000000] Policy zone: Normal
[    0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=43, Nodes=1
[    0.000000] ftrace: allocating 38704 entries in 227 pages
[    0.000000] ftrace: allocated 227 pages with 5 groups
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=43.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=43
[    0.000000] RCU Tasks Rude: Setting shift to 6 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=43.
[    0.000000] RCU Tasks Trace: Setting shift to 6 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=43.
[    0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[    0.000000] xive: Using IRQ range [0-2a]
[    0.000000] xive: Interrupt handling initialized with spapr backend
[    0.000000] xive: Using priority 6 for all interrupts
[    0.000000] xive: Using 64kB queues
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] time_init: decrementer frequency = 512.000000 MHz
[    0.000000] time_init: processor frequency   = 2134.000000 MHz
[    0.000001] time_init: 56 bit decrementer (max: 7fffffffffffff)
[    0.000735] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[    0.002016] clocksource: timebase mult[1f40000] shift[24] registered
[    0.002809] clockevent: decrementer mult[83126f] shift[24] cpu[0]
[    0.002967] Console: colour dummy device 80x25
[    0.003524] printk: legacy console [tty0] enabled
[    0.004385] printk: legacy console [hvc0] enabled
[    0.005019] printk: legacy bootconsole [udbg0] disabled
[    0.005768] pid_max: default: 44032 minimum: 344
[    0.005904] LSM: initializing lsm=capability,landlock,lockdown,yama,apparmor,tomoyo,bpf,ipe,ima,evm
[    0.006096] landlock: Up and running.
[    0.006139] Yama: disabled by default; enable with sysctl kernel.yama.*
[    0.006342] AppArmor: AppArmor initialized
[    0.006439] TOMOYO Linux initialized
[    0.006858] LSM support for eBPF active
[    0.007186] Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.007404] Mountpoint-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.008424] POWER9 performance monitor hardware support registered
[    0.008547] rcu: Hierarchical SRCU implementation.
[    0.008602] rcu:     Max phase no-delay instances is 1000.
[    0.008725] Timer migration: 2 hierarchy levels; 8 children per group; 2 crossnode level
[    0.009704] smp: Bringing up secondary CPUs ...
[    5.021136] Processor 8 is stuck.
[    5.040206] smp: Brought up 1 node, 42 CPUs
[    5.040269] numa: Node 0 CPUs: 0-42
[    5.539085] node 0 deferred pages initialised in 468ms
[    5.539259] Memory: 528141096K/536870912K available (16428K kernel code, 2300K rwdata, 16340K rodata, 5080K init, 1320K bss, 8695376K reserved, 0K cma-reserved)
[    5.592658] devtmpfs: initialized
[    5.604671] PCI host bridge /pci@800000020000000  ranges:
[    5.604729]   IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[    5.604796]  MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[    5.604861]  MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[    5.605039] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.605248] futex hash table entries: 16384 (order: 9, 2097152 bytes, linear)
[    5.607696] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    5.607954] audit: initializing netlink subsys (disabled)
[    5.608195] audit: type=2000 audit(1786980313.572:1): state=initialized audit_enabled=0 res=1
[    5.608317] thermal_sys: Registered thermal governor 'fair_share'
[    5.608344] thermal_sys: Registered thermal governor 'bang_bang'
[    5.608416] thermal_sys: Registered thermal governor 'step_wise'
[    5.608576] thermal_sys: Registered thermal governor 'user_space'
[    5.608659] thermal_sys: Registered thermal governor 'power_allocator'