Re: [PATCH v5 0/3] ppc/kvm: Handle CPU compatibility mode correctly for nested guests

On 04/08/26 11:59 PM, Amit Machhiwal wrote:
> On POWER systems, newer processor generations can operate in compatibility
> modes corresponding to earlier generations (e.g., a Power11 system running
> in Power10 compatibility mode). In such cases, the effective CPU level
> exposed to guests differs from the physical processor generation.
>
> This creates issues for nested virtualization. When booting a nested KVM
> guest, QEMU may derive the CPU model from the raw hardware PVR and attempt
> to configure the guest accordingly. However, the host is constrained by the
> compatibility level negotiated with the hypervisor, and requests exceeding
> that level are rejected by KVM, leading to guest boot failures such as:
>
>    KVM-NESTEDv2: couldn't set guest wide elements
>
> This series addresses the issue by preventing fallback to raw mode when the
> host itself is booted in a compatibility mode, and by querying the effective
> CPU compatibility modes supported by the host via KVM. The kernel interface
> uses copy_struct_from/to_user() for forward and backward ABI compatibility.
> With these changes, QEMU ensures that nested guests are configured with CPU
> models consistent with the host compatibility mode, allowing them to boot
> correctly.
>
> Patch summary:
>    [1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes
>    [2/3] target/ppc/kvm: Add support for querying host compatibility mode
>    [3/3] target/ppc/kvm: Use host compatibility mode for nested guests
>
> Changes in v5:
>    - Patch 1: Updated KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to
>      0xb8 to match the corresponding Linux kernel v6 series change; the
>      0xe0-0xe3 range is reserved for KVM_CREATE_DEVICE fd ioctls
>
> Testing (with kernel v6 patches):
>
> KVM APIv1 Testing
> =================
>    On P10 PowerNV machine (L0)
>    ---------------------------
>      - P10 L1 KVM guest -> works
>        - P10 nested L2 KVM guest -> works
>        - P9 compat nested L2 KVM guest -> works
>      - P9 compat L1 KVM guest -> works
>        - P9 nested L2 KVM guest -> works
>
>    On Powernv11 TCG Guest (L0)
>    ---------------------------
>      - P11 PowerNV TCG L0 guest -> works
>      - P11 L1 KVM guest -> works
>        - P11 L2 KVM guest -> works
>      - P10 compat L1 KVM guest -> works
>        - P10 L2 KVM guest -> works
>      - P9  compat L1 KVM guest -> works
>        - P9 L2 KVM guest -> works
>
> KVM APIv2 Testing
> =================
>    On P11 PowerVM LPAR (L1)
>    ------------------------
>      - P11 L2 KVM guest -> works
>      - P10 compat L2 KVM guest -> works
>      - P9 compat L2 KVM guest fails to boot as expected
>      - Without QEMU patches but Linux patches
>        - P11 L2 KVM guest -> works
>        - P10 compat L2 KVM guest -> works
>        - P9 compat L2 KVM guest fails to boot as expected
>      - Without Linux patches but QEMU patches
>        - P11 L2 KVM guest -> works
>        - P10 compat L2 KVM guest -> works
>
>    On P11 LPAR in P10 compat (L1)
>    ------------------------------
>      - P10 (host compat) L2 KVM guest -> works
>      - Without QEMU patch but Linux patches
>        - P10 guest fails to boot as expected (error: kvm run failed Invalid argument)
>      - Without Linux patch but QEMU patches
>        - P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea)
>
>    On P10 PowerVM LPAR (L1)
>    ------------------------
>      - P10 L2 KVM guest -> works
>      - P9 compat L2 KVM guest fails to boot as expected
>
> TCG pSeries Guest
> =================
>      - P11 (default) pSeries guest boots fine
>
> ABI Extensibility Testing (struct size 32, extra member)
> =========================================================
>      - Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG,
>        QEMU retries with correct size)
>      - New struct on Linux kernel, older QEMU -> works (kernel zero-pads
>        trailing fields, QEMU gets correct data)
>
> Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates
> that will be synced separately once the corresponding kernel patches are
> merged.
>
> The corresponding Linux kernel patches (v6) are being posted concurrently.
>
> v4: https://lore.kernel.org/all/[email protected]/
> v3: https://lore.kernel.org/all/[email protected]/
> v2: https://lore.kernel.org/all/[email protected]/
> v1: https://lore.kernel.org/all/[email protected]/
>
> Previous kernel patch versions:
> v6: https://lore.kernel.org/all/[email protected]/
> v5: https://lore.kernel.org/all/[email protected]/
> v4: https://lore.kernel.org/all/[email protected]/
> v3: https://lore.kernel.org/all/[email protected]/
> v2: https://lore.kernel.org/all/[email protected]/
> v1: https://lore.kernel.org/all/[email protected]/
>
> Amit Machhiwal (3):
>    [DO_NOT_MERGE] linux-headers: Add uapi header changes
>    target/ppc/kvm: Add support for querying host compatibility mode
>    target/ppc/kvm: Use host compatibility mode for nested guests
>
>   hw/ppc/spapr_hcall.c            | 14 ++++++
>   linux-headers/asm-powerpc/kvm.h | 19 +++++++
>   linux-headers/linux/kvm.h       |  3 ++
>   target/ppc/kvm.c                | 87 +++++++++++++++++++++++++++++++++
>   target/ppc/kvm_ppc.h            |  7 +++
>   5 files changed, 130 insertions(+)
>
>
> base-commit: b428fe036233cbd15d37e3c027ab6ca4d3661a80
Hi Amit,
I have tested this patch and it works as expected. Here is my analysis :

I booted a host with Power10 compat mode and tried following scenarios -


lscpu on host :

Architecture:                ppc64le
   Byte Order:                Little Endian
CPU(s):                      8
   On-line CPU(s) list:       0-7
Model name:                  POWER10 (architected), altivec supported


Before applying the patch :

When I am trying to bringup the guest on a compat mode host it was 
bringing up a Power11 guest and was failing as

[ 1411.578944] [   T2928] KVM-NESTEDv2: couldn't set guest wide elements
[ 1411.578963] [   T2928] vcpu 000000000b9c4155 (0):
[ 1411.578968] [   T2928] pc  = 000000007daf9790  msr = 
8000000000103000  trap = ffffffea
[ 1411.578973] [   T2928] r 0 = 8000000000003000  r16 = 0000000000000000
[ 1411.578978] [   T2928] r 1 = 000000007e581e20  r17 = 0000000000000000
[ 1411.578982] [   T2928] r 2 = 000000007db26c00  r18 = 0000000000000000
[ 1411.578985] [   T2928] r 3 = 0000000000000000  r19 = 0000000000000000
[ 1411.578989] [   T2928] r 4 = 0000000002e30c80  r20 = 0000000000000000
[ 1411.578993] [   T2928] r 5 = 000000007df80000  r21 = 0000000000000000
[ 1411.578996] [   T2928] r 6 = 0000000000200000  r22 = 00000000018c5fd6
[ 1411.579000] [   T2928] r 7 = 000000007df80000  r23 = 000000007db21cc0
[ 1411.579003] [   T2928] r 8 = 000000007db6e5d8  r24 = 000000007db66000
[ 1411.579006] [   T2928] r 9 = 000000007e6655d8  r25 = 000000007e665508
[ 1411.579010] [   T2928] r10 = 000000007db6e5d0  r26 = 00000000018c5fd6
[ 1411.579013] [   T2928] r11 = 0000000000003000  r27 = 0000000000000003
[ 1411.579017] [   T2928] r12 = 8000000000000001  r28 = 000000007db6e5e0
[ 1411.579020] [   T2928] r13 = 0000000000000000  r29 = 000000007db224b0
[ 1411.579024] [   T2928] r14 = 0000000000000000  r30 = 000000007daf274c
[ 1411.579028] [   T2928] r15 = 0000000000000000  r31 = 000000007db76000
[ 1411.579033] [   T2928] ctr = 000000007daf1b44  lr  = 000000007daf1b7c
[ 1411.579037] [   T2928] srr0 = 000000007daf9790 srr1 = 8000000000102000
[ 1411.579041] [   T2928] sprg0 = 0000000000000000 sprg1 = 000000000000ff10
[ 1411.579045] [   T2928] sprg2 = 0000000000000000 sprg3 = 0000000000000000
[ 1411.579049] [   T2928] cr = 20000402  xer = 0000000020040000 dsisr = 
00000000
[ 1411.579054] [   T2928] dar = 0000000000000000
[ 1411.579057] [   T2928] fault dar = 0000000000000000 dsisr = 00000000
[ 1411.579061] [   T2928] SLB (0 entries):
[ 1411.579064] [   T2928] lpcr = 0040000000020400 sdr1 = 
0000000000000000 last_inst = ffffffffffffffff
[ 1411.579069] [   T2928] trap=0xffffffea | pc=0x7daf9790 | 
msr=0x8000000000103000





After applying this patch along with the kernel built with it's 
dependent patch 
(https://lore.kernel.org/all/[email protected]/):

I am able to bringup a guest and it got boot up with Power10 by default:


lscpu on guest -

ltcbonn53-vm2:~ # lscpu
Architecture:                ppc64le
   Byte Order:                Little Endian
CPU(s):                      8
   On-line CPU(s) list:       0-7
Model name:                  POWER10 (architected), altivec supported
   Model:                     2.0 (pvr 0082 0200)
   Thread(s) per core:        2
   Core(s) per socket:        4
   Socket(s):                 1



Please feel free to add my tested-by:

Tested-by: Anushree Mathur <[email protected]>


Thank you,
Anushree Mathur