Re: [PATCH v6 0/4] KVM: PPC: Expose CPU compatibility modes for nested guests

On 04/08/26 11:37 PM, Amit Machhiwal wrote:
> On POWER systems, newer processor generations can operate in compatibility
> modes corresponding to earlier generations (e.g., a Power11 system running
> in Power10 compatibility mode). In such cases, the effective CPU level
> exposed to guests differs from the physical processor generation.
>
> This creates a problem for nested virtualization. When booting a nested KVM
> guest (L2) inside a host KVM guest (L1) running in a compatibility mode,
> userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR
> and attempt to configure the nested guest accordingly. However, the L1
> partition is constrained by the compatibility level negotiated with the
> hypervisor (L0), and requests exceeding that level are rejected, leading to
> guest boot failures such as:
>
>    KVM-NESTEDv2: couldn't set guest wide elements
>
> This series provides a mechanism for userspace to query the effective CPU
> compatibility modes supported by the host, so it can select an appropriate
> CPU model for nested guests.
>
> To achieve this, the series introduces a new KVM capability and ioctl
> (KVM_CAP_PPC_COMPAT_CAPS / KVM_PPC_GET_COMPAT_CAPS) that expose the
> compatibility modes supported by the host.
>
> Why a new UAPI?
> ===============
> While cpu-version is available in /proc/device-tree/cpus/<cpu#>/cpu-version
> on both L1 booted on PowerNV and PowerVM LPARs, the UAPI approach is
> preferable for several reasons:
>
> 1. pHYP (L0) capabilities: On PowerVM, we need to rely on capabilities
>     negotiated with pHYP in KVM, not just device tree properties. The
>     cpu-version property depicts the current compat mode but doesn't point
>     to what all compat modes are supported for the nested guest.
>
> 2. procfs dependency: Not all systems run with procfs enabled (CONFIG_PROC_FS
>     is optional). Minimal configurations like buildroot might disable it, but
>     KVM ioctl works regardless since it accesses kernel data structures
>     directly.
>
> 3. Kernel validation: The kernel validates and normalizes the compatibility
>     information, ensuring userspace gets validated, consistent data.
>
> 4. Abstraction & stability: /proc/device-tree is an implementation detail.
>     The UAPI provides a stable interface that won't break if the underlying
>     mechanism changes.
>
> 5. Semantic clarity: KVM_PPC_GET_COMPAT_CAPS clearly expresses what
>     compatibility modes can be used for KVM guests, vs. parsing device tree
>     which requires understanding the semantic meaning of cpu-version.
>
> The implementation supports both:
>
>    - KVM on PowerVM (nested API v2), where compatibility information is
>      served from the cached nested_capabilities value, originally obtained
>      via the H_GUEST_GET_CAPABILITIES hypercall at module init.
>    - KVM on PowerNV (nested API v1), where compatibility is derived from the
>      device tree ("cpu-version") representing the effective processor
>      compatibility level.
>
> This allows userspace (e.g., QEMU) to select a CPU model consistent with
> the host compatibility mode, avoiding mismatches and enabling successful
> nested guest boot.
>
> Note: This series is built on top of patch [1] which must be applied first.
> Patch [1] ensures arch_compat is validated against the host compatibility
> mode before this series adds the capability query mechanism.
> Commit e4de1b9cb3b5 ("powerpc/dt_cpu_ftrs: Set CPU_FTR_P11_PVR for Power11
> and later processors") which was also a prerequisite has been merged upstream.
>
> Changes in v6:
>    - Changed KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to 0xb8 to
>      avoid placing it in the KVM_CREATE_DEVICE fd ioctl range (0xe0-0xe3);
>      relocated definition to sit alongside other PPC vm ioctls (patch 1)
>      - [Gautam]
>    - kvmppc_map_compat_capabilities(): changed parameter type from
>      'const __be32' to 'u32' to fix Sparse type annotation warning (patch 3)
>      - [Sashiko]
>    - kvmppc_get_compat_caps(): replaced of_get_property() + be32_to_cpup()
>      with of_property_read_u32() for implicit length validation and cleaner
>      endianness handling (patch 3) - [Sashiko]
>    - Documentation: corrected :Parameters: from (out) to (in/out) since
>      userspace must set size and flags before calling (patch 4) - [Sashiko]
>
> Patch summary:
>    [1/4] Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
>    [2/4] Implement capability retrieval for KVM on PowerVM (API v2)
>    [3/4] Add KVM on PowerNV support (API v1)
>    [4/4] Document the new ioctl
>
> Testing (with QEMU v4 patches and on top of patch [1]):
>
> KVM APIv1 Testing
> =================
>    On P10 PowerNV machine (L0)
>    ---------------------------
>      - P10 L1 KVM guest -> works
>        - P10 nested L2 KVM guest -> works
>        - P9 compat nested L2 KVM guest -> works
>      - P9 compat L1 KVM guest -> works
>        - P9 nested L2 KVM guest -> works
>
>    On Powernv11 TCG Guest (L0)
>    ---------------------------
>      - P11 PowerNV TCG L0 guest -> works
>      - P11 L1 KVM guest -> works
>        - P11 L2 KVM guest -> works
>      - P10 compat L1 KVM guest -> works
>        - P10 L2 KVM guest -> works
>      - P9  compat L1 KVM guest -> works
>        - P9 L2 KVM guest -> works
>
> KVM APIv2 Testing
> =================
>    On P11 PowerVM LPAR (L1)
>    ------------------------
>      - P11 L2 KVM guest -> works
>      - P10 compat L2 KVM guest -> works
>      - P9 compat L2 KVM guest fails to boot as expected
>      - Without QEMU patches but Linux patches
>        - P11 L2 KVM guest -> works
>        - P10 compat L2 KVM guest -> works
>        - P9 compat L2 KVM guest fails to boot as expected
>      - Without Linux patches but QEMU patches
>        - P11 L2 KVM guest -> works
>        - P10 compat L2 KVM guest -> works
>
>    On P11 LPAR in P10 compat (L1)
>    ------------------------------
>      - P10 (host compat) L2 KVM guest -> works
>      - Without QEMU patch but Linux patches
>        - P10 guest fails to boot as expected (error: kvm run failed Invalid argument)
>      - Without Linux patch but QEMU patches
>        - P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea)
>
>    On P10 PowerVM LPAR (L1)
>    ------------------------
>      - P10 L2 KVM guest -> works
>      - P9 compat L2 KVM guest fails to boot as expected
>
> TCG pSeries Guest
> =================
>      - P11 (default) pSeries guest boots fine
>
> ABI Extensibility Testing (struct size 32, extra member)
> =========================================================
>      - Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG,
>        QEMU retries with correct size)
>      - New struct on Linux kernel, older QEMU -> works (kernel zero-pads
>        trailing fields, QEMU gets correct data)
>
> With this series, nested guests boot successfully in configurations where
> they previously failed due to compatibility mismatches.
>
> Related QEMU series:
> ====================
> A corresponding QEMU v5 series will be sent soon.
>
> Previous QEMU versions:
> v4: https://lore.kernel.org/all/[email protected]/
> v3: https://lore.kernel.org/all/[email protected]/
> v2: https://lore.kernel.org/all/[email protected]/
> v1: https://lore.kernel.org/all/[email protected]/
>
> Previous versions:
> ==================
> v5: https://lore.kernel.org/linuxppc-dev/[email protected]/
> v4: https://lore.kernel.org/linuxppc-dev/[email protected]/
> v3: https://lore.kernel.org/linuxppc-dev/[email protected]/
> v2: https://lore.kernel.org/linuxppc-dev/[email protected]/
> v1: https://lore.kernel.org/linuxppc-dev/[email protected]/
>
> References:
> ===========
> [1] https://lore.kernel.org/all/[email protected]/
>
> Amit Machhiwal (4):
>    KVM: PPC: Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
>    KVM: PPC: Book3S HV: Implement compat CPU capability retrieval for KVM
>      on PowerVM
>    KVM: PPC: Book3S HV: Add support for compat CPU capabilities for KVM
>      on PowerNV
>    KVM: PPC: Document KVM_PPC_GET_COMPAT_CAPS ioctl
>
>   Documentation/virt/kvm/api.rst      | 79 +++++++++++++++++++++++++++++
>   arch/powerpc/include/asm/kvm_ppc.h  |  1 +
>   arch/powerpc/include/uapi/asm/kvm.h | 18 +++++++
>   arch/powerpc/kvm/book3s_hv.c        | 56 ++++++++++++++++++++
>   arch/powerpc/kvm/powerpc.c          | 71 ++++++++++++++++++++++++++
>   include/uapi/linux/kvm.h            |  3 ++
>   6 files changed, 228 insertions(+)
>
>
> base-commit: 848acc8ffe1b7cd5f1bf427b93069becfebc2c9d
> prerequisite-patch-id: 7755786f0e4f415e47065ff1972765008727fe10

Hi Amit,
I have tested this patch and it works as expected. Here is my analysis :

I booted a host with Power10 compat mode and tried following scenarios -


lscpu on host :

Architecture:                ppc64le
   Byte Order:                Little Endian
CPU(s):                      8
   On-line CPU(s) list:       0-7
Model name:                  POWER10 (architected), altivec supported


Before applying the patch :

When I am trying to bringup the guest on a compat mode host it was 
bringing up a Power11 guest and was failing as

[ 1411.578944] [   T2928] KVM-NESTEDv2: couldn't set guest wide elements
[ 1411.578963] [   T2928] vcpu 000000000b9c4155 (0):
[ 1411.578968] [   T2928] pc  = 000000007daf9790  msr = 
8000000000103000  trap = ffffffea
[ 1411.578973] [   T2928] r 0 = 8000000000003000  r16 = 0000000000000000
[ 1411.578978] [   T2928] r 1 = 000000007e581e20  r17 = 0000000000000000
[ 1411.578982] [   T2928] r 2 = 000000007db26c00  r18 = 0000000000000000
[ 1411.578985] [   T2928] r 3 = 0000000000000000  r19 = 0000000000000000
[ 1411.578989] [   T2928] r 4 = 0000000002e30c80  r20 = 0000000000000000
[ 1411.578993] [   T2928] r 5 = 000000007df80000  r21 = 0000000000000000
[ 1411.578996] [   T2928] r 6 = 0000000000200000  r22 = 00000000018c5fd6
[ 1411.579000] [   T2928] r 7 = 000000007df80000  r23 = 000000007db21cc0
[ 1411.579003] [   T2928] r 8 = 000000007db6e5d8  r24 = 000000007db66000
[ 1411.579006] [   T2928] r 9 = 000000007e6655d8  r25 = 000000007e665508
[ 1411.579010] [   T2928] r10 = 000000007db6e5d0  r26 = 00000000018c5fd6
[ 1411.579013] [   T2928] r11 = 0000000000003000  r27 = 0000000000000003
[ 1411.579017] [   T2928] r12 = 8000000000000001  r28 = 000000007db6e5e0
[ 1411.579020] [   T2928] r13 = 0000000000000000  r29 = 000000007db224b0
[ 1411.579024] [   T2928] r14 = 0000000000000000  r30 = 000000007daf274c
[ 1411.579028] [   T2928] r15 = 0000000000000000  r31 = 000000007db76000
[ 1411.579033] [   T2928] ctr = 000000007daf1b44  lr  = 000000007daf1b7c
[ 1411.579037] [   T2928] srr0 = 000000007daf9790 srr1 = 8000000000102000
[ 1411.579041] [   T2928] sprg0 = 0000000000000000 sprg1 = 000000000000ff10
[ 1411.579045] [   T2928] sprg2 = 0000000000000000 sprg3 = 0000000000000000
[ 1411.579049] [   T2928] cr = 20000402  xer = 0000000020040000 dsisr = 
00000000
[ 1411.579054] [   T2928] dar = 0000000000000000
[ 1411.579057] [   T2928] fault dar = 0000000000000000 dsisr = 00000000
[ 1411.579061] [   T2928] SLB (0 entries):
[ 1411.579064] [   T2928] lpcr = 0040000000020400 sdr1 = 
0000000000000000 last_inst = ffffffffffffffff
[ 1411.579069] [   T2928] trap=0xffffffea | pc=0x7daf9790 | 
msr=0x8000000000103000





After applying this patch along with the qemu built with it's dependent 
patch 
(https://lore.kernel.org/all/[email protected]/):

I am able to bringup a guest and it got boot up with Power10 by default:


lscpu on guest -

ltcbonn53-vm2:~ # lscpu
Architecture:                ppc64le
   Byte Order:                Little Endian
CPU(s):                      8
   On-line CPU(s) list:       0-7
Model name:                  POWER10 (architected), altivec supported
   Model:                     2.0 (pvr 0082 0200)
   Thread(s) per core:        2
   Core(s) per socket:        4
   Socket(s):                 1



Please feel free to add my tested-by:
Tested-by: Anushree Mathur <[email protected]>


Thank you,
Anushree Mathur