[PATCH 0/8] nitro_enclaves: Support multi-NUMA CPU pools and per-node allocation
Alexander Graf <[email protected]>
| Newsgroups | org.kernel.vger.linux-doc,org.kernel.vger.linux-kernel |
|---|---|
| Message-ID | <[email protected]> |
An enclave draws its vCPU cores from the driver's single CPU pool, and by design every CPU in that pool sits on one NUMA node: ne_cpus= refuses a wider CPU list with -EINVAL and "CPUs with different NUMA nodes" in dmesg. An operator who wants two enclaves on two nodes has nowhere to put the second one's cores, and an enclave wider than one node has to spare cannot be built at all. Lift it, and let a caller name the node. A pool spanning nodes records itself as node-agnostic. NE_SET_ALLOC_NUMA_NODE sets a sticky allocation target on the enclave fd and NE_ALLOC_NUMA_NODE_ANY drops it; NE_ADD_VCPU with vcpu_id 0 draws from the target and returns the id it picked, so placing an enclave across two nodes is setting the target once per node and draining the count. A named node does not spill, yielding a core on that node or NE_ERR_NO_CPUS_AVAIL_IN_POOL. A cpu_pool attribute group on the misc device reports mode, total, used and avail as CPU-list strings; pool threads are offline, so the per-node view comes from intersecting avail with the node<N>/cpu<M> symlinks under /sys/devices/system/node. I picked a per-fd target over a second NE_ADD_VCPU carrying the node on every call. NE_ADD_VCPU already reports the id it chose, so a VMM spreading an enclave over several nodes needs no new call, only the target and the count it already drains; the variant would make that same VMM learn a new ioctl for behaviour it already has. Patches 1 to 3 are not the feature: a missing .compat_ioctl, a pool mutex taken before mutex_init(), and a thread-mask index that collides cores from different packages, the last preparation with no functional change. Both fixes carry Fixes: and neither Cc: stable, the mutex splat needing CONFIG_DEBUG_MUTEXES to appear. Nothing consumes this yet; the kernel API goes first, and its shape is what nitro-cli, or a VMM like QEMU, gets built against. A dynamic pool mode and CMA-sourced enclave memory, both reusing this target, follow in separate series. I wrote this series with an AI coding assistant, which drafted the code and the changelogs, this cover letter included; I reviewed and reworked all of it before sending, and each commit carries an Assisted-by: trailer. All eight commits were built detached in three configurations (x86_64 defconfig with CONFIG_NITRO_ENCLAVES=m, the same with CONFIG_NUMA=n, and allmodconfig) for 24 warning-free driver builds, and scripts/checkpatch.pl --strict is clean on all eight. Alexander Graf (8): nitro_enclaves: Let 32-bit processes use the ioctl interface nitro_enclaves: Initialise the CPU pool mutex statically nitro_enclaves: Slot pool cores by sibling mask nitro_enclaves: Allow the CPU pool to span NUMA nodes nitro_enclaves: Add NE_SET_ALLOC_NUMA_NODE nitro_enclaves: Expose CPU pool state under sysfs Documentation: ABI: Describe nitro_enclaves cpu_pool sysfs Documentation: virt: Describe multi-NUMA Nitro Enclaves CPU pools .../sysfs-devices-virtual-misc-nitro_enclaves | 44 ++ Documentation/virt/ne_overview.rst | 25 +- MAINTAINERS | 1 + drivers/virt/nitro_enclaves/ne_misc_dev.c | 489 +++++++++++++----- drivers/virt/nitro_enclaves/ne_misc_dev.h | 16 +- include/uapi/linux/nitro_enclaves.h | 62 ++- 6 files changed, 508 insertions(+), 129 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff -- 2.47.1