Re: [RFC PATCH v2 00/11] KVM: Allow alternative providers of guest_memfd backed by PFNMAP memory

David Woodhouse <[email protected]> writes:

> For dedicated hosting environments, there are a few reasons to prefer
> using external PFNMAP memory for guests, rather than kernel-managed
> memory:
>
>  • No struct page overheads
>
>  • Easily managed in 1GiB pages (or larger chunks); no fragmentation/THP
>
>  • Faster kexec/KHO live update (every millisecond spent faffing around
>    with memory management is an extra millisecond of steal time taken
>    from guests)
>

Also, no need to split/merge pages when converting from private to
shared if the host mappings can't depend on the refcounts in struct
pages :).

> Today, there is only one implementation of guest_memfd: the internal
> shmem-based page cache in virt/kvm/guest_memfd.c. This series allows
> for other code to provide "a guest_memfd". This is an early path-
> finding proof of concept tested with AMD SEV-SNP as well as non-CoCo
> guests on x86 and arm64. I'm planning to build a memory-based file
> system which gives files for both guest_memfd as well as memory-based
> storage to allow things like userspace VM and orchestrator state
> to be passed over KHO/kexec.
>

Thanks for raising this, good to know that there is more interest!

Vishal and I have been working on guest_memfd with HugeTLB for a while
now. I'll be restarting upstream work on HugeTLB after the conversions
series merges.

Vishal, Frank and I brought up having guest_memfd with HugeTLB with
VM_PFNMAP mappings at a guest_memfd biweekly meeting a while ago. The
consensus was that guest_memfd HugeTLB must support GUP and so VM_PFNMAP
can't be the only way that guest_memfd supports HugeTLB.

Your proposal is true non-page-struct memory, which has been on the
discussion guest_memfd biweekly discussion backlog for a while
now. Google is definitely also interested in this.

Frank also described some use cases [1] other than HugeTLB and the
regular PAGE_SIZE pages from the buddy that is currently supported.

I submitted a proposal for LPC 2026 to discuss the interface. What I had
in mind was to have the KVM_GUEST_MEMFD_CREATE ioctl take a "template"
fd. guest_memfd will then read off what it needs from the "template" fd
(ops, like you have here), and the "template" fd could be VFIO [2] or
HugeTLB.

You have a provider_fd = open("/dev/gmem_provider"), how about passing
that provider_fd to guest_memfd, where /dev/gmem_provider could
represent the page-less memory, like:

  ioctl(kvm_fd, KVM_GUEST_MEMFD_CREATE, provider_fd);

It would be cool if the fallback could be set up flexibly, like:

  ioctl(kvm_fd, KVM_GUEST_MEMFD_CREATE,
        [provider_fd, fallback_provider_fd, ...]);

I think guest_memfd should be the wrapper around other memory providers
because then the CoCo shared/private logic and special page table
handling for both host and stage 2 page tables is centralized in
guest_memfd, and also, KVM doesn't have to understand other, different
kinds of fds other than guest_memfd.

[1] https://lore.kernel.org/all/CAPTztWajm_JLpp9BjRcX=h72r25ELrXeGkOXVachybBxLJGS=g@mail.gmail.com/
[2] https://lore.kernel.org/all/CAEvNRgFpJWQ5M5sQhGpQUV3GbBq9N+MQhhaxdxa=D8ky94SCsw@mail.gmail.com/

> This series lifts guest_memfd's operations into a small ops struct and
> converts KVM's own gmem to use it, so it becomes one implementation
> among peers rather than the only game in town.  It also adapts the
> SEV-SNP fast paths to page-less operation, plumbs iommufd to map a
> guest_memfd-backing dma-buf as CPU RAM (not MMIO), and adds the
> provider->KVM revocation primitive that makes overcommit-style reclaim
> possible.
>
> It's largely AI-built at the moment. My main design concerns are around
> how we tell that a given file is "a guest_memfd", and how we plug into
> IOMMUFD. Presenting a dma-buf was the less intrusive choice because that
> path already has support for taking pages away, but I'm less convinced
> that it's the cleaner choice in the long term.
>

IIUC from this series, you're creating a dmabuf fd out of a guest_memfd,
what does "taking pages away" mean, in this case, from where/who?

>
> [...snip...]
>