Re: [PATCH 0/7] x86: Address Space Isolation, part 1: per-domain area mapping rework

On 21.08.2026 17:17, George Dunlap wrote:
> On Fri, Aug 21, 2026 at 9:45 AM Jan Beulich <[email protected]> wrote:
>> On 20.08.2026 19:43, George Dunlap wrote:
>>> One point reviewers may want to look at specifically: patch 1 changes
>>> where the per-domain page-tables are allocated from, and its commit
>>> message discusses the (minor) NUMA-placement consequence.
>>
>> While I don't recall which recent patch (series) it was, I can't very well
>> say "no new xenheap allocations please" there without also saying so here.
>> I've read over patch 1's description, and while it tries to justify this
>> accordingly, I still remain concerned. I think we simply have to accept
>> the mapping overhead, to avoid allocating from a pool which - over time -
>> is representing a decreasing portion of total memory systems have (on
>> average, and not even considering systems with extremely sparse memory
>> layouts, and with perhaps PDX compression not doing good enough to
>> compensate).
> 
> You should certainly have the same resistance to adding new xenheap
> allocations.  But looking at the numbers, I don't see that we're
> anywhere near the point where we say, "Absolutely no new xenheap
> allocations, regardless of the cost."

Well, that depends, and in part on the longer term plans with ASI. It has
been my (silent) assumption that eventually the directmap would go away
altogether when ASI is in use, with the VA space freed (almost?) all
becoming available for vmap(). With the disappearance of directmap, the
xenheap would naturally disappear as well. Hence putting stuff there in
new work actually adds to our technical debt.

>  domheap+vmap looks like it was
> cheap and easy alternative for Teddy, but the alternatives here
> aren't, compared to the cost of extra xenheap allocations.
> 
> So let's lay everything out.
> 
> My understanding is that we have the following two issues allocating
> things in the xenheap on systems larger than 4T:
> 
> - The total amount of xenheap space is limited to 4 TiB of virtual
>   address space.  On some systems, this may correspond to 4 TiB of
>   actual RAM; but on a machine whose RAM layout is sparser, the
>   actual RAM addressable in this window may be far less
> 
> - It's not symmetric NUMA-wise; so the larger the system, the more of
>   the xenheap will end up being from the same NUMA node.  This will
>   limit Xen's ability to have NUMA-local data structures, and its
>   ability to give NUMA-local data to guests running on node 0.
> 
> Looking at this series as a whole, although the first patch adds pages
> to the xenheap, the end goal of the rest of the work is to remove
> pages from the xenheap. Things added in:
> 
> - Making the perdomain area per-vCPU, with its pagetables allocated
>   from the xenheap, adds a per-vCPU L3 plus an L2+L1 pair for each
>   slot in use.  This totals 5 pages/vCPU for HVM guests and 8 pages/vCPU
>   for PV guests.
> 
>   (Note that the GDT/LDT L1s are already allocated from the xenheap
>   today, but per-domain rather than per-vCPU.)
> 
> Things removed:
> 
> - Per-pCPU stacks -- 8 xenheap pages / pCPU
> 
> - AMD VMCB - one xenheap page / vCPU
> 
> - VMX guest MSR area: 1 page per vCPU
> 
> - sub-page XSAVE areas (~2.7 KiB/vCPU of xmalloc pool today; planned
>   follow-on work aggregating other miscellaneous xmalloc'd guest state
>   should take this to about a page per vCPU)
> 
> To do some math: current security-supported limits for x86 are 4096
> pCPUs on a 12TiB system.  Suppose we have an 8:1 vCPU:pCPU ratio, and
> an average of 8 vcpus per domain.  So 32768 total vCPUs and 4096
> domains.  On a Full ASI system, vcpu-pt on all domains, per-CPU stacks
> on, all Intel HVM domains, we get numbers like the following:
> 
> Added to xenheap:
> 
> - Per-vCPU tables, 5/vCPU (L3; mapcache L2+L1; state-window L2+L1):
>   5 × 32,768 = 163,840 pages = 640 MiB
> - Per-pCPU stack tables, 2/pCPU: 2 × 4,096 = 8,192 pages = 32 MiB
>   (→ 0: these are only written at CPU bring-up and tear-down, so we
>   have already moved them to the domheap in the working branch --
>   which also makes them NUMA-local unconditionally)
> - Per-domain tables: replaced by the per-vCPU sets in vcpu-pt mode → 0
> - Total added: 172,032 pages = 672 MiB
> 
> Removed from xenheap:
> 
> - Stacks, 8/pCPU: 8 × 4,096 = 32,768 pages = 128 MiB
> - XSAVE, ~2.7 KiB/vCPU from the xmalloc pools: 32,768 × 2.7 KiB ≈
>   21,600 pages ≈ 86 MiB (0 if guests get AMX — those areas are domheap
>   today)
> - VMX guest MSR page: lazily allocated, typically absent → 0 (upper
>   bound 128 MiB if every vCPU used one)
> - Total removed: ≈ 54,400 pages ≈ 214 MiB
> 
> Net: +117,600 pages ~ +458 MiB — against a 4 TiB window (0.011%), on
> a 12 TiB host (0.0036%).

While these percentiles in particular of course look very tiny, they are
applicable only on systems having no meaningful gaps in the physical
address map. And even more generally I find all of these calculations
only partly convincing, not the least because you start out from numbers
which look pretty contrived when comparing to actual systems which would
run the new code. (Using more realistic real-system values may end up
going in favor of what you want to convey, or it may not.)

> I have explored a number of other options, to various levels of depth.
> 
> One is map_domain_page_irqoff(): If the caller promises to keep
> interrupts disabled until unmap_domain_page_irqoff(), we can safely
> perform maps in a context switch without having to worry about
> sync_lazy_execstate.  (This was actually implemented and almost sent
> on Tuesday evening, when I noticed your review of Roger's v2 saying,
> "Question is whether it's a good idea in the first place to start
> using map_domain_page() from the context switch path.  Surely there
> are possible alternatives.")  This maps all vcpu pages from the
> domheap, adding nothing to the xenheap *or* the vmap area.  But it
> costs 9 map/unmap pairs *per context switch*.

But why would not using vmap() be a necessary conclusion of my initial
comment? All I'm objecting to are new uses of the xenheap.

> I absolutely reject the idea that because on a 12TiB system with 32k
> PV vCPUs, we take up an extra 0.02% of the xenheap area, that a laptop
> running QubesOS has to do 9 maps and unmaps per context switch.  That
> is not a valid cost/benefits tradeoff.  In the worst case we could
> just add a switch to such a system, allowing people who find their
> xenheap too full to use the mapcache version instead.  (We could even
> turn this on automatically at boot based on projected xenheap
> utilization.)

Maybe, yet extra overhead may be a necessary (but hopefully only
transient) price to pay in the course of the transformation.

> There are other options I've explored:
> 
> - domheap + vmap; basically, allocate from domheap, map in the vmap
>   area.  On paper this sounds like the same thing; the problem is that
>   we don't have a simple MFN -> VA mapping, as we do in the xenheap
>   case, so the walk is a lot harder; we start to have to do lookups,
>   significantly increasing the cost over simple memory reads and math.
>   (This is the difference from the intremap table on the VT-d thread:
>   that's a leaf structure reached from a single pointer, so a
>   permanent vmap costs nothing there.  Pagetable hierarchies are
>   exactly the case where the MFN -> VA step is critical: each entry
>   read yields an MFN, which the walk has to turn into the next VA.)

The pages used here are entirely private to logic handling those page
tables. Hence a struct page_info field can very likely be used to stash
the VA of a permanent mapping. (Feels like similarly I must have
suggested this somewhere else recently, yet I don't recall the context.)

>   And if we're concerned about "xenheap creep", when we have a 4 TiB
>   ceiling, shouldn't we also be worried about "vmap creep", when we
>   have a 64 GiB ceiling?

Absolutely, and I have been mentioning the need to consider growing this
area in a number of situations (one iirc again pretty recently).

> - Stash everything we need; basically, an extension of the current
>   gdt_ldt_l1tab functionality.  Allocate everything from the domheap,
>   map it in the vmap area (moving gdt_ldt_l1tab there as well), keep
>   pointers to all the things we need to modify on context switch, so
>   we don't need to walk the tables.  This would basically be, three
>   pointers per vCPU: a pointer to its GDT/LDT L1, a pointer to its
>   per-vCPU L3, and a pointer to the per-vCPU root_pgt.  (This would
>   put ~384 MiB of mappings into the 64 GiB vmap region -- 0.6%, shared
>   with ioremap and the fixmap -- to avoid 0.02% of the xenheap
>   window.)
> 
> Both the vmap options have two complications, compared to the posted
> option.  One thing to worry about here would be the additional stress
> on the vmap allocator: It's a linear bitmap scan under one global
> lock, designed for dozens-to-hundreds of ioremaps, not ~100k
> long-lived single-page mappings (32k vCPUs x 3 pages per vCPU in the
> "stash everything" case).

Indeed, heavier use of that allocator may require work to be done there.

> The second is that we begin to run into bootstrapping issues.  With
> the xenheap approach, we can begin building and walking pagetables
> very early in boot in the same manner in which they'll be walked
> throughout Xen's lifecycle.  With the vmap approach, we need to deal
> with the fact that the vmap area itself isn't up until later.

Valid concern, yet surely possible to deal with.

> The final option I looked at was mapping the incoming vcpu's linear
> map to edit it ("altlinmap").  That still adds a map/unmap per context
> switch, and requires some additional complication to handle
> ASI/non-ASI systems.
> 
> Xen already consistently allocates its page tables from the xenheap
> whenever it needs to access them during a context switch:
> alloc_xen_pagetable() has allocated from the domheap since Hongyan's
> directmap-removal preparation (those tables are only ever walked in
> contexts where map_domain_page() works), but XPTI's per-CPU root_pgt
> is alloc_xenheap_page(), precisely because it has to be written on the
> context-switch path.  The same for the PV GDT / LDT L1 tables.  The
> series follows the same rule for the same reason.

"Rule" is a strong word. XPTI at the time needed to be done quickly.
The inability to map_domain_page() from the context switch path left
xenheap as the only viable option. Whereas with ASI, as said at the
top, phasing out directmap (and hence xenheap) as a concept is (imo) a
mid- to long-term goal.

> Ultimately, I think there's a lot of wisdom in the saying, "Premature
> optimization is the root of all evil."  As I said, it's certainly
> right to be on our guard against adding things to xenheap, and look at
> alternatives; but we're nowhere near the point where we need to say,
> "Absolutely nothing added, regardless of the cost."  The design here
> is not locking us into the pages long-term; alternate designs have a
> significant cost in terms of authoring, reviewing, code complexity and
> maintenance, and code performance.  At such time as we find systems
> where the xenheap allocations introduced in this series become a
> problem, we have a number of potential ways to mitigate the problem,
> including switching to mapcache *on systems with the problem*, or
> switching to a number of the other more complicated approaches.

I'm a little puzzled by you talking of "optimization" (premature or
not) here. In my initial reply I did point out a functional aspect, and
I made clear that I'm aware that this is going to have a performance
impact. I.e. quite the opposite of "optimization".

Jan