Re: [PATCH 0/7] x86: Address Space Isolation, part 1: per-domain area mapping rework

On Tue, Aug 25, 2026 at 2:28 PM Jan Beulich <[email protected]> wrote:
>
> On 25.08.2026 13:42, George Dunlap wrote:
> > On Mon, Aug 24, 2026 at 10:02 AM Jan Beulich <[email protected]> wrote:
> >> While these percentiles in particular of course look very tiny, they are
> >> applicable only on systems having no meaningful gaps in the physical
> >> address map. And even more generally I find all of these calculations
> >> only partly convincing, not the least because you start out from numbers
> >> which look pretty contrived when comparing to actual systems which would
> >> run the new code. (Using more realistic real-system values may end up
> >> going in favor of what you want to convey, or it may not.)
> >
> > To be honest, I'm inclined to think that they're not very convincing
> > because you don't actually have an idea what the problem is.  You
> > didn't specify what you were worried about, so I tried to guess a
> > scenario that I considered 95th-percentile worse case.  I don't know
> > what kinds of sparse memory layout machines you have in mind -- are
> > they written down anywhere, so that contributors can read and
> > understand what they need to consider *before* implementing?  Even now
> > you haven't even said what about my scenario you consider unrealistic,
> > much less told me parameters you think are more realistic.
>
> What I specifically considered unrealistic is that you use huge pCPU and
> vCPU counts. Yes, you're trying to do a worst case estimate, yet at the
> same time you're assuming huge amounts of memory to be available (which
> doesn't represent a "worst case").

Let me point out that you still haven't named exact numbers -- you're
still offloading that to me to try to guess or imagine.

The v1 series I posted adds a few pages per vCPU and a few pages per
pCPU into the xenheap. The problem is using up too much of the xenheap
address space.  So obviously to make a reasonable worst-case that
you're not going to dismiss as contrived, I need to maximize my pCPU
count and vCPU count.  pCPUs is easy -- we're documented as supporting
4096.  How many is a reasonable number of domains and vcpus?  Well, in
general, pCPUs are an effective limit to how many vCPUs you have total
on the system; an 8:1 vCPU overcommit is high, but not preposterously
high.

I don't understand your point about huge amounts of memory.  If you're
talking about *total RAM used*, it doesn't matter whether it comes
from the domheap or the xenheap.  The only possible reason to say
domheap is OK but xenheap is not is if you're concerned about RAM
above the 4TiB boundary.  Which can only happen on system with large
amounts of RAM, or systems with really sparse memory layouts.  Does
the analysis really change at all whether you're using 12TiB or 6TiB?

> As to sparse layouts - ones which have led to the two forms of PDX
> compression are well known (I think). The need for more recent (offset)
> form is a good example of what could go wrong here: New machines can
> always come with new layouts, potentially requiring new compressions
> approaches. So what I'm concerned about is effectively _any_ sparse
> layout that we may encounter without having a suitable PDX compression
> method readily available.

"There may be some new layout that doesn't compress well" -- it's not
uncommon for random bits of new hardware not to work well until we
supply a patch to fix it.  The position you're supporting is
effectively: "We must absolutely avoid a situation where some unknown
system is temporarily restricted in how many vCPUs it can create due
to a sparse address space, even if it means making the context switch
4x as expensive for every single current user."  I just don't think
that's a reasonable position in any shape or form.

> > I don't even know exactly what failure mode you're worried about.  Two
> > kinds of potential failures I know about:
> >  - Performance impacted because pages can't be NUMA-local
> >  - Toolstack operations (including domain creation) fail because
> > xenheap has been exhausted.
>
> One thing I can't help thinking you keep overlooking throughout your
> reply: xenheap and domheap aren't separate. There being only a
> relatively small part of it needed for the worst case estimate you did
> means nothing as to exhausting the xenheap in practice: Almost the
> entirety of it (with the DMA reserve being somewhat protected) can be
> used to build domains. Once in that state, allocations would fail no
> matter that large swathes of domheap might (have become) available
> (again).

Right, so if Xen allocates too much domheap from the directmap region,
the xenheap may be not be able to allocate any more, even if there's
plenty of memory.

So something like the following:

We have 8TiB of RAM, 4 nodes, 2TiB per node.  The user wants to start
4 2TiB guests, one pinned to each node; so she starts d1 on node 1, d2
on node 2, then tries d3 and can't start it because although there's
still 4TiB of RAM left, all the memory below 4TiB was handed out to
guests already.

Is that what you had in mind?

If I didn't agree that the xenheap represents technical debt that
needs to be removed anyway, I'd say a simpler solution would be to do
do some simple xenheap reservation, based on various factors
(including number of pCPUs, and the total amount of RAM).  Reserving
6GiB on an 8TiB system would have very little impact (the first guest
would either need to be a bit smaller, or have, and would make the
whole problem go away essentially.  The first guest would either need
to be less than 0.1% smaller, or have 0.1% of its pages on a different
node.

> That said, with what you indicated at the very bottom of your reply,
> it looks like this part of the discussion has become largely moot.

Yes, but I also want to challenge your operating principles -- to get
you to state more clearly what you're concerned about.  Also, in order
to either get you to relax a bit about the xenheap growing, or to help
you articulate more clearly what problems which contributors need to
address.

 -George