Re: 5 gigs of contiguos kernel VM shared with userspace

On Wed, Aug 19, 2026 at 11:03:42PM +0300, Vadim Goncharov wrote:
> On Mon, 10 Aug 2026 11:22:29 -0400
> Mark Johnston <[email protected]> wrote:
> 
> > On Mon, Aug 10, 2026 at 05:12:49AM +0300, Vadim Goncharov wrote:
> > > On Sun, 9 Aug 2026 19:15:59 -0400
> > > Mark Johnston <[email protected]> wrote:
> > >   
> > > > On Mon, Aug 10, 2026 at 01:32:02AM +0300, Vadim Goncharov wrote:  
> > > > > Hello,
> > > > > 
> > > > > Are there ways to allocate in kernel a range of contiguous virtual
> > > > > address space, share it with userspace, and preferably run memory
> > > > > allocator inside it which has it's pointers relative to base address?
> > > > > Same as processes cooperating via mmap(), but for kernel<->user.
> > > > > 
> > > > > I'd like just run UMA inside it, but seems UMA zones cannot be told to
> > > > > take from particular range, or am I missing something? On user side,
> > > > > jemalloc mentions something like arenas, but seems it is too
> > > > > unprepared for even mmap() between processes case (may be adapting
> > > > > some TLSF allocator [*] is possible, but that's a premature question
> > > > > before VM one).    
> > > > 
> > > > UMA is really two fairly distinct things: a slab allocator, and a cache.
> > > > UMA zones provide per-CPU and per-NUMA domain caches, and UMA kegs
> > > > allocate and manage slabs.
> > > > 
> > > > You can have a zone without a keg: if you have some collection of
> > > > objects you want to allocate to consumers, you can use
> > > > uma_zcache_create() to create a "cache zone" which provides the usual
> > > > per-CPU etc. caches to store objects returned by a custom uz_import
> > > > callback that you provide when creating the zone.  For instance, if you
> > > > have some static array of objects (i.e., there is no need to subdivide a
> > > > slab) and just want to slap a scalable cache in front of it,
> > > > uma_zcache_create() is handy.
> > > > 
> > > > The other thing you could do is provide a custom slab allocator for the
> > > > keg via uma_zone_set_allocf().  There, you provide some page allocator,
> > > > and you can impose whatever constraints you want.  uma_small_alloc() is
> > > > used by default for page-sized slabs, and page_alloc() is used by  
> > > 
> > > Is that all ported/portable to userspace?  
> > 
> > No.  Most of it probably wouldn't be too bad.  I'd move some parts of it
> > to a uma_kern.c (e.g., uma_small_alloc() and page_alloc(), referenced
> > above, would live there). 
> 
> Have jemalloc similar thing to slabs, even if only for some limited set of
> sizes? I expect many of my objects being 32-byte size.

Yes.  jemalloc will use mmap() to allocate a large "extent", and each
extent is used as a slab for a particular size class.  It appears to be
possible to hook this, search for "extent_hooks" in the jemalloc code
and documentation.

> > But you'd also need to decide how per-CPU
> > caches should be implemented in userspace.
> 
> Oh, I did not think about that at all - so far, I presumed single-threaded
> userspace process.
>  
> > > > default for multi-page slabs.  Your implementation could return pages
> > > > from some pre-allocated, contiguous range of memory.  
> > > 
> > > That's the main [prerequisite] question: how to do in kernel thing similar
> > > to mmap(0, 0x5000000, MAP_SHARED|MAP_ALIGNED_SUPER, ...) and share this
> > > region with userspace process(es).  
> > 
> > It depends, I guess you want the region to be demand-paged? 
> 
> I don't know for sure. Likely, I'll need it wired, or both variants for
> different setups. The question is because being swapped is simpler?

It is simpler if the memory is wired.

> 
> > If so, something like:
> > 1. kmem_subinit() to find some region of the kernel address space it can
> >    use to map the area.
> > 2. Allocate a VM object.
> > 3. Use vm_map_find() to allocate the kernel virtual address region from
> >    the submap, and to associate that region with your VM object.
> > 4. Map that VM object into userspace as desired, e.g., by returning the
> >    VM object from some device node's d_mmap_single implementation.
> > 5. Now the kernel and participating userspace programs can fault on
> >    their respective mappings of that VM object.
> 
> I haven't found man page for kmem_subinit, where can I read more background on it?

I'm not aware of any documentation for it.  It's not used very much in
the kernel today.  I suggested it only because your question was "how
would I implement mmap(...) in the kernel", and in my interpretation,
this means allowing the kernel to fault on the newly allocated KVA.  By
default faulting is not allowed in the kernel map, i.e., MAP_NOFAULT is
set on kernel map entries, so you need to create a submap.

If you are willing to wire all of this memory in advance, and you don't
care where in the kernel map your memory gets mapped, then you might
look at POSIX shared memory objects and shm_map(9).  That function
does probably most of what you want.