Re: Fun with RDMA and NFS

On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> does an assortment of things that are file system specific to get the file's
> data and then it copies that data to the iovec passed in as an argument.
> 
> I still want to do exactly the same stuff, except get the pages (or a kernel
> virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> so that I can bus dma map the page(s) instead of copying data from them
> to other page(s) the NFS server allocates above the VOP_READ().
> 
> I don't understand how this "wrapper" would do that?
> (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> reads the data in to them? I'd be concerned that is less efficient
> that the heavily
> exercised code path VOP_READ() uses, reading blocks into buffers.)
> I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> time and only
> the fs knows how big that is.

The wrapper would do the following:
- look up the pages from the specified range in the vnode v_object page
  queue
- if the page is there and is valid, it is busied and recorded into the
  ma[] array.
- if the page is not in the queue, a fresh page is allocated and the
  vm_pager_get_page() request is performed to obtain the content.
  For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
  is aware of fs-specific magic.  For tmpfs, it is swap-in, and so on.
- The page is busied.  If called for remote read, pmap_remove_write()
  will be called to ensure that no parallel modifications from user
  space is possible.  If called for write, pmap_remove_all() will be
  called, to ensure that no other consumers could see the partially
  done io.
This is a description of what would happen logically, but VM provides
enough code to hide the logic, which is why I am calling the function
a wrapper.

You can construct the scatter-gather list from the ma[] array and
pass it to the rdma function to construct the WQE to send/receive
over a QP.

This would be complementary to what the current NFS server code does
with VOP_READ() to stuff a mbuf chain for READ RPC.

When done with the RDMA OP, NFS server would need to unbusy the pages,
we already have some helpers in the VM subsystem for this.

The initial proposed signature was
int vm_object_get_pages(vm_object_t obj, vm_prot_t prot, vm_page_t ma[],
    int total_ma_len, vm_ooffset_t start, vm_ooffset_t end, int *ma_len);

I am reverting back to it, dropping the asyncronicity.  The ma_len is
the place where the actual number of the filled pages is returned, so
filesystem plus wrapper would decide how many of pages are provided,
but not more than total_ma_len.

It would fill all continious slots with the valid pages, and if the page
is read, then typically UFS or ZFS does read-ahead, which provide more
valid pages past the read one.