Re: Fun with RDMA and NFS

On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> >
> > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > does an assortment of things that are file system specific to get the file's
> > > data and then it copies that data to the iovec passed in as an argument.
> > >
> > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > so that I can bus dma map the page(s) instead of copying data from them
> > > to other page(s) the NFS server allocates above the VOP_READ().
> > >
> > > I don't understand how this "wrapper" would do that?
> > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > reads the data in to them? I'd be concerned that is less efficient
> > > that the heavily
> > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > time and only
> > > the fs knows how big that is.
> >
> > The wrapper would do the following:
> > - look up the pages from the specified range in the vnode v_object page
> >   queue
> > - if the page is there and is valid, it is busied and recorded into the
> >   ma[] array.
> > - if the page is not in the queue, a fresh page is allocated and the
> >   vm_pager_get_page() request is performed to obtain the content.
> >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> >   is aware of fs-specific magic.
> Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> does to fill in the pages, then that should be fine.

For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
creates the buffer that contains the requested page, and does bread()
on it.  This implictly validates the page.
There is no uiomove() call, not even UIO_NOCOPY.

ZFS performs reads as needed.

> 
> rick
> 
> > For tmpfs, it is swap-in, and so on.
> > - The page is busied.  If called for remote read, pmap_remove_write()
> >   will be called to ensure that no parallel modifications from user
> >   space is possible.  If called for write, pmap_remove_all() will be
> >   called, to ensure that no other consumers could see the partially
> >   done io.
> > This is a description of what would happen logically, but VM provides
> > enough code to hide the logic, which is why I am calling the function
> > a wrapper.
> >
> > You can construct the scatter-gather list from the ma[] array and
> > pass it to the rdma function to construct the WQE to send/receive
> > over a QP.
> >
> > This would be complementary to what the current NFS server code does
> > with VOP_READ() to stuff a mbuf chain for READ RPC.
> >
> > When done with the RDMA OP, NFS server would need to unbusy the pages,
> > we already have some helpers in the VM subsystem for this.
> >
> > The initial proposed signature was
> > int vm_object_get_pages(vm_object_t obj, vm_prot_t prot, vm_page_t ma[],
> >     int total_ma_len, vm_ooffset_t start, vm_ooffset_t end, int *ma_len);
> >
> > I am reverting back to it, dropping the asyncronicity.  The ma_len is
> > the place where the actual number of the filled pages is returned, so
> > filesystem plus wrapper would decide how many of pages are provided,
> > but not more than total_ma_len.
> >
> > It would fill all continious slots with the valid pages, and if the page
> > is read, then typically UFS or ZFS does read-ahead, which provide more
> > valid pages past the read one.
> >