Re: Fun with RDMA and NFS

On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
>
> On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > >
> > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > does an assortment of things that are file system specific to get the file's
> > > > data and then it copies that data to the iovec passed in as an argument.
> > > >
> > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > >
> > > > I don't understand how this "wrapper" would do that?
> > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > reads the data in to them? I'd be concerned that is less efficient
> > > > that the heavily
> > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > time and only
> > > > the fs knows how big that is.
> > >
> > > The wrapper would do the following:
> > > - look up the pages from the specified range in the vnode v_object page
> > >   queue
> > > - if the page is there and is valid, it is busied and recorded into the
> > >   ma[] array.
> > > - if the page is not in the queue, a fresh page is allocated and the
> > >   vm_pager_get_page() request is performed to obtain the content.
> > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > >   is aware of fs-specific magic.
> > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > does to fill in the pages, then that should be fine.
>
> For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> creates the buffer that contains the requested page, and does bread()
> on it.  This implictly validates the page.
> There is no uiomove() call, not even UIO_NOCOPY.
>
> ZFS performs reads as needed.
What about the case where you are writing a page that already has file
data?
What I'm thinking is, that if you are writing the entire page, you don't
want to read the data into the page before doing so.

rick

>
> >
> > rick
> >
> > > For tmpfs, it is swap-in, and so on.
> > > - The page is busied.  If called for remote read, pmap_remove_write()
> > >   will be called to ensure that no parallel modifications from user
> > >   space is possible.  If called for write, pmap_remove_all() will be
> > >   called, to ensure that no other consumers could see the partially
> > >   done io.
> > > This is a description of what would happen logically, but VM provides
> > > enough code to hide the logic, which is why I am calling the function
> > > a wrapper.
> > >
> > > You can construct the scatter-gather list from the ma[] array and
> > > pass it to the rdma function to construct the WQE to send/receive
> > > over a QP.
> > >
> > > This would be complementary to what the current NFS server code does
> > > with VOP_READ() to stuff a mbuf chain for READ RPC.
> > >
> > > When done with the RDMA OP, NFS server would need to unbusy the pages,
> > > we already have some helpers in the VM subsystem for this.
> > >
> > > The initial proposed signature was
> > > int vm_object_get_pages(vm_object_t obj, vm_prot_t prot, vm_page_t ma[],
> > >     int total_ma_len, vm_ooffset_t start, vm_ooffset_t end, int *ma_len);
> > >
> > > I am reverting back to it, dropping the asyncronicity.  The ma_len is
> > > the place where the actual number of the filled pages is returned, so
> > > filesystem plus wrapper would decide how many of pages are provided,
> > > but not more than total_ma_len.
> > >
> > > It would fill all continious slots with the valid pages, and if the page
> > > is read, then typically UFS or ZFS does read-ahead, which provide more
> > > valid pages past the read one.
> > >
>
>