Re: Fun with RDMA and NFS

On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
>
> On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > does an assortment of things that are file system specific to get the file's
> > data and then it copies that data to the iovec passed in as an argument.
> >
> > I still want to do exactly the same stuff, except get the pages (or a kernel
> > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > so that I can bus dma map the page(s) instead of copying data from them
> > to other page(s) the NFS server allocates above the VOP_READ().
> >
> > I don't understand how this "wrapper" would do that?
> > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > reads the data in to them? I'd be concerned that is less efficient
> > that the heavily
> > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > time and only
> > the fs knows how big that is.
>
> The wrapper would do the following:
> - look up the pages from the specified range in the vnode v_object page
>   queue
> - if the page is there and is valid, it is busied and recorded into the
>   ma[] array.
> - if the page is not in the queue, a fresh page is allocated and the
>   vm_pager_get_page() request is performed to obtain the content.
>   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
>   is aware of fs-specific magic.
Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
does to fill in the pages, then that should be fine.

rick

> For tmpfs, it is swap-in, and so on.
> - The page is busied.  If called for remote read, pmap_remove_write()
>   will be called to ensure that no parallel modifications from user
>   space is possible.  If called for write, pmap_remove_all() will be
>   called, to ensure that no other consumers could see the partially
>   done io.
> This is a description of what would happen logically, but VM provides
> enough code to hide the logic, which is why I am calling the function
> a wrapper.
>
> You can construct the scatter-gather list from the ma[] array and
> pass it to the rdma function to construct the WQE to send/receive
> over a QP.
>
> This would be complementary to what the current NFS server code does
> with VOP_READ() to stuff a mbuf chain for READ RPC.
>
> When done with the RDMA OP, NFS server would need to unbusy the pages,
> we already have some helpers in the VM subsystem for this.
>
> The initial proposed signature was
> int vm_object_get_pages(vm_object_t obj, vm_prot_t prot, vm_page_t ma[],
>     int total_ma_len, vm_ooffset_t start, vm_ooffset_t end, int *ma_len);
>
> I am reverting back to it, dropping the asyncronicity.  The ma_len is
> the place where the actual number of the filled pages is returned, so
> filesystem plus wrapper would decide how many of pages are provided,
> but not more than total_ma_len.
>
> It would fill all continious slots with the valid pages, and if the page
> is read, then typically UFS or ZFS does read-ahead, which provide more
> valid pages past the read one.
>