Re: Fun with RDMA and NFS

Moving this to arch@.

On Sat, Aug 08, 2026 at 05:03:56PM -0700, Rick Macklem wrote:
> On Sat, Aug 8, 2026 at 2:53 PM Konstantin Belousov <[email protected]> wrote:
> >
> > On Sat, Aug 08, 2026 at 01:15:00PM -0700, Rick Macklem wrote:
> > > Hi,
> > >
> > > As you all no doubt remember, in around 1986, Kirk proposed
> > > something like this:
> > > #ifdef notdef
> > >     /* Not Yet.. */
> > >     if (uiop->uio_iov->iov_op != NULL)
> > >           (*(uiop->uio_iov->iov_op))(mbufcp, uiocp, xfer);
> > >     else
> > > #endif
> > > The idea was that iov_op would "copy" data without
> > > actually doing a memory->memory copying. It never
> > > happened, afaik.
> > >
> > > Well, after only 40years, it looks like NFS might be
> > > able to do that.
> > >
> > > For example, with RDMA a Read will look something like:
> > > - NFS VOP_READ() allocates a buffer cache block and
> > >   passes the pages (in b_pages[]) to the RPC code.
> > >   - The RPC code will map the pages into bus dma space
> > >     and make the RPC call (referring to the memory region
> > >     for these pages). (What ofed calls FRWR.)
> > > On the NFS server...
> > > - The server will allocate pages for the read reply data and
> > >   map those into bus address space.
> > >   It will make a VOP_READ() call to copy the data into these
> > >   pages.
> > >   The RPC code will get the NIC to copy the data from the pages
> > >   to the pages on the NFS client (the buffer cache block in the client
> > >   using that FRWR stuff) and then send a small RPC reply noting
> > >   the Read has completed.
> > > A write would be similar, except the data is copied into the NFS
> > > server's pages before the VOP_WRITE().
> > >
> > > So, at this point, there is still a memory->memory copy done
> > > in the exported file system's VOP_READ()/VOP_WRITE().
> > >
> > > I think that it should be possible to implement a couple of
> > > new VOP_xxx() calls to avoid this memory->memory copy.
> > > VOP_PAGEIO() - Would return an array of pages with the
> > >     read data in them or where the data can be written into
> > >     them (it could just return a "struct iovec iov[]"
> > >     with the vm_page_t for each page in the iov_base entries).
> > >     Presumably the pages would be buffer cache or ARC or ??
> > >     for the server file system.
> > > VOP_DONE_PAGEIO() - Would be called once the transfer
> > >     is done to release the pages.
> > >
> > > Does this sound feasible?
> > Yes, I think it is reasonable.
> >
> > Perhaps the following signature would be ok:
> > int VOP_PAGEIO(struct vnode *vp, foff_t offset, size_t nbytes,
> >     enum uio_rw dir, vm_page_t ma[], int max_ma_len, int *ma_len);
> > The VOP would return the ma array filled with the busied pages, which
> > provide the file content from trunc_page(offset) to round_page(offset+nbytes).
> >
> > The twist there is that ma_len returns the number of pages that are
> > actually filled into the ma array. For instance, for UFS, it is not
> > feasible to fill more than a single buffer in time.
> >
> > From the first look, this is quite easy for UFS and tmpfs.
> > If you want I can write the prototype.
> Sounds fine to me. No rush, it'll be a while before I get through
> initial testing and, since that is the server side, I won't be looking
> at it until after that.
> 
> >
> > >
> > > On the NFS client end, it should also be possible for O_DIRECT
> > > to have the process's buffer mapped in, so that there is no
> > > need for a buffer cache block. (This is more overhead than
> > > the FRWR that can be used for kernel pages, but still might
> > > be worth the effort.)
> > But then we loose the cache coherency.
> > Otherwise yes, vm_fault_quick_hold_pages() would provide the ma array for
> > the user io buffer.  The easiest example is probably kern/kern_physio.c.
> I'm not worried about O_DIRECT, at least for now. It currently works
> through the buffer cache by default, and that should work the same as
> without O_DIRECT (except when the read and write RPCs are done).
> (To be honest, my current prototype code just refuses to allow
> RDMA mounts when newnfs_directio_enable (the sysctl variable) when
> it is set. It is 0 by default.

I started looking at implementing this stuff, and realized that really
it is not needed in the proposed form.  I am not saying that some helpers
should be written, but I currently think that the proposed form is not
useful.

So lets state the desired operation:
given the vnode, and the array of pointers to vm_page, the KPI should
fill the array with actual pointers to the pages owned by the vnode,
with the content.  That is, the pages have valid content for read, or
could be written to, in this case what is written to pages become the
file content.  Pages should be available until some explicit operation
is done freeing them.

First, I do not see why this would need to be a VOP.  In fact, there is
nothing that requires that the operation is done on vnode, and not on
a vm_object.  VM subsystem already handles similar operations, creating
and filling the pages for fault, or preparing the valid pages for paging
out.

Second, I believe that we should specify the KPI somewhat modern by 
making it async.  After all, the supposed usage of it is RDMA, where
the main operation is async by its nature, so it might be a useful
option to not block the caller.

Right now I wrote the following KPI definition:

diff --git a/sys/vm/vm_object.h b/sys/vm/vm_object.h
index 5a3a78b5b5a1..6a13989117dd 100644
--- a/sys/vm/vm_object.h
+++ b/sys/vm/vm_object.h
@@ -391,6 +391,20 @@ void vm_object_unwire(vm_object_t object, vm_ooffset_t offset,
     vm_size_t length, uint8_t queue);
 struct vnode *vm_object_vnode(vm_object_t object);
 bool vm_object_is_active(vm_object_t obj);
+
+#define	VM_OBJECT_PGREQ_MAXPG	32
+struct vm_object_pgreq {
+	vm_object_t obj;
+	vm_prot_t prot;
+	vm_page_t ma[VM_OBJECT_PGREQ_MAXPG];
+	vm_ooffset_t start;
+	vm_ooffset_t end;
+	int ma_len;
+	void *priv;
+	void (*comp)(int error, struct vm_object_pgreq *req);
+};
+int vm_object_get_pages_async(struct vm_object_pgreq *req);
+
 #endif				/* _KERNEL */
 
 #endif				/* _VM_OBJECT_ */

The function is called with the struct vm_object_pgreq which encodes the
request, and in future will carry an auxillary data needed to execute
the request.  On completion, successful or not, the req->comp method is
called.  The priv pointer is available for caller to stuff its own data.

Does it sound ok?