Re: Fun with RDMA and NFS

On Sat, Aug 08, 2026 at 01:15:00PM -0700, Rick Macklem wrote:
> Hi,
> 
> As you all no doubt remember, in around 1986, Kirk proposed
> something like this:
> #ifdef notdef
>     /* Not Yet.. */
>     if (uiop->uio_iov->iov_op != NULL)
>           (*(uiop->uio_iov->iov_op))(mbufcp, uiocp, xfer);
>     else
> #endif
> The idea was that iov_op would "copy" data without
> actually doing a memory->memory copying. It never
> happened, afaik.
> 
> Well, after only 40years, it looks like NFS might be
> able to do that.
> 
> For example, with RDMA a Read will look something like:
> - NFS VOP_READ() allocates a buffer cache block and
>   passes the pages (in b_pages[]) to the RPC code.
>   - The RPC code will map the pages into bus dma space
>     and make the RPC call (referring to the memory region
>     for these pages). (What ofed calls FRWR.)
> On the NFS server...
> - The server will allocate pages for the read reply data and
>   map those into bus address space.
>   It will make a VOP_READ() call to copy the data into these
>   pages.
>   The RPC code will get the NIC to copy the data from the pages
>   to the pages on the NFS client (the buffer cache block in the client
>   using that FRWR stuff) and then send a small RPC reply noting
>   the Read has completed.
> A write would be similar, except the data is copied into the NFS
> server's pages before the VOP_WRITE().
> 
> So, at this point, there is still a memory->memory copy done
> in the exported file system's VOP_READ()/VOP_WRITE().
> 
> I think that it should be possible to implement a couple of
> new VOP_xxx() calls to avoid this memory->memory copy.
> VOP_PAGEIO() - Would return an array of pages with the
>     read data in them or where the data can be written into
>     them (it could just return a "struct iovec iov[]"
>     with the vm_page_t for each page in the iov_base entries).
>     Presumably the pages would be buffer cache or ARC or ??
>     for the server file system.
> VOP_DONE_PAGEIO() - Would be called once the transfer
>     is done to release the pages.
> 
> Does this sound feasible?
Yes, I think it is reasonable.

Perhaps the following signature would be ok:
int VOP_PAGEIO(struct vnode *vp, foff_t offset, size_t nbytes,
    enum uio_rw dir, vm_page_t ma[], int max_ma_len, int *ma_len);
The VOP would return the ma array filled with the busied pages, which
provide the file content from trunc_page(offset) to round_page(offset+nbytes).

The twist there is that ma_len returns the number of pages that are
actually filled into the ma array. For instance, for UFS, it is not
feasible to fill more than a single buffer in time.

From the first look, this is quite easy for UFS and tmpfs.
If you want I can write the prototype.

> 
> On the NFS client end, it should also be possible for O_DIRECT
> to have the process's buffer mapped in, so that there is no
> need for a buffer cache block. (This is more overhead than
> the FRWR that can be used for kernel pages, but still might
> be worth the effort.)
But then we loose the cache coherency.
Otherwise yes, vm_fault_quick_hold_pages() would provide the ma array for
the user io buffer.  The easiest example is probably kern/kern_physio.c.

> 
> So, after years of procrastinating on this, I finally got prodded
> to do this, thanks to Vinicius's NFS-over-RDMA server work.
> 
> And, thanks to the Netperf folk, I now have a way of testing
> NFS-over-RDMA code. ([email protected], bz@, pho@
> and others)
> 
> Things that I thought others might be able to provide help with are:
> - Creation of the above VOP_xxx() calls { call them whatever
>   you like } and implementation of them for UFS, ZFS, etc.
> 
> - If you have servers with RDMA capable hardware, you could
>   "ibv_devinfo -v" and email me what it dumps out. (If it doesn't
>   find any configured IB devices, it might just need a driver loaded.
>   For example, Mellanox requires mlx5ib to be loaded. Not so sure
>   about Chelsio or Intel?)
>   This info is useful to me, since it tells me what capabilities I can
>   count on from the NIC, such as how many scatter/gather entries
>   it supports.
> 
> Vinicius has done the server side of NFS-over-RDMA:
> https://github.com/viniciusferrao/freebsd-src/pull/1
> and I am working on client side code.
> - Testing. Although I haven't talked to Vinicius yet, I hope he can
>   set up a fork/branch of freebsd-src that has what he thinks others
>   should use for testing and keep that branch relatively up-to-date
>   with FreeBSD's main.
>   Once I get the NFS client code I am now testing working ok, I
>   plan on committing it to main.
>   --> Then Vinicius's fork/branch will pick it up and provide a place
>         where others can download sources for testing.
> 
> Vinicius's code was created with AI usage, so it will be months
> before I know if it can be pulled into main. If not, maybe it can
> become a module in ports or ??
> (But please, please do not make this email thread an AI discussion;-)
> 
> Just thought some of you might find this interesting and wanted
> to thank those that have already provided assistance, rick
> ps: Please let me know if you get working on the new VOP_xxx()
>       calls.