Fun with RDMA and NFS

Hi,

As you all no doubt remember, in around 1986, Kirk proposed
something like this:
#ifdef notdef
    /* Not Yet.. */
    if (uiop->uio_iov->iov_op != NULL)
          (*(uiop->uio_iov->iov_op))(mbufcp, uiocp, xfer);
    else
#endif
The idea was that iov_op would "copy" data without
actually doing a memory->memory copying. It never
happened, afaik.

Well, after only 40years, it looks like NFS might be
able to do that.

For example, with RDMA a Read will look something like:
- NFS VOP_READ() allocates a buffer cache block and
  passes the pages (in b_pages[]) to the RPC code.
  - The RPC code will map the pages into bus dma space
    and make the RPC call (referring to the memory region
    for these pages). (What ofed calls FRWR.)
On the NFS server...
- The server will allocate pages for the read reply data and
  map those into bus address space.
  It will make a VOP_READ() call to copy the data into these
  pages.
  The RPC code will get the NIC to copy the data from the pages
  to the pages on the NFS client (the buffer cache block in the client
  using that FRWR stuff) and then send a small RPC reply noting
  the Read has completed.
A write would be similar, except the data is copied into the NFS
server's pages before the VOP_WRITE().

So, at this point, there is still a memory->memory copy done
in the exported file system's VOP_READ()/VOP_WRITE().

I think that it should be possible to implement a couple of
new VOP_xxx() calls to avoid this memory->memory copy.
VOP_PAGEIO() - Would return an array of pages with the
    read data in them or where the data can be written into
    them (it could just return a "struct iovec iov[]"
    with the vm_page_t for each page in the iov_base entries).
    Presumably the pages would be buffer cache or ARC or ??
    for the server file system.
VOP_DONE_PAGEIO() - Would be called once the transfer
    is done to release the pages.

Does this sound feasible?

On the NFS client end, it should also be possible for O_DIRECT
to have the process's buffer mapped in, so that there is no
need for a buffer cache block. (This is more overhead than
the FRWR that can be used for kernel pages, but still might
be worth the effort.)

So, after years of procrastinating on this, I finally got prodded
to do this, thanks to Vinicius's NFS-over-RDMA server work.

And, thanks to the Netperf folk, I now have a way of testing
NFS-over-RDMA code. ([email protected], bz@, pho@
and others)

Things that I thought others might be able to provide help with are:
- Creation of the above VOP_xxx() calls { call them whatever
  you like } and implementation of them for UFS, ZFS, etc.

- If you have servers with RDMA capable hardware, you could
  "ibv_devinfo -v" and email me what it dumps out. (If it doesn't
  find any configured IB devices, it might just need a driver loaded.
  For example, Mellanox requires mlx5ib to be loaded. Not so sure
  about Chelsio or Intel?)
  This info is useful to me, since it tells me what capabilities I can
  count on from the NIC, such as how many scatter/gather entries
  it supports.

Vinicius has done the server side of NFS-over-RDMA:
https://github.com/viniciusferrao/freebsd-src/pull/1
and I am working on client side code.
- Testing. Although I haven't talked to Vinicius yet, I hope he can
  set up a fork/branch of freebsd-src that has what he thinks others
  should use for testing and keep that branch relatively up-to-date
  with FreeBSD's main.
  Once I get the NFS client code I am now testing working ok, I
  plan on committing it to main.
  --> Then Vinicius's fork/branch will pick it up and provide a place
        where others can download sources for testing.

Vinicius's code was created with AI usage, so it will be months
before I know if it can be pulled into main. If not, maybe it can
become a module in ports or ??
(But please, please do not make this email thread an AI discussion;-)

Just thought some of you might find this interesting and wanted
to thank those that have already provided assistance, rick
ps: Please let me know if you get working on the new VOP_xxx()
      calls.