Re: Fun with RDMA and NFS

On Sun, Aug 9, 2026 at 11:56 PM Konstantin Belousov <[email protected]> wrote:
>
> On Sun, Aug 09, 2026 at 06:18:10PM -0700, Rick Macklem wrote:
> > On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
> > >
> > > On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > > > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > > > >
> > > > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > > > does an assortment of things that are file system specific to get the file's
> > > > > > data and then it copies that data to the iovec passed in as an argument.
> > > > > >
> > > > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > > > >
> > > > > > I don't understand how this "wrapper" would do that?
> > > > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > > > reads the data in to them? I'd be concerned that is less efficient
> > > > > > that the heavily
> > > > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > > > time and only
> > > > > > the fs knows how big that is.
> > > > >
> > > > > The wrapper would do the following:
> > > > > - look up the pages from the specified range in the vnode v_object page
> > > > >   queue
> > > > > - if the page is there and is valid, it is busied and recorded into the
> > > > >   ma[] array.
> > > > > - if the page is not in the queue, a fresh page is allocated and the
> > > > >   vm_pager_get_page() request is performed to obtain the content.
> > > > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > > > >   is aware of fs-specific magic.
> > > > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > > > does to fill in the pages, then that should be fine.
> > >
> > > For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> > > creates the buffer that contains the requested page, and does bread()
> > > on it.  This implictly validates the page.
> > > There is no uiomove() call, not even UIO_NOCOPY.
> > >
> > > ZFS performs reads as needed.
> > What about the case where you are writing a page that already has file
> > data?
> > What I'm thinking is, that if you are writing the entire page, you don't
> > want to read the data into the page before doing so.
>
> Yes, I considered it, and should have write explicitly about the case.
> It is is indeed fine in case the write was complete.
>
> But suppose that the write was short.  If we provide an invalid page for
> io, then it become a combination of the valid bytes (not blocks) written
> by remote, and some invalid bytes of the undefined content.  I do not
> see a way to recover from this situation easily.
If it is a VOP_xxx() call and the argument is an array of a "struct" that
has offset and length fields, then the VOP_xxx() call would know if
it is a full page. If not, it could read the page in before returning
it. (I said "struct iovec" earlier, but it doesn't have an offset.)
--> So, NFS would fill in the offset, length fields and the VOP_xxx()
      call would add the vm_page_t's (or physical addresses or
      whatever is most convenient).
      I assume this is for arches that support PMAP_HAS_DMAP.

For the NFS server, the first page might have a non-zero offset
(and a length from there to the end of the page) and the last one
might have a length less than PAGE_SIZE. The rest are full pages,
since NFS Writes are a single byte aligned data blob (called a chunk
by RFC8166).
--> The idea is VOP_xxx() sets up the "pages" (or buffer of pages,
     if that is clearer) so that the rdma can be done into it after the
     call.
     An rdma Read from the client can start part way through a
     page and end part way through a page. (Mellanox actually
     allows incomplete pages in the middle, but the other NICs
     don't do that and I don't think it is needed.)

I'm not giving up easily on this VOP_xxx() thing.;-)

Have a good week, rick

>
> There is a similar problem for e.g. UFS when we optimizing the ffs_write()
> filling the whole buffer, but copying from userspace faults.  Right now
> we drop the buffer, see ca39f23347e1416a28.  I am not sure that this is
> an acceptable behavior for NFS.
>