Re: Fun with RDMA and NFS

On Mon, Aug 10, 2026 at 4:28 AM Rick Macklem <[email protected]> wrote:
>
> On Mon, Aug 10, 2026 at 4:09 AM Rick Macklem <[email protected]> wrote:
> >
> > On Sun, Aug 9, 2026 at 11:56 PM Konstantin Belousov <[email protected]> wrote:
> > >
> > > On Sun, Aug 09, 2026 at 06:18:10PM -0700, Rick Macklem wrote:
> > > > On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
> > > > >
> > > > > On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > > > > > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > > > > > >
> > > > > > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > > > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > > > > > does an assortment of things that are file system specific to get the file's
> > > > > > > > data and then it copies that data to the iovec passed in as an argument.
> > > > > > > >
> > > > > > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > > > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > > > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > > > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > > > > > >
> > > > > > > > I don't understand how this "wrapper" would do that?
> > > > > > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > > > > > reads the data in to them? I'd be concerned that is less efficient
> > > > > > > > that the heavily
> > > > > > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > > > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > > > > > time and only
> > > > > > > > the fs knows how big that is.
> > > > > > >
> > > > > > > The wrapper would do the following:
> > > > > > > - look up the pages from the specified range in the vnode v_object page
> > > > > > >   queue
> > > > > > > - if the page is there and is valid, it is busied and recorded into the
> > > > > > >   ma[] array.
> > > > > > > - if the page is not in the queue, a fresh page is allocated and the
> > > > > > >   vm_pager_get_page() request is performed to obtain the content.
> > > > > > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > > > > > >   is aware of fs-specific magic.
> > > > > > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > > > > > does to fill in the pages, then that should be fine.
> > > > >
> > > > > For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> > > > > creates the buffer that contains the requested page, and does bread()
> > > > > on it.  This implictly validates the page.
> > > > > There is no uiomove() call, not even UIO_NOCOPY.
> > > > >
> > > > > ZFS performs reads as needed.
> > > > What about the case where you are writing a page that already has file
> > > > data?
> > > > What I'm thinking is, that if you are writing the entire page, you don't
> > > > want to read the data into the page before doing so.
> > >
> > > Yes, I considered it, and should have write explicitly about the case.
> > > It is is indeed fine in case the write was complete.
> > >
> > > But suppose that the write was short.  If we provide an invalid page for
> > > io, then it become a combination of the valid bytes (not blocks) written
> > > by remote, and some invalid bytes of the undefined content.  I do not
> > > see a way to recover from this situation easily.
> > If it is a VOP_xxx() call and the argument is an array of a "struct" that
> > has offset and length fields, then the VOP_xxx() call would know if
> > it is a full page. If not, it could read the page in before returning
> > it. (I said "struct iovec" earlier, but it doesn't have an offset.)
> > --> So, NFS would fill in the offset, length fields and the VOP_xxx()
> >       call would add the vm_page_t's (or physical addresses or
> >       whatever is most convenient).
> Oops, I got this wrong.
>
> The offset, length should be in the array of structures in the reply,
> but the file offset and length should be separate arguments.
> (I realized that the "buffer" in the file system might not be page aligned.)
> --> The restriction is that the first page can start part way through the
>       page, going to the end of the page and the last page must start at
>       the beginning of the page, but can end part way through the page.
>       The pages in between must be full pages. (Basically the same as
>        an M_EXTPG mbuf.)
>
> rick
>
> >       I assume this is for arches that support PMAP_HAS_DMAP.
> >
> > For the NFS server, the first page might have a non-zero offset
> > (and a length from there to the end of the page) and the last one
> > might have a length less than PAGE_SIZE. The rest are full pages,
> > since NFS Writes are a single byte aligned data blob (called a chunk
> > by RFC8166).
> > --> The idea is VOP_xxx() sets up the "pages" (or buffer of pages,
> >      if that is clearer) so that the rdma can be done into it after the
> >      call.
> >      An rdma Read from the client can start part way through a
> >      page and end part way through a page. (Mellanox actually
> >      allows incomplete pages in the middle, but the other NICs
> >      don't do that and I don't think it is needed.)
> >
> > I'm not giving up easily on this VOP_xxx() thing.;-)
Maybe something like...

struct page_io {
      vm_page_t    page;
      uint32_t         offset;  /* Offset in page where data starts. */
      uint32_t         len;      /* Length of data in page. */
      void               *bufref; /* Struct buf * or whatever, for
VOP_END_PAGEIO() */
};

VOP_START_PAGE_IO(struct vnode *vp, off_t file_offset,
     size_t *total_length, enum uio_rw rw, struct page_io *pg[]);
- where the caller would provide a "pg" large enough,
  doing something like "howmany(length + 2 * PAGE_SIZE - 1, PAGE_SIZE)".
- "*total_length" can be returned smaller than request for UIO_READ and then
  not all "pg" entries will be vald.

VOP_END_PAGE_IO(struct vnode *vp, struct page_io *pg[]);
- Do release the resources wired down by VOP_START_PAGEIO().
  (Could also have the file_offset and total_length, if those might be
   useful?)

rick

> >
> > Have a good week, rick
> >
> > >
> > > There is a similar problem for e.g. UFS when we optimizing the ffs_write()
> > > filling the whole buffer, but copying from userspace faults.  Right now
> > > we drop the buffer, see ca39f23347e1416a28.  I am not sure that this is
> > > an acceptable behavior for NFS.
> > >