Re: Fun with RDMA and NFS

On Mon, Aug 10, 2026 at 1:13 PM Konstantin Belousov <[email protected]> wrote:
>
> On Mon, Aug 10, 2026 at 07:27:55AM -0700, Rick Macklem wrote:
> > On Mon, Aug 10, 2026 at 4:28 AM Rick Macklem <[email protected]> wrote:
> > >
> > > On Mon, Aug 10, 2026 at 4:09 AM Rick Macklem <[email protected]> wrote:
> > > >
> > > > On Sun, Aug 9, 2026 at 11:56 PM Konstantin Belousov <[email protected]> wrote:
> > > > >
> > > > > On Sun, Aug 09, 2026 at 06:18:10PM -0700, Rick Macklem wrote:
> > > > > > On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
> > > > > > >
> > > > > > > On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > > > > > > > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > > > > > > > >
> > > > > > > > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > > > > > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > > > > > > > does an assortment of things that are file system specific to get the file's
> > > > > > > > > > data and then it copies that data to the iovec passed in as an argument.
> > > > > > > > > >
> > > > > > > > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > > > > > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > > > > > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > > > > > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > > > > > > > >
> > > > > > > > > > I don't understand how this "wrapper" would do that?
> > > > > > > > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > > > > > > > reads the data in to them? I'd be concerned that is less efficient
> > > > > > > > > > that the heavily
> > > > > > > > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > > > > > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > > > > > > > time and only
> > > > > > > > > > the fs knows how big that is.
> > > > > > > > >
> > > > > > > > > The wrapper would do the following:
> > > > > > > > > - look up the pages from the specified range in the vnode v_object page
> > > > > > > > >   queue
> > > > > > > > > - if the page is there and is valid, it is busied and recorded into the
> > > > > > > > >   ma[] array.
> > > > > > > > > - if the page is not in the queue, a fresh page is allocated and the
> > > > > > > > >   vm_pager_get_page() request is performed to obtain the content.
> > > > > > > > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > > > > > > > >   is aware of fs-specific magic.
> > > > > > > > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > > > > > > > does to fill in the pages, then that should be fine.
> > > > > > >
> > > > > > > For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> > > > > > > creates the buffer that contains the requested page, and does bread()
> > > > > > > on it.  This implictly validates the page.
> > > > > > > There is no uiomove() call, not even UIO_NOCOPY.
> > > > > > >
> > > > > > > ZFS performs reads as needed.
> > > > > > What about the case where you are writing a page that already has file
> > > > > > data?
> > > > > > What I'm thinking is, that if you are writing the entire page, you don't
> > > > > > want to read the data into the page before doing so.
> > > > >
> > > > > Yes, I considered it, and should have write explicitly about the case.
> > > > > It is is indeed fine in case the write was complete.
> > > > >
> > > > > But suppose that the write was short.  If we provide an invalid page for
> > > > > io, then it become a combination of the valid bytes (not blocks) written
> > > > > by remote, and some invalid bytes of the undefined content.  I do not
> > > > > see a way to recover from this situation easily.
> > > > If it is a VOP_xxx() call and the argument is an array of a "struct" that
> > > > has offset and length fields, then the VOP_xxx() call would know if
> > > > it is a full page. If not, it could read the page in before returning
> > > > it. (I said "struct iovec" earlier, but it doesn't have an offset.)
> > > > --> So, NFS would fill in the offset, length fields and the VOP_xxx()
> > > >       call would add the vm_page_t's (or physical addresses or
> > > >       whatever is most convenient).
> > > Oops, I got this wrong.
> > >
> > > The offset, length should be in the array of structures in the reply,
> > > but the file offset and length should be separate arguments.
> > > (I realized that the "buffer" in the file system might not be page aligned.)
> > > --> The restriction is that the first page can start part way through the
> > >       page, going to the end of the page and the last page must start at
> > >       the beginning of the page, but can end part way through the page.
> > >       The pages in between must be full pages. (Basically the same as
> > >        an M_EXTPG mbuf.)
> > >
> > > rick
> > >
> > > >       I assume this is for arches that support PMAP_HAS_DMAP.
> > > >
> > > > For the NFS server, the first page might have a non-zero offset
> > > > (and a length from there to the end of the page) and the last one
> > > > might have a length less than PAGE_SIZE. The rest are full pages,
> > > > since NFS Writes are a single byte aligned data blob (called a chunk
> > > > by RFC8166).
> > > > --> The idea is VOP_xxx() sets up the "pages" (or buffer of pages,
> > > >      if that is clearer) so that the rdma can be done into it after the
> > > >      call.
> > > >      An rdma Read from the client can start part way through a
> > > >      page and end part way through a page. (Mellanox actually
> > > >      allows incomplete pages in the middle, but the other NICs
> > > >      don't do that and I don't think it is needed.)
> > > >
> > > > I'm not giving up easily on this VOP_xxx() thing.;-)
> > Maybe something like...
> >
> > struct page_io {
> >       vm_page_t    page;
> >       uint32_t         offset;  /* Offset in page where data starts. */
> >       uint32_t         len;      /* Length of data in page. */
> >       void               *bufref; /* Struct buf * or whatever, for
> > VOP_END_PAGEIO() */
> > };
> >
> > VOP_START_PAGE_IO(struct vnode *vp, off_t file_offset,
> >      size_t *total_length, enum uio_rw rw, struct page_io *pg[]);
> > - where the caller would provide a "pg" large enough,
> >   doing something like "howmany(length + 2 * PAGE_SIZE - 1, PAGE_SIZE)".
> > - "*total_length" can be returned smaller than request for UIO_READ and then
> >   not all "pg" entries will be vald.
> >
> > VOP_END_PAGE_IO(struct vnode *vp, struct page_io *pg[]);
> > - Do release the resources wired down by VOP_START_PAGEIO().
> >   (Could also have the file_offset and total_length, if those might be
> >    useful?)
>
> I still do not think this needs to be VOPs.  VM can iomplement this using
> existing operations which eventually result in the call to VOP_GETPAGES().
> Also the object size is enough (and there is a pointer to the vnode)
> which would give the correct fill of the page at the EOF.
>
> What I described as the issue that makes me prefer to validate even the
> completely covered pages for the write case, is that the rdma read might
> be short.  But this is ortogonal to the question of whether the interface
> should live as VOP or as a VM helper.
>
Ok, since you've volunteered to implement it.

I am working with Vinicius to set up an unofficial port with his NFS
server work in it.  Hopefully, that will allow you guys to work on it.
I'll email to freebsd-current@ when the port is available.

rick