Re: Fun with RDMA and NFS

On Mon, Aug 10, 2026 at 07:27:55AM -0700, Rick Macklem wrote:
> On Mon, Aug 10, 2026 at 4:28 AM Rick Macklem <[email protected]> wrote:
> >
> > On Mon, Aug 10, 2026 at 4:09 AM Rick Macklem <[email protected]> wrote:
> > >
> > > On Sun, Aug 9, 2026 at 11:56 PM Konstantin Belousov <[email protected]> wrote:
> > > >
> > > > On Sun, Aug 09, 2026 at 06:18:10PM -0700, Rick Macklem wrote:
> > > > > On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
> > > > > >
> > > > > > On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > > > > > > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > > > > > > >
> > > > > > > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > > > > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > > > > > > does an assortment of things that are file system specific to get the file's
> > > > > > > > > data and then it copies that data to the iovec passed in as an argument.
> > > > > > > > >
> > > > > > > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > > > > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > > > > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > > > > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > > > > > > >
> > > > > > > > > I don't understand how this "wrapper" would do that?
> > > > > > > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > > > > > > reads the data in to them? I'd be concerned that is less efficient
> > > > > > > > > that the heavily
> > > > > > > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > > > > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > > > > > > time and only
> > > > > > > > > the fs knows how big that is.
> > > > > > > >
> > > > > > > > The wrapper would do the following:
> > > > > > > > - look up the pages from the specified range in the vnode v_object page
> > > > > > > >   queue
> > > > > > > > - if the page is there and is valid, it is busied and recorded into the
> > > > > > > >   ma[] array.
> > > > > > > > - if the page is not in the queue, a fresh page is allocated and the
> > > > > > > >   vm_pager_get_page() request is performed to obtain the content.
> > > > > > > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > > > > > > >   is aware of fs-specific magic.
> > > > > > > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > > > > > > does to fill in the pages, then that should be fine.
> > > > > >
> > > > > > For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> > > > > > creates the buffer that contains the requested page, and does bread()
> > > > > > on it.  This implictly validates the page.
> > > > > > There is no uiomove() call, not even UIO_NOCOPY.
> > > > > >
> > > > > > ZFS performs reads as needed.
> > > > > What about the case where you are writing a page that already has file
> > > > > data?
> > > > > What I'm thinking is, that if you are writing the entire page, you don't
> > > > > want to read the data into the page before doing so.
> > > >
> > > > Yes, I considered it, and should have write explicitly about the case.
> > > > It is is indeed fine in case the write was complete.
> > > >
> > > > But suppose that the write was short.  If we provide an invalid page for
> > > > io, then it become a combination of the valid bytes (not blocks) written
> > > > by remote, and some invalid bytes of the undefined content.  I do not
> > > > see a way to recover from this situation easily.
> > > If it is a VOP_xxx() call and the argument is an array of a "struct" that
> > > has offset and length fields, then the VOP_xxx() call would know if
> > > it is a full page. If not, it could read the page in before returning
> > > it. (I said "struct iovec" earlier, but it doesn't have an offset.)
> > > --> So, NFS would fill in the offset, length fields and the VOP_xxx()
> > >       call would add the vm_page_t's (or physical addresses or
> > >       whatever is most convenient).
> > Oops, I got this wrong.
> >
> > The offset, length should be in the array of structures in the reply,
> > but the file offset and length should be separate arguments.
> > (I realized that the "buffer" in the file system might not be page aligned.)
> > --> The restriction is that the first page can start part way through the
> >       page, going to the end of the page and the last page must start at
> >       the beginning of the page, but can end part way through the page.
> >       The pages in between must be full pages. (Basically the same as
> >        an M_EXTPG mbuf.)
> >
> > rick
> >
> > >       I assume this is for arches that support PMAP_HAS_DMAP.
> > >
> > > For the NFS server, the first page might have a non-zero offset
> > > (and a length from there to the end of the page) and the last one
> > > might have a length less than PAGE_SIZE. The rest are full pages,
> > > since NFS Writes are a single byte aligned data blob (called a chunk
> > > by RFC8166).
> > > --> The idea is VOP_xxx() sets up the "pages" (or buffer of pages,
> > >      if that is clearer) so that the rdma can be done into it after the
> > >      call.
> > >      An rdma Read from the client can start part way through a
> > >      page and end part way through a page. (Mellanox actually
> > >      allows incomplete pages in the middle, but the other NICs
> > >      don't do that and I don't think it is needed.)
> > >
> > > I'm not giving up easily on this VOP_xxx() thing.;-)
> Maybe something like...
> 
> struct page_io {
>       vm_page_t    page;
>       uint32_t         offset;  /* Offset in page where data starts. */
>       uint32_t         len;      /* Length of data in page. */
>       void               *bufref; /* Struct buf * or whatever, for
> VOP_END_PAGEIO() */
> };
> 
> VOP_START_PAGE_IO(struct vnode *vp, off_t file_offset,
>      size_t *total_length, enum uio_rw rw, struct page_io *pg[]);
> - where the caller would provide a "pg" large enough,
>   doing something like "howmany(length + 2 * PAGE_SIZE - 1, PAGE_SIZE)".
> - "*total_length" can be returned smaller than request for UIO_READ and then
>   not all "pg" entries will be vald.
> 
> VOP_END_PAGE_IO(struct vnode *vp, struct page_io *pg[]);
> - Do release the resources wired down by VOP_START_PAGEIO().
>   (Could also have the file_offset and total_length, if those might be
>    useful?)

I still do not think this needs to be VOPs.  VM can iomplement this using
existing operations which eventually result in the call to VOP_GETPAGES().
Also the object size is enough (and there is a pointer to the vnode)
which would give the correct fill of the page at the EOF.

What I described as the issue that makes me prefer to validate even the
completely covered pages for the write case, is that the rdma read might
be short.  But this is ortogonal to the question of whether the interface
should live as VOP or as a VM helper.