Re: Fun with RDMA and NFS

On Sun, Aug 09, 2026 at 06:18:10PM -0700, Rick Macklem wrote:
> On Sun, Aug 9, 2026 at 3:50 PM Konstantin Belousov <[email protected]> wrote:
> >
> > On Sun, Aug 09, 2026 at 03:30:59PM -0700, Rick Macklem wrote:
> > > On Sun, Aug 9, 2026 at 3:14 PM Konstantin Belousov <[email protected]> wrote:
> > > >
> > > > On Sun, Aug 09, 2026 at 01:39:46PM -0700, Rick Macklem wrote:
> > > > > I'll admit I don't understand. Right now, the NFS does VOP_READ(), which
> > > > > does an assortment of things that are file system specific to get the file's
> > > > > data and then it copies that data to the iovec passed in as an argument.
> > > > >
> > > > > I still want to do exactly the same stuff, except get the pages (or a kernel
> > > > > virtual address I can turn into a page list via the PMAP_HAS_DMAP stuff)
> > > > > so that I can bus dma map the page(s) instead of copying data from them
> > > > > to other page(s) the NFS server allocates above the VOP_READ().
> > > > >
> > > > > I don't understand how this "wrapper" would do that?
> > > > > (Are you thinking mmap'd file where the wrapper touches the pages and the fs
> > > > > reads the data in to them? I'd be concerned that is less efficient
> > > > > that the heavily
> > > > > exercised code path VOP_READ() uses, reading blocks into buffers.)
> > > > > I'd also like it to do one buffer/block (ZFS calls it recordsize) at a
> > > > > time and only
> > > > > the fs knows how big that is.
> > > >
> > > > The wrapper would do the following:
> > > > - look up the pages from the specified range in the vnode v_object page
> > > >   queue
> > > > - if the page is there and is valid, it is busied and recorded into the
> > > >   ma[] array.
> > > > - if the page is not in the queue, a fresh page is allocated and the
> > > >   vm_pager_get_page() request is performed to obtain the content.
> > > >   For typical UFS or ZFS vnodes, it translates into VOP_GETPAGES() that
> > > >   is aware of fs-specific magic.
> > > Ok, if you believe that VOP_GETPAGES() will be as efficient as what VOP_READ()
> > > does to fill in the pages, then that should be fine.
> >
> > For UFS, ffs_getpages() is vfs_bio_getpages() AKA buffer pager, which
> > creates the buffer that contains the requested page, and does bread()
> > on it.  This implictly validates the page.
> > There is no uiomove() call, not even UIO_NOCOPY.
> >
> > ZFS performs reads as needed.
> What about the case where you are writing a page that already has file
> data?
> What I'm thinking is, that if you are writing the entire page, you don't
> want to read the data into the page before doing so.

Yes, I considered it, and should have write explicitly about the case.
It is is indeed fine in case the write was complete.

But suppose that the write was short.  If we provide an invalid page for
io, then it become a combination of the valid bytes (not blocks) written
by remote, and some invalid bytes of the undefined content.  I do not
see a way to recover from this situation easily.

There is a similar problem for e.g. UFS when we optimizing the ffs_write()
filling the whole buffer, but copying from userspace faults.  Right now
we drop the buffer, see ca39f23347e1416a28.  I am not sure that this is
an acceptable behavior for NFS.