Re: I/O read and write *latency*, why is it so much higher on NFSv4.2, compared to P9 and iSCSI? Re: BHYVE on NFSv4.2 filesystem?

On Fri, 5 Jun 2026 at 20:38, Dan Shelton <[email protected]> wrote:
>
> On Tue, 19 May 2026 at 00:39, Bakul Shah <[email protected]> wrote:
> >
> > On May 18, 2026, at 2:26 PM, Dan Shelton <[email protected]> wrote:
> > >
> > > On Fri, 20 Feb 2026 at 01:21, Bakul Shah <[email protected]> wrote:
> > >>
> > >> On Feb 19, 2026, at 11:45 AM, Dan Shelton <[email protected]> wrote:
> > >>>
> > >>> On Wed, 18 Feb 2026 at 22:45, Dan Shelton <[email protected]> wrote:
> > >>>>
> > >>>> Hello,
> > >>>>
> > >>>> Has anyone tried a BHYVE with a disk as file on a NFSv4.2 mount?
> > >>
> > >> Yes. [I tried this on a 15.0-RELEASE-p3 host, nfsv4.2 mounting
> > >> a filesystem from a 15.0-STABLE machine]
> > >
> > > How about the performance? Is it better than iSCSI?
> >
> > I don't know about iSCSI but comparing with p9fs:
> >
> > Test1:
> > dd bs=1m count=4000 > /dev/null < large-file
> >
> > nfsV3:
> > 32.3
> > 46.3
> > 51.3
> >
> > nfsV4:
> > 129.1
> > 59.9
> > 48.8
> >
> > p9fs:
> > 17.7
> > 17.5
> > 17.6
> >
> > Test2:
> > find /usr/src/ > /dev/null
> >
> > nfsV3:
> > 60.0
> > 39.0
> > 30.9
> >
> > nfsV4:
> > 54.0
> > 17.9
> > 35.8
> >
> > p9fs:
> > 6.9
> > 6.5
> > 6.6
> >
> >
> > So slower in all cases. In addition the variability in nfs numbers is concerning!
> >
> > p9fs doesn't cache but nfs does, so anything cached is served much faster.
>
> OUCH! Verify: Higher numbers means longer runtime?
>
> @Rick Macklem Could you please look at this? Why is there such a
> difference - because FreeBSD ndsd lives as a userland process?
>
> My primary concern is I/O read and write *latency*, and why is it so
> much higher on NFSv4.2, compared to P9 and iSCSI?

Rick? Anyone?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd