I/O read and write *latency*, why is it so much higher on NFSv4.2, compared to P9 and iSCSI? Re: BHYVE on NFSv4.2 filesystem?

On Tue, 19 May 2026 at 00:39, Bakul Shah <[email protected]> wrote:
>
> On May 18, 2026, at 2:26 PM, Dan Shelton <[email protected]> wrote:
> >
> > On Fri, 20 Feb 2026 at 01:21, Bakul Shah <[email protected]> wrote:
> >>
> >> On Feb 19, 2026, at 11:45 AM, Dan Shelton <[email protected]> wrote:
> >>>
> >>> On Wed, 18 Feb 2026 at 22:45, Dan Shelton <[email protected]> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> Has anyone tried a BHYVE with a disk as file on a NFSv4.2 mount?
> >>
> >> Yes. [I tried this on a 15.0-RELEASE-p3 host, nfsv4.2 mounting
> >> a filesystem from a 15.0-STABLE machine]
> >
> > How about the performance? Is it better than iSCSI?
>
> I don't know about iSCSI but comparing with p9fs:
>
> Test1:
> dd bs=1m count=4000 > /dev/null < large-file
>
> nfsV3:
> 32.3
> 46.3
> 51.3
>
> nfsV4:
> 129.1
> 59.9
> 48.8
>
> p9fs:
> 17.7
> 17.5
> 17.6
>
> Test2:
> find /usr/src/ > /dev/null
>
> nfsV3:
> 60.0
> 39.0
> 30.9
>
> nfsV4:
> 54.0
> 17.9
> 35.8
>
> p9fs:
> 6.9
> 6.5
> 6.6
>
>
> So slower in all cases. In addition the variability in nfs numbers is concerning!
>
> p9fs doesn't cache but nfs does, so anything cached is served much faster.

OUCH! Verify: Higher numbers means longer runtime?

@Rick Macklem Could you please look at this? Why is there such a
difference - because FreeBSD ndsd lives as a userland process?

My primary concern is I/O read and write *latency*, and why is it so
much higher on NFSv4.2, compared to P9 and iSCSI?

Dan
--
Dan Shelton - Cluster Specialist Win/Lin/Bsd