ZFS recv oddity

--17BJl2IaFT2CmkOS
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I have noticed that an incremental "zfs recv" to a backup pool consistently
stalls for an extended period (roughly 5 minutes) at the start before it
starts copying data.  During this time, there's basically no IO on the pool
(<1 IOPS) and basically no CPU activity (~0.5% increase vs the background).
I can understand the "zfs recv" needing to do some book-keeping when it
starts but I would expect that to involve non-trivial CPU and/or IO.  Once
the "zfs recv" starts actually writing data, everything transfers as
expected.  Does anyone have any suggestions as to what might be going on?

Overall backup approach:
  zfs import -N -d $DEVICE $POOL_ID bupool
  zfs send -RLeI @$OLD pool@$NEW | <<network and memory buffer>> | zfs recv=
 -Fuv bupool

The pool has just under 23k snapshots and just under 100 filesystems.

Reporting from buffer tool shows that it quickly fills to its 1GB limit and
writes about 1.8KiB to the "zfs recv" then sits for 5 minutes before the
"zfs recv" starts reporting "receiving incremental stream...".

I see similar behaviour whether the receiving side is a one or two HDD pool
on an arm64 system or a 6-HDD pool on a amd64 system.

--=20
Peter Jeremy

--17BJl2IaFT2CmkOS
Content-Type: application/pgp-signature; name=signature.asc

-----BEGIN PGP SIGNATURE-----

iQKvBAEBCgCZFiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmpn8/obFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwzXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25z
Lm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIy
N0UxNkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs09C0P/0kWft3xibHIxSsVAx6Y
u0BgP14jOsmw2VX5o45LyBxMTtpLPFlkqcSBOla3CmKHFbIpYSPDNqmDNzcE/MSM
BI0jXEcjUfV+m5ChTFFQgqZ4od72vq3FGpPK32aMGTgvdhngTdlkm4mf7FcrbLMB
WSl9XFIQ8UcWyJik4qk59wpO4/S14FStBm+fy4zntFPDZ0NdngD9WSU2kwnCvGek
FU5m2KirbSEewJZtzucE4SHjS4a62AasYhurqta6omCFKCKlIRGuoQm5Hk7HcYxD
1uWrZvzpFL1X8su9cEs2WfT4z7O2XtytqwFDKoUeMntus5KQBQioz27nbSjsmo9j
Efno/eEjn9CJomOT5DVVchG8iVrTqRE+xHSUUKSDfW3cfuc+x1W7VG0+t3iUKLNB
IgGeIUXzt7vFYDKAWAknPfUR45XCfJyCB9azjnGH0AFF3OiWD2Xw2D8FwdTJC8vv
FAJJoLaSougfrHHWT5WxKHQOQtg8gq8HBCJZYI/2KzfUICbx0XzkbOQkykqP95wd
QVoxmM9Kvuig2i5mRRZebz41eq1dcYc2CRqIjCBH8264Q2JWqCTvJhOlN7NmfJXZ
YR07ChjxS3Q424CiUlMbIGBo1nvvYsPJJT4xQvyn4HZ+CojAnaaMLXq/P+s3l0QO
vysQraGxGimH/37/y+I8KLsx
=to58
-----END PGP SIGNATURE-----

--17BJl2IaFT2CmkOS--