[Bug 296466] copy_file_range can be very slow on zfs even for small files

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296466

            Bug ID: 296466
           Summary: copy_file_range can be very slow on zfs even for small
                    files
           Product: Base System
           Version: 14.4-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

The following shows /usr/bin/install running very slowly when copying the same
small file 10 times:

$ df /tmp
Filesystem          1K-blocks Used      Avail Capacity  Mounted on
syspool/system/tmp 1579986920 2916 1579984004     0%    /tmp
$ zpool status syspool
  pool: syspool
 state: ONLINE
  scan: scrub repaired 0B in 00:22:36 with 0 errors on Mon Jun 22 12:24:49 2026
config:

        NAME               STATE     READ WRITE CKSUM
        syspool            ONLINE       0     0     0
          mirror-0         ONLINE       0     0     0
            gpt/zsysdisk0  ONLINE       0     0     0
            gpt/zsysdisk1  ONLINE       0     0     0

errors: No known data errors
$ seq 1 10000 | tee /tmp/hh | wc -c
   48894
$ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2'
        1.52 real         0.00 user         0.04 sys
$ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2'
        1.63 real         0.01 user         0.02 sys
$ truss -fc csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2'
syscall                     seconds   calls  errors
sigreturn               0.000210349      10      10
fchmod                  0.000451316      10       0
rename                  0.002175952      10       0
copy_file_range         1.406585436      20       0
getrandom               0.000238228      10       0
minherit                0.000193221      10       0
execve                  0.003492236      10      10
sigsuspend              1.471121187      10      10
umask                   0.000017425       1       0
lseek                   0.000472312      26       0
getpgrp                 0.000018428       1       0
wait4                   0.000570728      21      10
write                   0.000028104       1       0
recvfrom                0.000035205       2       0
poll                    0.000278623       2       0
sendto                  0.000062127       2       0
connect                 0.000048705       2       0
socket                  0.000046869       2       0
getpid                  0.000239035      13       0
getpeername             0.000016303       1       1
setitimer               0.000617037      32       0
fork                    0.002382183      11       0
dup                     0.001035424      56       0
pipe2                   0.000026176       1       0
__getcwd                0.000020123       1       0
sigprocmask             0.001605172      87       0
sigaction               0.000847859      45       0
ioctl                   0.000699790      35       6
dup2                    0.000074504       4       0
getegid                 0.000018361       1       0
geteuid                 0.000036304       2       0
getgid                  0.000019093       1       0
getuid                  0.000017192       1       0
openat                  0.001696058      35       1
getrusage               0.000443794      22       0
__sysctl                0.000128361       6       0
getdirentries           0.001201559      27       0
fcntl                   0.000956225      52       0
fstatat                 0.001112990      50       5
cpuset_getaffinity      0.000222640      11       0
readlink                0.000239186      11      11
sysarch                 0.000200259      11       0
munmap                  0.000825598      34       0
pread                   0.000444490      13       0
close                   0.004672428     202      33
read                    0.004842528      40       0
fstat                   0.002653986     133      56
open                    0.002064127      87      16
sigfastblock            0.000192375      11       0
issetugid               0.000485598      28       0
mprotect                0.001482952      70       0
mmap                    0.005792957     229       0
                      ------------- ------- -------
                        2.923359118    1513     169
$ 

One interesting thing to note is that files containing random data are fast:

$ dd if=/dev/random of=/tmp/hh bs=1 count=48894
48894+0 records in
48894+0 records out
48894 bytes transferred in 0.614548 secs (79561 bytes/sec)
$ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2'
        0.03 real         0.00 user         0.03 sys
$ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2'
        0.03 real         0.03 user         0.02 sys
$ 

I don't remember seeing this problem in FreeBSD 13.  I suspect it is somehow
related to the fact that zfs in FreeBSD 14 defaults to compression=on, which
even affects all old datasets that didn't set it off specifically.

I first noticed this when running freebsd-update install, which took over 10
minutes to update 8850 files from 14.4-RELEASE-p5 to 14.4-RELEASE-p6.

-- 
You are receiving this mail because:
You are the assignee for the bug.