Why do some folders backup very slowly while all others are fast at the same time?

Check out the following iotop output. Note that there are 4 rdiff-backups g=
oing at the same time, from the same source to the same destination. 3 of t=
he backups are going at 120-200 MB/s, but one of them is only going at 9 MB=
/s. We have observed this behavior multiple times, and always with the same=
 source folder. The  /zpool0/store51b/db_rsyncs/site816 folder is large com=
pared to the others (around 2TB), but the size of the folder should not imp=
act the transfer speed should it?

Total DISK READ :    1096.30 M/s | Total DISK WRITE :     458.39 M/s
Actual DISK READ:     157.71 M/s | Actual DISK WRITE:      20.10 M/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
3289403 be/4 root      128.61 M/s  183.57 M/s  0.00 % 38.48 % python3.6 -s =
/usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force /zpool0/sto=
re51b/db_rsyncs/site189 /remote_rdiffs.nfs/site189
3232252 be/4 root      163.58 M/s   41.02 M/s  0.00 % 18.51 % python3.6 -s =
/usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force /zpool0/sto=
re51b/db_rsyncs/site186 /remote_rdiffs.nfs/site186
348290 be/4 root        9.04 M/s    3.67 M/s  0.00 %  1.47 % python3.6 -s /=
usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force /zpool0/stor=
e51b/db_rsyncs/site816 /remote_rdiffs.nfs/site816
3056376 be/4 root      191.39 M/s  150.88 M/s  0.00 %  0.00 % python3.6 -s =
/usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force /zpool0/sto=
re51b/db_rsyncs/site179 /remote_rdiffs.nfs/site179

-Eric


Disclaimer : This email and any files transmitted with it are confidential =
and intended solely for intended recipients. If you are not the named addre=
ssee you should not disseminate, distribute, copy or alter this email. Any =
views or opinions presented in this email are solely those of the author an=
d might not represent those of Physician Select Management. Warning: Althou=
gh Physician Select Management has taken reasonable precautions to ensure n=
o viruses are present in this email, the company cannot accept responsibili=
ty for any loss or damage arising from the use of this email or attachments=
.