Re: Performance Questions

Hello,=20
rdiff-backup has many advantages but speed isn't one of those=2E I'm sure =
we can improve the situation but this takes time=2E=20
Kr, Eric

On 30 May 2025 16:20:29 CEST, Duane Abrames <duane@spaztic=2Enet> wrote:
>(I sent this yesterday, but have not seen it come back to me, nor in the
>online archives=2E  Not sure if there is some additional approval process=
, or
>if I was just too quick to send this after joining=2E  Apologies if it ma=
kes
>a duplicate=2E)
>
>Hello, new to the list and rdiff-backup=2E
>
>First let me describe my situation=2E  I have just started using rdiff-ba=
ckup
>to back up my home server, which contains various music, videos, ebooks,
>comics, and audio books ( 34=2E0 TB in 29,244 folders and 460,277 files)=
=2E
>The source and target servers are both Unraid boxes, so I am running
>rdiff-backup in a Docker container=2E  I had initially set up just one
>container on the target server, and just mounted the source data via NFS=
=2E
>My initial backup finished last night, coming in at about 6=2E5 days=2E T=
here
>seemed to be a lot of latency involved in copying the files; I was watchi=
ng
>files get created, and I watched a directory with 40 or so files of less
>than 10mb each take 5 minutes or more to copy, and this was the 'initial'
>backup, so there would be no comparison, just copying=2E
>I suspected that performance would be better over SSH than NFS, so I adde=
d
>a container to the source server and reconfigured things so that the path=
s
>over ssh would match the former NFS paths=2E  After setting all that up, =
I
>kicked off an increment=2E  That increment has been running for 10 hours =
now,
>and using rdiffweb I can see that it is going through checking the folder=
s,
>because the version times change from 10 hours ago (before checking) to 7
>days ago (after)=2E  It looks like even an increment will take days=2E  I=
 was
>hoping to get at least one a day, but if this is the best I can do, I mig=
ht
>have to fall back to weekly backups, or evaluate another option=2E  I wis=
h I
>had started this increment with additional verbosity so I might be able t=
o
>tell what it's actually doing=2E  I feel like at this speed, it must be d=
oing
>checksums or full comparisons, since I can get the 'metadata' (file time,
>size, owner, etc) for the whole share in about 3 minutes with ls -lR=2E D=
oes
>my experience seem typical, and if not, how should I begin nailing down t=
he
>source of the issue?  I kind of assumed that since there were probably le=
ss
>than 200 new/changed files, the increment would be measured in minutes (o=
r
>at worst hours) and not days=2E
>
>Some system specs, just so everyone knows I'm not trying to run this on a
>Dorito chip:
>Source:  AMD 5600G / 32gb RAM / 7=2E2k spinning SATA disk x 9 (with parit=
y)
>on IBM SAS controller
>Target:   AMD 5500 / 64gb RAM / 7=2E2k spinning SATA disk x6 (without par=
ity)
>on mobo / add-in SATA=2E
>Connectivity between the two is gigabit ethernet using a dumb switch=2E
>Servers are less than 10 feet apart, so cable length might be 20 or 30 fe=
et
>since cables run under the house=2E
>
>I don't have a lot of tools inside the containers to troubleshoot with, b=
ut
>they are being pulled from my personal github container repo, so I can
>always add things and rebuild as needed=2E  The base OS in the container =
is
>ubuntu/latest=2E  I went into this project excited (as excited as one can=
 be
>about backups anyway,) as having increments to go back to, and still havi=
ng
>the filesystem be usable as-is seemed like eating the cake and keeping it
>too (Is the cake a lie?) I am less enthused now, and wondering if I will
>end up with a plain rsync (which is basically what I had a few months ago
>before some hardware deaths) or using a backup product that packages
>backups in its own native format so that any use or restore from the syst=
em
>requires the use of the backup software=2E
>
>Thanks for your time and attention=2E
>-Duane