Re: cross-platform backup tool Unexpectedly Slow on Initial Backup

Hi,

just because I became curious, the numbers are probably not to be compared:

- iperf3 tells me ~170Mbit/s
- I transferred initially my Downloads repository from laptop to server 
(which has roughly as much disk as you have RAM :-P), 2.7GBytes in 
3.45minutes, a mixture of not too big files and quite some small files 
(see the size distribution below).
- this makes it roughly 90-100Mbit/s

As said, nothing comparable, just for comparaison. Even on a local 
backup, the initial backup takes quite a while and becomes better in 
delta mode.

I have the plan to improve performance, without knowing if there is 
really potential, but at this stage, I'm focussing on re-structuring the 
code and understanding it better.

KR, Eric

Neat one liner I found at 
https://superuser.com/questions/565443/generate-distribution-of-file-sizes-from-the-command-prompt
(added as utility at https://github.com/rdiff-backup/rdiff-backup/pull/668)

$ find Downloads -type f -print0 | xargs -0 ls -l | awk 
'{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 
2^i, size[i])}' | sort -n
          0   3
          4   4
          8   3
         16   6
         32   5
        128   4
        256   8
        512  14
       1024  27
       2048  29
       4096  30
       8192  18
      16384  15
      32768   9
      65536  16
     131072  15
     262144  12
     524288   6
    1048576   1
    2097152   3
    4194304   7
    8388608   5
   16777216   2
   33554432   4
   67108864   2
  268435456   1
  536870912   1
1073741824   1

On 07/02/2022 18:39, EricZolf wrote:
> Can you send me your hardware that I do some tests?
> 
> Seriously, no clue. It might be that you have many very small files but even then the drop in performance is impressive...
> Perhaps someone with a bigger estate can be more helpful.
> 
> Eric
> 
> 
> On 7 February 2022 03:20:15 UTC, Eric Robinson <[email protected]> wrote:
>>
>> I have two RHEL 8.5 servers with 48 cores, 1TB RAM, and NVME disks connected over a 25 Gbit network. I confirmed with iperf that I'm betting 20+ Gbits/sec bandwidth. Using rdiff-backup 2.0.5, I am only getting 15-25 MB/sec transfer speed. Why is rdiff-backup so slow? The target server is empty. This is the first backup attempt.
>>
>> -Eric
>>
>>
>>
>>
>> Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
>>
>