Re: Speed and regression

On 2024-09-27 22:00, Duffer wrote:
> On 2024-09-27 05:29, Eric Lavarde wrote:
>> Hi,
>> 
>> On 27/09/2024 07:09, Duffer wrote:
>>> On 2024-09-25 22:30, Duffer wrote:
>>>> Hi.
>>>> 
>>>> I'm using rdiff-backup 2.2.6 for a large amount of data. All traffic 
>>>> is encrypted via stunnel.
>>>> 
>>>> I have two concerns:
>>>> 
>>>> 1. Speed
>>>> 
>>>> With SSH, the speed I've seen is 25-50 Mbps, probably averaging 
>>>> 30-40 Mbps. iperf3 gets around 500 Mbps between my local system 
>>>> (destination) and the remote system (source). The SSH speed seems 
>>>> excessively slow to me - maybe 6-8% of the measured bandwidth.
>>>> 
>>>> I switched to mapping the data source via NFS. Now I'm seeing full 
>>>> speed for the transfers.
>>>> 
>>>> Why is SSH so incredibly slow? Is that expected? I'm okay using NFS 
>>>> but I liked the convenience of SSH.
>> 
>> Well, I can only guess that it's the pickling and unpickling necessary 
>> over the network. I'm working on improving performance but it takes 
>> time. I first need to refactor and simplify the code. It could also be 
>> worsened due to a weak CPU on one end or the other. Many small files 
>> also increase a lot the overhead.
> 
> Good to know. I think SSH has a lot of overhead, anyway. It's usually 
> far from the fastest protocol. There's at least one project attempting 
> to improve performance: https://www.psc.edu/hpn-ssh-home/
> 
>>>> 2. Regression
>>>> 
>>>> As noted, my backup was very slow so I stopped it (CTRL-C). The next 
>>>> time I ran the backup, it went into "regress" mode, so I lost all 
>>>> the progress. Is there no way to avoid this and resume somehow? I 
>>>> couldn't find anything in the FAQ.
>> 
>> No, but there is already an issue for this. Again, takes time...
> 
> Thanks. Yes, it's definitely something you want to get right before 
> deploying it.
> 
>>> The regress didn't take long at all but it's now been running over 10 
>>> hours and I've seen little progress. Disk usage hasn't increased at 
>>> all. Is this expected? I thought after the regress it would start 
>>> downloading, like it did before. I can't tell what it's doing but I 
>>> seem to see it reading a lot from the disk. I used "-v 3" but maybe I 
>>> should have opted for more output.
>> 
>> -v3 is the default value, I generally use -v5 to see progress (you can 
>> configure differently logfile and console output BTW).
>> Difficult to say what's happening from here, but did you check if 
>> files are appearing in your backup repo, especially under 
>> rdiff-backup-data?
>> If everything breaks, assuming Linux, you can try to guess what it's 
>> doing using strace.
>> 
>> KR, Eric
> 
> I'm seeing continuous creation of files like rdiff-backup.tmp.217452 in 
> the backup path. There are also lots of files under rdiff-backup-data, 
> including two one file and one directory from today.
> 
> It's been running 18 hours, so I hate to break it again but I really 
> wish I could figure out what it's doing. One of the backup directories 
> (where I see the tmp files) keeps growing and shrinking.

It's been running almost 24 hours now. I see lots of activity, accessing 
files in the actual data, .diff.gz files, and .tmp files. But the disk 
usage still hasn't changed. Is it possible it's re-verifying every 
single file? I wish I could figure out what it is doing. This seems 
completely different from the previous run that I stopped.