Re: Speed and regression

On 2024-09-28 11:54, Patrik Dufresne wrote:
> Hello Duffer,
> 
> My first guess is it's running regression. It's a long process and 
> almost
> all the files are reverified. Could you confirm which version you are 
> using
> ?
> 
> Le sam. 28 sept. 2024, à 04 h 27, Duffer <[email protected]> a écrit 
> :
> 
>> On 2024-09-27 22:00, Duffer wrote:
>> > On 2024-09-27 05:29, Eric Lavarde wrote:
>> >> Hi,
>> >>
>> >> On 27/09/2024 07:09, Duffer wrote:
>> >>> On 2024-09-25 22:30, Duffer wrote:
>> >>>> Hi.
>> >>>>
>> >>>> I'm using rdiff-backup 2.2.6 for a large amount of data. All traffic
>> >>>> is encrypted via stunnel.
>> >>>>
>> >>>> I have two concerns:
>> >>>>
>> >>>> 1. Speed
>> >>>>
>> >>>> With SSH, the speed I've seen is 25-50 Mbps, probably averaging
>> >>>> 30-40 Mbps. iperf3 gets around 500 Mbps between my local system
>> >>>> (destination) and the remote system (source). The SSH speed seems
>> >>>> excessively slow to me - maybe 6-8% of the measured bandwidth.
>> >>>>
>> >>>> I switched to mapping the data source via NFS. Now I'm seeing full
>> >>>> speed for the transfers.
>> >>>>
>> >>>> Why is SSH so incredibly slow? Is that expected? I'm okay using NFS
>> >>>> but I liked the convenience of SSH.
>> >>
>> >> Well, I can only guess that it's the pickling and unpickling necessary
>> >> over the network. I'm working on improving performance but it takes
>> >> time. I first need to refactor and simplify the code. It could also be
>> >> worsened due to a weak CPU on one end or the other. Many small files
>> >> also increase a lot the overhead.
>> >
>> > Good to know. I think SSH has a lot of overhead, anyway. It's usually
>> > far from the fastest protocol. There's at least one project attempting
>> > to improve performance: https://www.psc.edu/hpn-ssh-home/
>> >
>> >>>> 2. Regression
>> >>>>
>> >>>> As noted, my backup was very slow so I stopped it (CTRL-C). The next
>> >>>> time I ran the backup, it went into "regress" mode, so I lost all
>> >>>> the progress. Is there no way to avoid this and resume somehow? I
>> >>>> couldn't find anything in the FAQ.
>> >>
>> >> No, but there is already an issue for this. Again, takes time...
>> >
>> > Thanks. Yes, it's definitely something you want to get right before
>> > deploying it.
>> >
>> >>> The regress didn't take long at all but it's now been running over 10
>> >>> hours and I've seen little progress. Disk usage hasn't increased at
>> >>> all. Is this expected? I thought after the regress it would start
>> >>> downloading, like it did before. I can't tell what it's doing but I
>> >>> seem to see it reading a lot from the disk. I used "-v 3" but maybe I
>> >>> should have opted for more output.
>> >>
>> >> -v3 is the default value, I generally use -v5 to see progress (you can
>> >> configure differently logfile and console output BTW).
>> >> Difficult to say what's happening from here, but did you check if
>> >> files are appearing in your backup repo, especially under
>> >> rdiff-backup-data?
>> >> If everything breaks, assuming Linux, you can try to guess what it's
>> >> doing using strace.
>> >>
>> >> KR, Eric
>> >
>> > I'm seeing continuous creation of files like rdiff-backup.tmp.217452 in
>> > the backup path. There are also lots of files under rdiff-backup-data,
>> > including two one file and one directory from today.
>> >
>> > It's been running 18 hours, so I hate to break it again but I really
>> > wish I could figure out what it's doing. One of the backup directories
>> > (where I see the tmp files) keeps growing and shrinking.
>> 
>> It's been running almost 24 hours now. I see lots of activity, 
>> accessing
>> files in the actual data, .diff.gz files, and .tmp files. But the disk
>> usage still hasn't changed. Is it possible it's re-verifying every
>> single file? I wish I could figure out what it is doing. This seems
>> completely different from the previous run that I stopped.
>> 
>> 

Hi. As noted in my first post, I am using rdiff-backup version 2.2.6.

The output from rdiff-backup says that it started regressing and then 
went into the increment operation:

   WARNING: Previous backup seems to have failed, regressing destination 
now
   NOTE: Regressing to date/time [date/time]
   NOTE: Starting increment operation from source path /src to 
destination path /dst

Does the increment operation *follow* regression or is it part of 
regression? I assumed the former.