Re: too many lstat() syscalls, therefore too many IOPS

Ups, somehow I sent my e-mail twice... But at least now I know about 
ionice; never needed it, but it's even installed by default (it's part 
of util-linux, together with mount, fdisk, etc...).

I'm now browsing through the other utilities of this package, full of 
small jewels. You never stop learning...

Thanks, Eric

On 12/05/2021 23:59, griffin tucker wrote:
> there is a tool called ionice that comes with the util-linux package,
> which should help other read/write processes if you are multitasking.
> 
> ionice shouldn't slow down rdiff-backup if there are no other io
> processes going on.
> 
> On Thu, 13 May 2021 at 03:08, Eric L. Zolf <[email protected]> wrote:
>>
>> Hi,
>>
>> first, I don't see anything surprising in what you describe, so all
>> normal AFAICJ.
>>
>> Second, rdiff-backup needs to check each source file/directory and each
>> target, compare them and then copy (or not), so if you have some 2300
>> files to backup, that would sound about right. If the target or the
>> source file doesn't exist, it would give an error.
>>
>> If the files are small or don't have changes, the lstat happen a lot and
>> nothing much else; this is typical random access. It gives a much
>> different access pattern than the copying of bigger files, where more
>> sequential is typically done to read/write the file's data.
>>
>> There is no real way to improve the situation, rdiff-backup goes as fast
>> as it can and I personally don't know an I/O-equivalent of "nice" (and
>> if you limit the I/O, the backup will be even slower).
>>
>> You could try the --no-fsync option to improve speed:
>>
>>     --fsync, --no-fsync   [opt] do (or not) often sync the file system
>> (_not_ doing it is faster but can be dangerous)
>>
>> And, yes, the `rdiff-backup-data/increments` directory is used by
>> rdiff-backup to keep track of file and directory changes.
>>
>> Hope this helps,
>> Eric
>>
>> On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
>>>
>>> Hi rdiff-backup folks,
>>>
>>> Since recent, during backing up I can see spike in IOPS up to 500 which exhaust limit of a VM. Therefore backup process takes very long. I've straced a bit and what I can see is: many failed  lstat() syscalls:
>>> % time     seconds  usecs/call     calls    errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>>    42.71    0.040247           9      4608      1420 lstat
>>>    35.41    0.033370          12      2860           getdents
>>>     9.41    0.008865           6      1431           open
>>>     4.63    0.004363           3      1430           close
>>>     4.03    0.003797           3      1431           fstat
>>>     3.75    0.003536           2      1417           getuid
>>>     0.04    0.000039          39         1           unlink
>>>     0.01    0.000013           1         9           read
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00    0.094230                 13187      1420 total
>>> Seems rdiff-backup checks existence of some file/dir:
>>> 10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar", 0x7ffd832fa810) = -1 ENOENT (No such file or directory) <0.000020>
>>> After backup is done, there is still no such file.
>>> Seems the part in path -  /rdiff-backup-data/increments/ - is some "config" for rdiff-backup and probably it tryies to find something but can't?
>>>
>>> What might be wrong in my setup? What would you recommend to check to solve the issue if it is issue at all?
>>>
>>> ---
>>> Best Regards,
>>> Andrei Enshin
>>>
>>