Re: Rdiff-backup v2.2.6 list increments slowed by uid or gid name lookups

Hi,

sorry, I realized that I haven't answered your last e-mail.

The answer is sadly: works as designed.

Before starting anything, rdiff-backup checks the abilities of the used 
file system(s), and to do so, for read-only actions (like listing 
increments), it tries to find files with specific characteristics to 
check if they are supported. This most probably explains what you are 
seeing.

This said, I realized that we already have a caching mechanism for 
users/groups lookup, so it shouldn't take that much time under normal 
circumstances, but of course 50k lookups times 0.5 seconds as you wrote, 
mean 25k seconds, aka 6+ hours. That's a lot but I don't see that I can 
do much to improve the situation for what is an extremely bad reaction 
time. I would rather try to fix this sssd/AD performance issue, which 
should help everywhere, not only rdiff-backup.

Sorry, Eric

On 18/07/2025 08:44, Ty Boyack wrote:
> Hi Eric,
> 
> Thanks very much for the follow up and for keeping the ideas flowing!
> 
> The command I am using is just:
> rdiff-backup list increments /path/to/backup/
> so I'm not trying to get the size.
> 
> I like your thought though, so I just tried both:
> rdiff-backup list increments --no-size /path/to/backup/
> and
> rdiff-backup list increments --size /path/to/backup/
> and --no-size was the same behavior as not using any size related 
> option, while adding --size was slower (but of course has more info).
> 
> I can see the issue most clearly with a command like:
> strace -e newfstatat -f rdiff-backup list increments --no-size /path/to/ 
> backup/ 2>&1 | egrep '/path/to/backup'
> 
> (The pipe to egrep is just to clean up the output and not show all the 
> newfstatats to system files.)
> 
> Rdiff-backup appears to do:
> 1) a few newfstatat calls to files within the rdiff-backup-data folder, 
> then
> 2) a newfstatat of each top level folder in the backup repo, and then
> 3) newfstatat calls on files within the rdiff-backup-data folder like 
> all of the increments.date.dir and current_mirror files.
> 
> So if my backup repository looks like:
> /path/to/backup/directoryA
> /path/to/backup/directoryB
> /path/to/backup/directoryC
> /path/to/backup/rdiff-backup-data
> 
> I'd see groups of newfstatat calls in this order:
> /path/to/backup/
> /path/to/backup/rdiff-backup-data
> /path/to/backup/directoryA *
> /path/to/backup/directoryB *
> /path/to/backup/directoryC *
> /path/to/backup/rdiff-backup-data *
> /path/to/backup/rdiff-backup-data/increments.date.dir...
> /path/to/backup/rdiff-backup-data/increments.date.dir...
> /path/to/backup/rdiff-backup-data/increments.date.dir...
> /path/to/backup/rdiff-backup-data/current_mirror
> 
> The slow behavior is coming from the time it spends outside of the 
> rdiff-backup-data folder -- the ones marked with an asterisk.
> 
> I am running rdiff-backup 2.2.6 on Rocky Linux 10. Sorry I didn't 
> mention that before!
> 
> Again, thanks for any thoughts on this.
> 
> -Ty
> 
> 
> On 7/17/25 11:33 PM, [email protected] wrote:
>> ** Caution: EXTERNAL Sender **
>>
>> Hi,
>>
>> sorry, I misread you.
>>
>> Do you use the list increments with size? It takes the size directly
>> from the file system, as it isn't stored anywhere, and can take quite a
>> long time. If not, I'd have to dig deeper into the code.
>>
>> KR, Eric
>>
>> On 12/07/2025 09:53, Ty Boyack wrote:
>>> Thanks for that thought, Eric! Just for clarity, backups work great, are
>>> not unnecessarily slow, and I am happy with the way the user id mappings
>>> all work. It is just the "list increments" options that is causing the
>>> question.
>>>
>>> I think the --preserve-numerical-ids is not available during a "list
>>> increments" operation. My question is really wondering what happens
>>> during a list increments process. I would think it would look into the
>>> rdiff-backup-directory and get the list of increment directories or
>>> files and return the list of increments. But instead it stats all of the
>>> top level folders which is what takes so much time. Why does it need to
>>> stat all the top level directories? What is gained by stating all those
>>> rather than just pulling data from the rdiff-backup-data folder? in
>>> other words I don't want to request a complex caching mechanism if it is
>>> just a step that could be skipped altogether.
>>>
>>> Thanks for the help and thoughts,
>>>
>>> -Ty
>>>
>>>
>>>
>>> On 7/12/25 1:36 AM, [email protected] wrote:
>>>> ** Caution: EXTERNAL Sender **
>>>>
>>>> Hi,
>>>>
>>>> rdiff-backup saves by default user/group names so that they can be
>>>> restored on a different server even if they have different uid/gid.
>>>> You could try with --preserve-numerical-ids but I'm not sure this will
>>>> help because I think rdiff-backup will still try to save both, just
>>>> favorize one or the other.
>>>>
>>>> So, if that doesn't help, we could think about a caching mechanism but
>>>> not any time soon. I'd ask you to create an issue as RFE for this.
>>>>
>>>> KR, Eric
>>>>
>>>> On 11/07/2025 12:15, Ty Boyack wrote:
>>>>> Hi,b
>>>>>
>>>>> Like so many others I want to express my thanks for all the recent
>>>>> development (and previous development) that has gone into this great
>>>>> piece of software!
>>>>>
>>>>> I have noticed a slowdown that could be from a change in our system or
>>>>> from code changes in rdiff-backup, but either way I am wondering if
>>>>> performance could be improved.
>>>>>
>>>>> Our situation is that we have a volume of user network home 
>>>>> directories,
>>>>> which has over 50,000 top level directories in it (and millions of 
>>>>> files
>>>>> below that). Each directory is owned by a different user. I've been
>>>>> backing it up for years with rdiff-backup without any problems.
>>>>>
>>>>> As I am migrating to new storage servers, the "list increments" 
>>>>> command
>>>>> is painfully slow, taking hours to complete. Tracing the system 
>>>>> calls I
>>>>> see that it is doing a newfstatat() call on every one of those 50k top
>>>>> level directories. I don't know if this was done in previous 
>>>>> versions or
>>>>> if this is new. Our previous storage systems used NIS to look up
>>>>> usernames/uids (which was very fast), while the new one uses Active
>>>>> Directory/SSSD. The calls to SSSD are what is causing the slowness,
>>>>> taking around half a second per call. (I'll happily entertain thoughts
>>>>> that this is a problem and too slow, but for now I need to accept 
>>>>> it as
>>>>> a given.)
>>>>>
>>>>> My question for the rdiff-backup developers, is why do we need to do
>>>>> stat all these top level directories to get the list of increments at
>>>>> all? Shouldn't that information all be in the rdiff-backup-data 
>>>>> folder?
>>>>> If we do need to do some form of stat on these top directories, can it
>>>>> be done in such a way as to work with numberic uid/gid info rather 
>>>>> than
>>>>> initiating a call to name services? If we did not have to hit Active
>>>>> Directory for each of those folders the speed would be drastically
>>>>> improved in this use case.
>>>>>
>>>>> I have only spent a little time looking into this issue so I might not
>>>>> be seeing everything correctly, but I'd love to hear thoughts about
>>>>> this.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Ty
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>