Re: Rdiff-backup v2.2.6 list increments slowed by uid or gid name lookups

Hi,

sorry, I misread you.

Do you use the list increments with size? It takes the size directly 
from the file system, as it isn't stored anywhere, and can take quite a 
long time. If not, I'd have to dig deeper into the code.

KR, Eric

On 12/07/2025 09:53, Ty Boyack wrote:
> Thanks for that thought, Eric! Just for clarity, backups work great, are 
> not unnecessarily slow, and I am happy with the way the user id mappings 
> all work. It is just the "list increments" options that is causing the 
> question.
> 
> I think the --preserve-numerical-ids is not available during a "list 
> increments" operation. My question is really wondering what happens 
> during a list increments process. I would think it would look into the 
> rdiff-backup-directory and get the list of increment directories or 
> files and return the list of increments. But instead it stats all of the 
> top level folders which is what takes so much time. Why does it need to 
> stat all the top level directories? What is gained by stating all those 
> rather than just pulling data from the rdiff-backup-data folder? in 
> other words I don't want to request a complex caching mechanism if it is 
> just a step that could be skipped altogether.
> 
> Thanks for the help and thoughts,
> 
> -Ty
> 
> 
> 
> On 7/12/25 1:36 AM, [email protected] wrote:
>> ** Caution: EXTERNAL Sender **
>>
>> Hi,
>>
>> rdiff-backup saves by default user/group names so that they can be
>> restored on a different server even if they have different uid/gid.
>> You could try with --preserve-numerical-ids but I'm not sure this will
>> help because I think rdiff-backup will still try to save both, just
>> favorize one or the other.
>>
>> So, if that doesn't help, we could think about a caching mechanism but
>> not any time soon. I'd ask you to create an issue as RFE for this.
>>
>> KR, Eric
>>
>> On 11/07/2025 12:15, Ty Boyack wrote:
>>> Hi,b
>>>
>>> Like so many others I want to express my thanks for all the recent
>>> development (and previous development) that has gone into this great
>>> piece of software!
>>>
>>> I have noticed a slowdown that could be from a change in our system or
>>> from code changes in rdiff-backup, but either way I am wondering if
>>> performance could be improved.
>>>
>>> Our situation is that we have a volume of user network home directories,
>>> which has over 50,000 top level directories in it (and millions of files
>>> below that). Each directory is owned by a different user. I've been
>>> backing it up for years with rdiff-backup without any problems.
>>>
>>> As I am migrating to new storage servers, the "list increments" command
>>> is painfully slow, taking hours to complete. Tracing the system calls I
>>> see that it is doing a newfstatat() call on every one of those 50k top
>>> level directories. I don't know if this was done in previous versions or
>>> if this is new. Our previous storage systems used NIS to look up
>>> usernames/uids (which was very fast), while the new one uses Active
>>> Directory/SSSD. The calls to SSSD are what is causing the slowness,
>>> taking around half a second per call. (I'll happily entertain thoughts
>>> that this is a problem and too slow, but for now I need to accept it as
>>> a given.)
>>>
>>> My question for the rdiff-backup developers, is why do we need to do
>>> stat all these top level directories to get the list of increments at
>>> all? Shouldn't that information all be in the rdiff-backup-data folder?
>>> If we do need to do some form of stat on these top directories, can it
>>> be done in such a way as to work with numberic uid/gid info rather than
>>> initiating a call to name services? If we did not have to hit Active
>>> Directory for each of those folders the speed would be drastically
>>> improved in this use case.
>>>
>>> I have only spent a little time looking into this issue so I might not
>>> be seeing everything correctly, but I'd love to hear thoughts about 
>>> this.
>>>
>>> Thanks,
>>>
>>> -Ty
>>>
>>>
>>>
>>
>>
>