high memory usage transferring about 27.000.000 files

Valentin Kleibel via rsync <[email protected]>
Newsgroups gmane.network.rsync.general
Message-ID <[email protected]>
We are trying to back up a large number of small files and have issues 
with high memory usage.
On both ends debian 11.0 is running with rsync version 3.2.3  protocol 
version 31.
Ideally we want to use the options `-a --delete --backup -e ssh` to run 
this transfer and pull a backup from the target machine.
The memory usage continually rises until using all free memory on the 
initiator side, about 4.5GB split across two processes. On the sender 
side one process uses about half of that.
The folder to transfer contains about 11TB of data split over 
~27.000.000 files in a structure with ~19.000.000 directories.
We first tried to make sure that rsync is using it's incremental 
recursion mode. After reading the manual we explicitly used the 
`--delete-during` option as well as trying to run without deletion or 
backup mode with no noticeable improvements.
The command line options on the sender side, as far as i understand, 
indicate that incremental recursion mode is requested 
(`-vlogDtpre.iLsfxCIvu`).

We also tried --omit-dir-times to reduce memory usage without success.

We desperately try not to split up the transfer for subfolders as we 
would have to do so dynamically because the folder names change.

Does anyone have experience with such transfers and can help us with 
reducing the memory usage?
Or is there an issue and rsync shouldn't use that much memory?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.