Re: [PATCH] Reduce memory usage
Jindřich Makovička via rsync <[email protected]>
| Newsgroups | gmane.network.rsync.general |
|---|---|
| Message-ID | <20210927133822.7be99724@holly> |
On Sun, 26 Sep 2021 13:54:13 +0200 Jindřich Makovička <[email protected]> wrote: > Hi, > > ... > > Applying the attached patch, which reduces the default allocation to > 32 pointers, and preallocates 32K pointers only for the main file > lists in send_file_list and recv_file_list, reduces the peak memory > usage in my case from 142MB to 12MB. The original patch breaks the testsuite due to extra messages in the output. Fix attached. Regards, -- Jindrich Makovicka -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
0001-Reduce-memory-usage.patch
(text/x-patch, 1.8 KB)
From aa907eabe701550cd5649b6e3da2b22b79b38a06 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <[email protected]> Date: Sun, 26 Sep 2021 12:01:21 +0200 Subject: [PATCH] Reduce memory usage Start only with 32 entries for the partial file lists, instead of 32k. --- flist.c | 2 ++ rsync.h | 5 +++-- testsuite/rsync.fns | 1 + 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..37f70b69 100644 --- a/flist.c +++ b/flist.c @@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); flags |= FLAG_DIVERT_DIRS; @@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { if (flist->ndx_start == 1) diff --git a/rsync.h b/rsync.h index 88319732..17f8700e 100644 --- a/rsync.h +++ b/rsync.h @@ -918,8 +918,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START (32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START (32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB diff --git a/testsuite/rsync.fns b/testsuite/rsync.fns index 1e2b399f..220b9e21 100644 --- a/testsuite/rsync.fns +++ b/testsuite/rsync.fns @@ -89,6 +89,7 @@ v_filt() { -e '/^total: /d' \ -e '/^client charset: /d' \ -e '/^server charset: /d' \ + -e '/ expand file_list pointer array /d' \ -e '/^$/,$d' } -- 2.33.0