Re: [PATCH] Reduce memory usage
Jindřich Makovička via rsync <[email protected]>
| Newsgroups | gmane.network.rsync.general |
|---|---|
| Message-ID | <20210927164225.13e6bf9d@holly> |
On Mon, 27 Sep 2021 13:38:22 +0200 Jindřich Makovička <[email protected]> wrote: > On Sun, 26 Sep 2021 13:54:13 +0200 > Jindřich Makovička <[email protected]> wrote: > > Hi, > > > > ... > > > > Applying the attached patch, which reduces the default allocation to > > 32 pointers, and preallocates 32K pointers only for the main file > > lists in send_file_list and recv_file_list, reduces the peak memory > > usage in my case from 142MB to 12MB. > > The original patch breaks the testsuite due to extra messages in the > output. Actually it makes more sense to change the debug print to check that the original pointer is non-null and leave the testsuite as is. -- Jindrich Makovicka -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
0001-Reduce-memory-usage.patch
(text/x-patch, 1.9 KB)
From bdfdef1c5a4437e2492da148b824d39ba235704e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <[email protected]> Date: Sun, 26 Sep 2021 12:01:21 +0200 Subject: [PATCH] Reduce memory usage Start only with 32 entries for the partial file lists, instead of 32k. --- flist.c | 4 +++- rsync.h | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..0f7a64e6 100644 --- a/flist.c +++ b/flist.c @@ -305,7 +305,7 @@ static void flist_expand(struct file_list *flist, int extra) new_ptr = realloc_array(flist->files, struct file_struct *, flist->malloced); - if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) { + if (DEBUG_GTE(FLIST, 1) && flist->files) { rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n", who_am_i(), big_num(sizeof flist->files[0] * flist->malloced), @@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); flags |= FLAG_DIVERT_DIRS; @@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { if (flist->ndx_start == 1) diff --git a/rsync.h b/rsync.h index 88319732..17f8700e 100644 --- a/rsync.h +++ b/rsync.h @@ -918,8 +918,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START (32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START (32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB -- 2.33.0