Re: Is there a better way to transfer data that doesn't use so much cache?

On Wed, Aug 3, 2022 at 5:41 PM Robin Lee Powell via rsync <
[email protected]> wrote:

> On Wed, Aug 03, 2022 at 02:04:22PM -0400, Rob Campbell via rsync wrote:
> > The problem isn't that there are many syncs because the problem happens
> on
> > the first one that runs.
>
> You didn't actually say what the problem *is*.
>
> I can infer from the subject that you think it's bad that rsync is
> using a bunch of disk/buffer cache, but that's not rsync, that's
> Linux, and it's by design; Linux uses as much RAM as it possibly can
> for disk cache, always.  This improves performance.  In a
> well-performing Linux system, the "free" column of "free -h" is very
> low, and the "available" column is very high.
>

Linux does indeed try to put your RAM to good use, and often that means
caching data from disk in RAM.

However, if you transfer a large amount of data and do not intend to
retransmit that data any time soon, then the memory isn't really put to
good use, and can actually cause your system to slow down significantly -
particularly if there's a lot of such data transferred.

It is, however, theoretically possible to skip the buffer cache using
O_DIRECT, but that requires your application to have O_DIRECT support, or
to use something like https://stromberg.dnsalias.org/~strombrg/libodirect/

HTH.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html