Re: Rsync not deleting files using --remove-source-files

Hi there,

On Sun, 12 Apr 2026, Sean Blaes wrote:

> We use rsync to synchronize files up to a remote host ...  limited
> bandwidth (< 1 mbps) and is intermittent (often active only for
> about 10 minutes at a time, potentially hours between connections)

What's the actual connection?  Simply rsync doing transfers to/from a
remote host, or do you have something like a VPN in the mix?  If I
hadn't already tried something like that, I'd probably give it a whirl
just to see if the VPN configuration could offer more control over the
networking feedback to rsync, to prevent rsync from being killed.  But
I'm sure there's a better way, see below.

> The files being transferred are 100-200 megabytes ... several files
> queued ... often require several windows of connectivity ...

You seem to like to make things difficult for yourself. :)

> We use the following cli flags for this: `--verbose --progress
> --exclude=".*" --timeout=15 --partial-dir=.partial --partial --recursive
> --remove-source-files`.

Can I take it that you've tried longer timeouts (in the region of hours)?

> I observe that sometimes the transfer of file1 completes successfully, and
> then the process moves to file2. However, even after transferring file2 for
> several minutes, file1 remains on the client's filesystem.
> 
> I've been looking a bit at the rsync code, and I think I have determined
> that it prioritizes transferring buffered file data over sending control
> messages (such as the MSG_SUCCESS message that would result in the client
> side file being deleted). So my suspicion is that it's loading the data
> buffer with data from file2 and trying to transfer that data. Consequently,
> the MSG_SUCCESS message is never prioritized as rsync is killed (due to end
> of network connection) before it can make it through, even though file1 was
> successfully transferred several minutes earlier.
> 
> Does this sound right?

It sounds plausible to me.

> Are there any outstanding bugs or feature requests that might
> improve this situation (I searched and couldn't find any)?

Sorry, IDK.  It seems to me a very niche case which wouldn't be at the
top of my list (even if I were looking at such requests, which I'm not).

> It seems sensible to prioritize sending MSG_SUCCESS over file data
> so that this workflow works more reliably. Would that make sense?

In view of the potential for unexpected effects I'd be very hesitant
to do anything like that.

It would make a lot more sense to me to split the big files into small
ones at the sending end, then reassemble them at the receiving end.
Kinda like TCP/IP does with packets.

-- 

73,
Ged.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html