Rsync not deleting files using --remove-source-files

--===============1770121669957912949==
Content-Type: multipart/alternative; boundary="000000000000bb64ca064f3d279c"

--000000000000bb64ca064f3d279c
Content-Type: text/plain; charset="UTF-8"

We use rsync to synchronize files up to a remote host, intending to clear
files from the client filesystem once the rsync transfer succeeds. The
connection has limited bandwidth (< 1 mbps) and is intermittent (often
active only for about 10 minutes at a time, potentially hours between
connections), due to connectivity via satellite. The files being
transferred are 100-200 megabytes. Since we may have several files queued,
the transfers take time and often require several windows of connectivity
to complete. We use the following cli flags for this: `--verbose --progress
--exclude=".*" --timeout=15 --partial-dir=.partial --partial --recursive
--remove-source-files`.

I observe that sometimes the transfer of file1 completes successfully, and
then the process moves to file2. However, even after transferring file2 for
several minutes, file1 remains on the client's filesystem.

I've been looking a bit at the rsync code, and I think I have determined
that it prioritizes transferring buffered file data over sending control
messages (such as the MSG_SUCCESS message that would result in the client
side file being deleted). So my suspicion is that it's loading the data
buffer with data from file2 and trying to transfer that data. Consequently,
the MSG_SUCCESS message is never prioritized as rsync is killed (due to end
of network connection) before it can make it through, even though file1 was
successfully transferred several minutes earlier.

Does this sound right? Are there any outstanding bugs or feature requests
that might improve this situation (I searched and couldn't find any)? It
seems sensible to prioritize sending MSG_SUCCESS over file data so that
this workflow works more reliably. Would that make sense?

--000000000000bb64ca064f3d279c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">We use rsync to synchronize files up to a remote host, int=
ending to clear files from the client filesystem once the rsync transfer su=
cceeds. The connection has limited bandwidth (&lt; 1 mbps) and is intermitt=
ent (often active only for about 10 minutes=C2=A0at a time, potentially hou=
rs between connections), due to connectivity via satellite. The files being=
 transferred are 100-200 megabytes. Since we may have several files queued,=
 the transfers take time and often require several windows of connectivity =
to complete. We use the following cli flags for this: `--verbose --progress=
 --exclude=3D&quot;.*&quot; --timeout=3D15 --partial-dir=3D.partial --parti=
al --recursive --remove-source-files`.<div><br></div><div>I observe that so=
metimes the transfer of file1 completes successfully, and then the process =
moves to file2. However, even after transferring file2 for several minutes,=
 file1 remains on the client&#39;s filesystem.</div><div><br></div><div>I&#=
39;ve been looking a bit at the rsync code, and I think I have determined t=
hat it prioritizes transferring buffered file data over sending control mes=
sages (such as the MSG_SUCCESS message that would result in the client side=
 file being deleted). So my suspicion is that it&#39;s loading the data buf=
fer with data from file2 and trying to transfer that data. Consequently, th=
e MSG_SUCCESS message is never prioritized as rsync is killed (due to end o=
f network connection) before it can make it through, even though file1 was =
successfully transferred several minutes earlier.</div><div><br></div><div>=
Does this sound right? Are there any outstanding bugs or feature requests t=
hat might improve this situation (I searched and couldn&#39;t find any)? It=
 seems sensible to prioritize sending MSG_SUCCESS over file data so that th=
is workflow works more reliably. Would that make sense?</div></div>

--000000000000bb64ca064f3d279c--


--===============1770121669957912949==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--===============1770121669957912949==--