Re: big picture: is use of rsync actually useful

On Sat, 11 Mar 2023 at 15:22, Greg Troxel <[email protected]> wrote:
>
> The first question, extended slightly, is:
>
>   Unison allows using an external copy program via copythreshold, on the
>   theory that this is more efficient.  Is there any recent evidence that
>   this is actually true?   If not, should we just remove the external
>   copyprog mechanism?   If so, would it make sense to improve the
>   internal sync and still remove the external support?

My understanding of the origins of this functionality is this (note,
these are not facts! I wasn't anywhere close to Unison's development
until very recently). In the early days (we're talking 20+ years ago),
Unison's rsync algorithm implementation was a bit untuned/simplistic,
also the networking code had some limitations. This caused the sync of
big files to be slower than what rsync could do. I don't know if it
really was slower or not. Since then, there have been many
improvements to Unison's networking code and rsync algorithm
implementation. I think the latest wave of improvements was during
2009 - 2010. These improvements, combined with improved speed of disks
and network connections, likely have removed any speed advantage that
using rsync may have had.

Even if rsync remains faster in terms of the core rsync algorithm
(which is likely), what does this "faster" mean? It can only come from
rsync being able to produce a smaller delta than Unison. Reading from
disk speed and read amount is the same for both, writing to disk speed
is the same for both, network speed is the same for both. In this
context, unless rsync can produce a delta of 1 MB where Unison would
produce a delta of 1 GB for the same changes, I don't see any reason
to use external rsync. I don't have any empirical data to know if
rsync can regularly produce considerably smaller deltas.

If there are real issues with Unison's implementation of rsync
algorithm (such that syncing of VM images, for example, becomes
miserable) then I would prefer improving that implementation over
using the external rsync. Maybe librsync could also be something to
look at.

> The second question asked in the ticket by @rrthomas was:
>
>   Why not remove Unison's internal copying, and just use rsync, replacing
>   copyprog and copyprogrest with a single preference to specify extra
>   rsync options? (I bet there are good reasons, but I'd like to know what
>   they are!)

I doubt this is the way to go. Depending on an unstable interface,
spawning a process for every transfer, introducing a new runtime
dependency, this all makes for a very unpredictable environment and
introduces new failure points, which are black boxes on top of
everything. Let's also not forget that we have to make sure this works
on well all platforms, including Windows and (possibly in future)
Android and iOS.

I can see librsync being considered. But I don't know anything about
this project or whether it is even suitable for this purpose.