Re: Memory exhaustion issue (#1068)

This is a multi-part message in MIME format.
--===============0621183758541986721==
Content-Type: multipart/alternative;
 boundary="------------o6vs28Zm2IE30RH1cEKiJ2Kz"
Content-Language: en-US

This is a multi-part message in MIME format.
--------------o6vs28Zm2IE30RH1cEKiJ2Kz
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 05/01/2025 15:39, Greg Troxel wrote:
> What are you doing about swap space?  Does your system page?  If you are
> running a RPI3 in aarch64 mode, you could configure 32GB of swap space
> on a USB-attached SSD, and then unison should run.  Have you tried that?
I originally had 3 GB of swap space on a USB-attached magnetic disk. The
result is that, as soon as physical memory is exhausted, the system
becomes unusable because it spends most if its time waiting for swapping
operations to complete. An SSD might be faster, but the connection is
still over USB 2.0, so it might not even make a difference at all.
>> I=E2=80=99d be curious to know what the =E2=80=9Caverage=E2=80=9D use c=
ase for Unison looks
>> like. Does anyone happen to have an idea of how people out there are
>> using Unison?
> There are about 1000 people on the users list, so I'd guess there are at
> least 10K users.  But I'm guessing.   Unison, being honorable Free
> Software, does not contain tracking code to report.
Thank God there=E2=80=99s no tracking code in Unison :-) that=E2=80=99s wh=
at I expected,
but I thought you guys might have an idea based on discussions on this
lists, bug reports etc. You=E2=80=99ve probably seen a lot more usage scen=
arios
than I have.
>> For my use case, around a terabyte of data in a million files, accessed
>> by a handful of users and fairly static, would be well within the means
>> of a Pi 3 with 1 GB of RAM =E2=80=93 as long as we=E2=80=99re just talk=
ing about file
>> sharing via CIFS or SFTP. It=E2=80=98s only when Unison gets involved t=
hat the
>> system reaches its limit.
> CIFS/SFTP is not a fair comparison, because that is access not sync.
This was not meant to imply that others do the job better, but to
illustrate the perspective of a user who adds sync capabilities to their
existing file server: without sync, 1 GB (probably even less) will work
just fine. But as soon as sync enters the picture, a lot more memory is
suddenly needed. Other tools might have similar issues =E2=80=93 except th=
at the
=E2=80=9Ccompetitors=E2=80=9D (Syncthing, git etc.) work quite differently=
 from Unison
and are definitely not a drop-in replacement.
> You could invent virtual memory and use it :-)
See above =E2=80=93 disk access times make this impractical.
> Or you could implement
> application-specific virtual memory.   This would look like on-disk
> storage for archives, with a cache of objects in memory, and reading
> them on demand.
The general issue with any kind of virtual memory and swapping is to
predict what data will be needed, and what is OK to swap out. Keeping
the archive on disk and caching just parts in memory is a kind of memory
virtualization what swapping as well, except it happens in the
application and the code can be tuned to minimize swap operations. For
example, if archive entries are identified by their hash code, one could
make the order in which files are compared dependent on hash codes, so
each chunk of the archive needs to be loaded into memory only once in
the comparison step.
> You could also choose to generate profiles for subparts of the file tree
> and run them sequentially.  I have organized files by directory and sync
> them separately anyway, because I want to control which directories get
> synced to which subset of places, for various reasons not about unison.

That=E2=80=99s what I did before. However, I sometimes move files between =
these
subparts (and incidentally, these tend to be on the larger side, several
GB per file). These files would then get copied over the wire again,
instead of the move operation just being mirrored on the other end. And
with a copy-on-write filesystem with snapshots, there are further
implications regarding disk space.

--------------o6vs28Zm2IE30RH1cEKiJ2Kz
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-=
8">
  </head>
  <body>
    <div class=3D"moz-cite-prefix">On 05/01/2025 15:39, Greg Troxel wrote:=
<br>
    </div>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <pre wrap=3D"" class=3D"moz-quote-pre">What are you doing about swap=
 space?  Does your system page?  If you are
running a RPI3 in aarch64 mode, you could configure 32GB of swap space
on a USB-attached SSD, and then unison should run.  Have you tried that?</=
pre>
    </blockquote>
    I originally had 3 GB of swap space on a USB-attached magnetic disk.
    The result is that, as soon as physical memory is exhausted, the
    system becomes unusable because it spends most if its time waiting
    for swapping operations to complete. An SSD might be faster, but the
    connection is still over USB 2.0, so it might not even make a
    difference at all.<span style=3D"white-space: pre-wrap">
</span>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <blockquote type=3D"cite">
        <pre wrap=3D"" class=3D"moz-quote-pre">I=E2=80=99d be curious to k=
now what the =E2=80=9Caverage=E2=80=9D use case for Unison looks
like. Does anyone happen to have an idea of how people out there are
using Unison?
</pre>
      </blockquote>
      <pre wrap=3D"" class=3D"moz-quote-pre">
There are about 1000 people on the users list, so I'd guess there are at
least 10K users.  But I'm guessing.   Unison, being honorable Free
Software, does not contain tracking code to report.</pre>
    </blockquote>
    Thank God there=E2=80=99s no tracking code in Unison :-) that=E2=80=99=
s what I
    expected, but I thought you guys might have an idea based on
    discussions on this lists, bug reports etc. You=E2=80=99ve probably se=
en a
    lot more usage scenarios than I have.<br>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <blockquote type=3D"cite">
        <pre wrap=3D"" class=3D"moz-quote-pre">For my use case, around a t=
erabyte of data in a million files, accessed
by a handful of users and fairly static, would be well within the means
of a Pi 3 with 1 GB of RAM =E2=80=93 as long as we=E2=80=99re just talking=
 about file
sharing via CIFS or SFTP. It=E2=80=98s only when Unison gets involved that=
 the
system reaches its limit.
</pre>
      </blockquote>
      <pre wrap=3D"" class=3D"moz-quote-pre">
CIFS/SFTP is not a fair comparison, because that is access not sync.</pre>
    </blockquote>
    This was not meant to imply that others do the job better, but to
    illustrate the perspective of a user who adds sync capabilities to
    their existing file server: without sync, 1 GB (probably even less)
    will work just fine. But as soon as sync enters the picture, a lot
    more memory is suddenly needed. Other tools might have similar
    issues =E2=80=93 except that the =E2=80=9Ccompetitors=E2=80=9D (Syncth=
ing, git etc.) work
    quite differently from Unison and are definitely not a drop-in
    replacement.<br>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <pre wrap=3D"" class=3D"moz-quote-pre">You could invent virtual memo=
ry and use it :-)</pre>
    </blockquote>
    See above =E2=80=93 disk access times make this impractical.<br>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <pre wrap=3D"" class=3D"moz-quote-pre">Or you could implement
application-specific virtual memory.   This would look like on-disk
storage for archives, with a cache of objects in memory, and reading
them on demand.</pre>
    </blockquote>
    The general issue with any kind of virtual memory and swapping is to
    predict what data will be needed, and what is OK to swap out.
    Keeping the archive on disk and caching just parts in memory is a
    kind of memory virtualization what swapping as well, except it
    happens in the application and the code can be tuned to minimize
    swap operations. For example, if archive entries are identified by
    their hash code, one could make the order in which files are
    compared dependent on hash codes, so each chunk of the archive needs
    to be loaded into memory only once in the comparison step.<br>
    <blockquote type=3D"cite" cite=3D"mid:[email protected]">
      <pre wrap=3D"" class=3D"moz-quote-pre">You could also choose to gene=
rate profiles for subparts of the file tree
and run them sequentially.  I have organized files by directory and sync
them separately anyway, because I want to control which directories get
synced to which subset of places, for various reasons not about unison.
</pre>
    </blockquote>
    <p>That=E2=80=99s what I did before. However, I sometimes move files b=
etween
      these subparts (and incidentally, these tend to be on the larger
      side, several GB per file). These files would then get copied over
      the wire again, instead of the move operation just being mirrored
      on the other end. And with a copy-on-write filesystem with
      snapshots, there are further implications regarding disk space.<br>
    </p>
  </body>
</html>

--------------o6vs28Zm2IE30RH1cEKiJ2Kz--

--===============0621183758541986721==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Unison-hackers mailing list
[email protected]
https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers

--===============0621183758541986721==--