Re: Memory exhaustion issue (#1068)
Michael von Glasow <[email protected]> Sat, 23 Nov 2024 23:26:43 +0200
| Newsgroups | gmane.network.unison.devel |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============8937841426179453067== Content-Type: multipart/alternative; boundary="------------maZJLDxD5eeF0XkGLgz7y4bn" Content-Language: en-US This is a multi-part message in MIME format. --------------maZJLDxD5eeF0XkGLgz7y4bn Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable On 23/11/2024 23:01, Greg Troxel wrote: > Michael von Glasow<[email protected]> writes: > >> Switching profiles should be sufficient =E2=80=93 this created a new un= ison >> process on the server, while the old one gradually freed up its memory >> (but kept running). > Sorry, I guess I was unclear. I am not really looking for just > sufficient to make your case, but the simplest possible way to reproduce > the problem, expressed programmatically so that others can run it (after > reading the code to feel it is safe). So that's no GUI, no persistent > server, and everything in the profile to be synced created by the > script. I=E2=80=98ll look into it, but reproducing would also involve creating a h= uge set of files to sync. > For comparing with other people, I think it will be more useful to talk > about KB or MB of memory usage vs %. Lots of people will have different > amounts of RAM and other loads, although 1 GB such as RPI3 is pretty > common. In principle, +1. However, the tools I have spit out percentages. One % is roughly 10 MB, so the math is fairly easy. > I personally am not interested in debugging anything older than 2.53.7 > (or 6, but there is no reason to use 6 if you are compiling). You do you, but the trouble is that distro repositories are a bit slow to catch up. Part of the whole update exercise was so I could finally return to Unison from the distro repos and not have to build my own any more. Especially since building for non-Intel platforms is quite a hassle =E2=80=93 I tried to get that on CI some time ago, but my attempts stalled as setting up a cross-compilation environment proved challenging. > So it sort of sounds like memory is allocated proportional to the size > of the transferred file, and not freed. > > And, that other memory uesd for scan/etc. is reused. It is roughly proportional to the size of the transferred file, although 16000M uses more in comparison than 12800M does (roughly 55M vs. 34M). Also, archive file size seems to matter. >> Looking at the docs, what comes to mind is: >> >> - copyprog, copyprogthreshold (use external program <copyprog> for >> copying files larger than <copyprogthreshold> kB) > That is about to be deleted. That would be too bad, because that seems to be a valid workaround. I added `copythreshold =3D 163840` (1% of 16G, roughly where the issues started) to my profile and was able to sync. >> Or is there a way to tell Unison to stop being smart and just copy the >> damn thing (which is presumably less memory-hungry) if a file is larger >> than a certain size? > I don't think so, but really that should not be necessary. If there is > code that uses memory when it shouldn't, we should find that and fix it. For =E2=80=9Ccut the smartness above file size X=E2=80=9D, `copythreshold`= (not `copyprogthreshold`, that was a typo) seems to do the trick. Please keep that feature until memory efficiency is improved, unless there is an alternative. --------------maZJLDxD5eeF0XkGLgz7y4bn Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-= 8"> </head> <body> <div class=3D"moz-cite-prefix">On 23/11/2024 23:01, Greg Troxel wrote:= <br> </div> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <pre wrap=3D"" class=3D"moz-quote-pre">Michael von Glasow <a class= =3D"moz-txt-link-rfc2396E" href=3D"mailto:[email protected]"><micha= [email protected]></a> writes: </pre> <blockquote type=3D"cite"> <pre wrap=3D"" class=3D"moz-quote-pre">Switching profiles should b= e sufficient =E2=80=93 this created a new unison process on the server, while the old one gradually freed up its memory (but kept running). </pre> </blockquote> <pre wrap=3D"" class=3D"moz-quote-pre"> Sorry, I guess I was unclear. I am not really looking for just sufficient to make your case, but the simplest possible way to reproduce the problem, expressed programmatically so that others can run it (after reading the code to feel it is safe). So that's no GUI, no persistent server, and everything in the profile to be synced created by the script.</pre> </blockquote> I=E2=80=98ll look into it, but reproducing would also involve creating= a huge set of files to sync.<br> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <pre wrap=3D"" class=3D"moz-quote-pre">For comparing with other peop= le, I think it will be more useful to talk about KB or MB of memory usage vs %. Lots of people will have different amounts of RAM and other loads, although 1 GB such as RPI3 is pretty common.</pre> </blockquote> In principle, +1. However, the tools I have spit out percentages. One % is roughly 10 MB, so the math is fairly easy.<br> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <pre wrap=3D"" class=3D"moz-quote-pre">I personally am not intereste= d in debugging anything older than 2.53.7 (or 6, but there is no reason to use 6 if you are compiling).</pre> </blockquote> You do you, but the trouble is that distro repositories are a bit slow to catch up. Part of the whole update exercise was so I could finally return to Unison from the distro repos and not have to build my own any more. Especially since building for non-Intel platforms is quite a hassle =E2=80=93 I tried to get that on CI some time ago, b= ut my attempts stalled as setting up a cross-compilation environment proved challenging.<br> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <pre wrap=3D"" class=3D"moz-quote-pre">So it sort of sounds like mem= ory is allocated proportional to the size of the transferred file, and not freed. And, that other memory uesd for scan/etc. is reused.</pre> </blockquote> It is roughly proportional to the size of the transferred file, although 16000M uses more in comparison than 12800M does (roughly 55M vs. 34M). Also, archive file size seems to matter.<span style=3D"white-space: pre-wrap"> </span> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <blockquote type=3D"cite"> <pre wrap=3D"" class=3D"moz-quote-pre">Looking at the docs, what c= omes to mind is: - copyprog, copyprogthreshold (use external program <copyprog> for copying files larger than <copyprogthreshold> kB) </pre> </blockquote> <pre wrap=3D"" class=3D"moz-quote-pre"> That is about to be deleted.</pre> </blockquote> That would be too bad, because that seems to be a valid workaround. I added `copythreshold =3D 163840` (1% of 16G, roughly where the issues started) to my profile and was able to sync.<br> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <blockquote type=3D"cite"> <pre wrap=3D"" class=3D"moz-quote-pre">Or is there a way to tell U= nison to stop being smart and just copy the damn thing (which is presumably less memory-hungry) if a file is larger than a certain size? </pre> </blockquote> <pre wrap=3D"" class=3D"moz-quote-pre"> I don't think so, but really that should not be necessary. If there is code that uses memory when it shouldn't, we should find that and fix it.</= pre> </blockquote> For =E2=80=9Ccut the smartness above file size X=E2=80=9D, `copythresh= old` (not `copyprogthreshold`, that was a typo) seems to do the trick. Please keep that feature until memory efficiency is improved, unless there is an alternative. </body> </html> --------------maZJLDxD5eeF0XkGLgz7y4bn-- --===============8937841426179453067== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Unison-hackers mailing list [email protected] https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers --===============8937841426179453067==--