Re: help in tracking down 1481 memory leak (with reproduction steps)
DC* <[email protected]> Sun, 07 Oct 2018 20:57:21 -0300
| Newsgroups | gmane.network.freenet.devel |
|---|---|
| Message-ID | <[email protected]> |
------DNO502WTUKZX56I4FMWS2DZV5G05QN
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
Are there any debug/logging/stack trace setting we could enable to see wher=
e it died?
I'm gonna setup an container to try this out=2E
On October 7, 2018 6:23:10 PM GMT-03:00, Arne Babenhauserheide <arne_bab@w=
eb=2Ede> wrote:
>Hi,
>
>The past weeks I=E2=80=99ve been stumped trying to track down a severe me=
mory
>leak which prevents releasing 1481=2E
>
>I have a hard time tracking down where and why exactly it happens,
>therefore I=E2=80=99d be very grateful for your help=2E
>
>The following shows how to reproduce the problem on GNU/Linux: Getting
>Freenet 1481 to crash with an Out-of-Memory error within less than 30
>minutes=2E The gist is: Upload a file=2E
>
>tee freenet-1481-OOM-reproduction=2Esh << EOF
>
>wget
>https://github=2Ecom/freenet/fred/releases/download/build01481/new_instal=
ler_offline_1481=2Ejar
>java -jar new_installer_offline_1481=2Ejar
># click through the setup wizard and the in-browser first-run wizard,
>give Freenet high upload bandwidth (i=2Ee=2E 164kiB/s)
>
># give freenet time to start the FCP server
>sleep 180
>
># prepare a file to upload
>INSERTFILE=3D"$(mktemp /tmp/insert=2Etemp=2EXXXXXXXXXX)"
>head -c 100M < /dev/urandom > "$INSERTFILE"
>IDENT=3Dtestupload"${INSERTFILE##*=2E}"
>
># prepare the command to connect to freenet and upload the file
># connect with HELLO
>TEMPFILE=3D"$(mktemp /tmp/insert=2Etemp=2EXXXXXXXXXX)"
>echo ClientHello > $TEMPFILE
>echo "Name=3DUpload-Test${INSERTFILE##*=2E}" >> $TEMPFILE
>echo ExpectedVersion=3D2 >> $TEMPFILE
>echo End >> $TEMPFILE
>echo >> $TEMPFILE
>
># upload with ClientPut
>echo ClientPut >> $TEMPFILE
>echo "DontCompress=3Dtrue" >> $TEMPFILE
>echo "URI=3DCHK@/testupload" >> $TEMPFILE
>echo "Identifier=3D$IDENT" >> $TEMPFILE
>echo MaxRetries=3D-1 >> $TEMPFILE
>echo UploadFrom=3Ddirect >> $TEMPFILE
>echo DataLength=3D$(ls -l $INSERTFILE | cut -d " " -f 5) >> $TEMPFILE
>echo Persistence=3Dforever >> $TEMPFILE
>echo Global=3Dtrue >> $TEMPFILE
>echo End >> $TEMPFILE
>cat $INSERTFILE >> $TEMPFILE
>
># run the insert
>(cat $TEMPFILE | nc 127=2E0=2E0=2E1 9481) &
>
># watch how long the node lives
>for i in {1=2E=2E100}; do curl
>'http://127=2E0=2E0=2E1:8888/stats/?fproxyAdvancedMode=3D2' 2>/dev/null |=
grep
>-io nodeUptimeSession=2E*'<' | grep -io '[^;]*s<' | grep -io '=2E*s' ; cu=
rl
>'http://127=2E0=2E0=2E1:8888/stats/?fproxyAdvancedMode=3D2' 2>/dev/null |=
grep
>-io '[^>]* java memory=2E*&' | grep -io '[^&]*'; sleep 5; done
>
>EOF
>
>I hope this allows you to reproduce the problem =E2=80=94 and I would be =
very
>happy if you could find and fix the source of the problem! This has
>been
>blocking the release of 1481 for far too long=2E
>
>With 33 peers as target (but only up to 25 actually connected, my
>connection isn=E2=80=99t that fast), this gets Freenet to die with an OOM
>within
>less than 15 minutes (last successful stats site at 14m17s)=2E
>
>Best wishes,
>Arne
>--
>Unpolitisch sein
>hei=C3=9Ft politisch sein
>ohne es zu merken
--=20
Sent from mobile=2E Please excuse my brevity=2E
------DNO502WTUKZX56I4FMWS2DZV5G05QN
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><head></head><body>Are there any debug/logging/stack trace setting we=
could enable to see where it died?<br><br>I'm gonna setup an container to =
try this out=2E<br><br><div class=3D"gmail_quote">On October 7, 2018 6:23:1=
0 PM GMT-03:00, Arne Babenhauserheide <arne_bab@web=2Ede> wrote:<bloc=
kquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0=2E8ex; border-l=
eft: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class=3D"k9mail">Hi,<br><br>The past weeks I=E2=80=99ve been stumped =
trying to track down a severe memory<br>leak which prevents releasing 1481=
=2E<br><br>I have a hard time tracking down where and why exactly it happen=
s,<br>therefore I=E2=80=99d be very grateful for your help=2E<br><br>The fo=
llowing shows how to reproduce the problem on GNU/Linux: Getting<br>Freenet=
1481 to crash with an Out-of-Memory error within less than 30<br>minutes=
=2E The gist is: Upload a file=2E<br><br>tee freenet-1481-OOM-reproduction=
=2Esh << EOF<br><br>wget <a href=3D"https://github=2Ecom/freenet/fred=
/releases/download/build01481/new_installer_offline_1481=2Ejar">https://git=
hub=2Ecom/freenet/fred/releases/download/build01481/new_installer_offline_1=
481=2Ejar</a><br>java -jar new_installer_offline_1481=2Ejar<br># click thro=
ugh the setup wizard and the in-browser first-run wizard, give Freenet high=
upload bandwidth (i=2Ee=2E 164kiB/s)<br><br># give freenet time to start t=
he FCP server<br>sleep 180<br><br># prepare a file to upload<br>INSERTFILE=
=3D"$(mktemp /tmp/insert=2Etemp=2EXXXXXXXXXX)"<br>head -c 100M < /dev/ur=
andom > "$INSERTFILE"<br>IDENT=3Dtestupload"${INSERTFILE##*=2E}"<br><br>=
# prepare the command to connect to freenet and upload the file<br># connec=
t with HELLO<br>TEMPFILE=3D"$(mktemp /tmp/insert=2Etemp=2EXXXXXXXXXX)"<br>e=
cho ClientHello > $TEMPFILE<br>echo "Name=3DUpload-Test${INSERTFILE##*=
=2E}" >> $TEMPFILE<br>echo ExpectedVersion=3D2 >> $TEMPFILE<br>=
echo End >> $TEMPFILE<br>echo >> $TEMPFILE<br><br># upload with=
ClientPut<br>echo ClientPut >> $TEMPFILE<br>echo "DontCompress=3Dtru=
e" >> $TEMPFILE<br>echo "URI=3DCHK@/testupload" >> $TEMPFILE<br=
>echo "Identifier=3D$IDENT" >> $TEMPFILE<br>echo MaxRetries=3D-1 >=
> $TEMPFILE<br>echo UploadFrom=3Ddirect >> $TEMPFILE<br>echo DataL=
ength=3D$(ls -l $INSERTFILE | cut -d " " -f 5) >> $TEMPFILE<br>echo P=
ersistence=3Dforever >> $TEMPFILE<br>echo Global=3Dtrue >> $TEM=
PFILE<br>echo End >> $TEMPFILE<br>cat $INSERTFILE >> $TEMPFILE<=
br><br># run the insert<br>(cat $TEMPFILE | nc 127=2E0=2E0=2E1 9481) &<=
br><br># watch how long the node lives<br>for i in {1=2E=2E100}; do curl 'h=
ttp://127=2E0=2E0=2E1:8888/stats/?fproxyAdvancedMode=3D2' 2>/dev/null | =
grep -io nodeUptimeSession=2E*'<' | grep -io '[^;]*s<' | grep -io '=
=2E*s' ; curl 'http://127=2E0=2E0=2E1:8888/stats/?fproxyAdvancedMode=3D2' 2=
>/dev/null | grep -io '[^>]* java memory=2E*&' | grep -io '[^&=
;]*'; sleep 5; done<br><br>EOF<br><br>I hope this allows you to reproduce t=
he problem =E2=80=94 and I would be very<br>happy if you could find and fix=
the source of the problem! This has been<br>blocking the release of 1481 f=
or far too long=2E<br><br>With 33 peers as target (but only up to 25 actual=
ly connected, my<br>connection isn=E2=80=99t that fast), this gets Freenet =
to die with an OOM within<br>less than 15 minutes (last successful stats si=
te at 14m17s)=2E<br><br>Best wishes,<br>Arne<br>--<br>Unpolitisch sein<br>h=
ei=C3=9Ft politisch sein<br>ohne es zu merken<br></pre></blockquote></div><=
br>-- <br>Sent from mobile=2E Please excuse my brevity=2E</body></html>
------DNO502WTUKZX56I4FMWS2DZV5G05QN--