Re: linstor-controller 1.28.0 hangs after startup

Gábor Hernádi <[email protected]> Tue, 16 Jul 2024 16:31:37 +0200
Newsgroups gmane.comp.linux.drbd
Message-ID <CAL16tNZHO4D8rJeH3MBSiBMUL1vZ9qqfr1sUbx-ZMVTM-YybHw@mail.gmail.com>
--000000000000baa1ea061d5e345a
Content-Type: text/plain; charset="UTF-8"

Hello,

However, after a short amount of time, the linstor controller just stops
> responding again, with basic commands like "linstor node list" timing
> out after 5 minutes.
>

We were fixing a few deadlock-related issues in the recent LINSTOR release,
but also have found another one (will be included in the next release).
However, if you (or any other reader here) face a hanging controller, you
would help us to capture us a thread-dump of the java process and send it
to me (directly is fine, no need to spam the ML with thread-dumps, since
they scale with CPU count and can easily get >100kb).
To get such a thread dump there are 3 options, whereas the first two might
require a JDK installed instead of just a JRE. The third option works also
with a JRE:
1) jstack <pid>
2) jcmd <pid> Thread.print
3) kill -3 <pid>

The third option will NOT kill the process, but it will only print the
thread dump on stdout, so likely into your journalctl.

-- 
Best regards,
Gabor Hernadi

--000000000000baa1ea061d5e345a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello,<br></div><br><div class=3D"gmail_quote"><block=
quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1=
px solid rgb(204,204,204);padding-left:1ex">
However, after a short amount of time, the linstor controller just stops <b=
r>
responding again, with basic commands like &quot;linstor node list&quot; ti=
ming <br>
out after 5 minutes.<br></blockquote><div><br></div><div>We were fixing a f=
ew deadlock-related issues in the recent LINSTOR release, but also have fou=
nd another one (will be included in the next release). However, if you (or =
any other reader here) face a hanging controller, you would help us to capt=
ure us a thread-dump of the java process and send it to me (directly is fin=
e, no need to spam the ML with thread-dumps, since they scale with CPU coun=
t and can easily get &gt;100kb).</div><div>To get such a thread dump there =
are 3 options, whereas the first two might require a JDK installed instead =
of just a JRE. The third option works also with a JRE:</div><div>1) jstack =
&lt;pid&gt;<br>2) jcmd &lt;pid&gt; Thread.print<br>3) kill -3 &lt;pid&gt;<b=
r></div><div><br></div><div>The third option will NOT kill the process, but=
 it will only print the
 thread dump on stdout, so likely into your journalctl.<br></div><div>=C2=
=A0<br></div></div><span class=3D"gmail_signature_prefix">-- </span><br><di=
v dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"l=
tr"><div>Best regards,<br></div>Gabor Hernadi<br></div></div></div></div></=
div>

--000000000000baa1ea061d5e345a--