Re: Netplex issue with VServer PID namespace
Thomas Calderon <[email protected]> Tue, 2 Jun 2015 14:23:21 +0200
| Newsgroups | gmane.comp.lang.ocaml.lib.net.devel |
|---|---|
| Message-ID | <CA+1ewKZcgO5eavAsi4RRvGMCEvCF5r4AKqfGf5CXeYswHE_3dA@mail.gmail.com> |
--===============5962687081846520949==
Content-Type: multipart/alternative; boundary=001a1136e9f44f49e30517880306
--001a1136e9f44f49e30517880306
Content-Type: text/plain; charset=UTF-8
Hi,
After further investigation, I can reproduce it on a standard Linux kernel.
Here's the gdb backtrace when the controller is stuck (seems fork() fails
at low-level):
(gdb) bt
#0 __lll_lock_wait_private () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1 0x00007f8e37e91eeb in _L_lock_13840 () from
/lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f8e37e8ffb8 in __GI___libc_realloc (oldmem=0x15b2260, bytes=574)
at malloc.c:3025
#3 0x00007f8e37e7f2db in _IO_vasprintf (result_ptr=0x7fff5a19b2f0,
format=<optimized out>,
args=args@entry=0x7fff5a19b1c8) at vasprintf.c:84
#4 0x00007f8e37e61657 in ___asprintf (string_ptr=string_ptr@entry
=0x7fff5a19b2f0,
format=format@entry=0x7f8e37f8d830 "%s%s%s:%u: %s%sAssertion `%s'
failed.\n%n") at asprintf.c:35
#5 0x00007f8e37e3cae2 in __assert_fail_base (fmt=0x7f8e37f8d830
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7f8e37f90a38 "({ __typeof (self->tid)
__value; if (sizeof (__value) == 1) asm volatile (\"movb %%fs:%P2,%b0\" :
\"=q\" (__value) : \"0\" (0), \"i\" (__builtin_offsetof (struct pthread,
tid))); else if (sizeof (__value) == "..., file=file@entry=0x7f8e37f90a00
"../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c",
line=line@entry=141, function=function@entry=0x7f8e37f8b38d
<__PRETTY_FUNCTION__.11207> "__libc_fork")
at assert.c:57
#6 0x00007f8e37e3cc32 in __GI___assert_fail (
assertion=0x7f8e37f90a38 "({ __typeof (self->tid) __value; if (sizeof
(__value) == 1) asm volatile (\"movb %%fs:%P2,%b0\" : \"=q\" (__value) :
\"0\" (0), \"i\" (__builtin_offsetof (struct pthread, tid))); else if
(sizeof (__value) == "..., file=0x7f8e37f90a00
"../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c", line=141,
function=0x7f8e37f8b38d <__PRETTY_FUNCTION__.11207> "__libc_fork") at
assert.c:101
#7 0x00007f8e37ece252 in __libc_fork () at
../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c:141
#8 0x000000000062b7a6 in unix_fork ()
#9 0x00000000004b88b4 in camlNetplex_mp__fun_1577 () at netplex_mp.ml:80
#10 0x00000000004d54ef in camlNetplex_controller__fun_3861 () at
netplex_controller.ml:359
#11 0x00000000004d5dd8 in camlNetplex_controller__fun_3830 () at
netplex_controller.ml:265
#12 0x00000000004c8355 in camlNetplex_workload__fun_2015 () at
netplex_workload.ml:332
#13 0x00000000004c8b06 in camlNetplex_workload__fun_1982 () at
netplex_workload.ml:230
#14 0x00000000004d445e in camlNetplex_controller__fun_4158 () at
netplex_controller.ml:665
#15 0x00000000004d339d in camlNetplex_controller__fun_4275 () at
netplex_controller.ml:896
#16 0x000000000050fbdc in camlRpc_server__protect_1582 () at
rpc_server.ml:504
#17 0x000000000050fbdc in camlRpc_server__protect_1582 () at
rpc_server.ml:504
#18 0x00000000005144fa in camlRpc_server__handle_incoming_message_1710 ()
at rpc_server.ml:889
#19 0x00000000005392ba in camlUq_multiplex__anyway_1042 () at
uq_multiplex.ml:20
#20 0x0000000000531abe in camlUq_multiplex__fun_3594 () at
uq_multiplex.ml:464
#21 0x000000000051b33e in camlUnixqueue_pollset__forward_event_to_1567 ()
at unixqueue_pollset.ml:768
#22 0x0000000000517f88 in camlEqueue__fun_1262 () at equeue.ml:166
#23 0x00000000005eaea9 in camlQueue__iter_1048 () at queue.ml:135
#24 0x0000000000518a02 in camlEqueue__run_1070 () at equeue.ml:159
#25 0x000000000051cdc9 in camlUnixqueue_pollset__fun_3391 () at
unixqueue_pollset.ml:999
#26 0x00000000004e54a4 in camlNetplex_main__run_controller_1077 () at
netplex_main.ml:130
#27 0x00000000004e4348 in camlNetplex_main__fun_1295 () at
netplex_main.ml:312
---Type <return> to continue, or q <return> to quit---
#28 0x00000000004e574d in camlNetplex_main__redirect_logger_1094 () at
netplex_main.ml:187
#29 0x00000000004e4a20 in camlNetplex_main__fun_1287 () at
netplex_main.ml:294
#30 0x000000000043c2f4 in camlServer__entry ()
#31 0x00000000004063e9 in caml_program ()
#32 0x0000000000642934 in caml_start_program ()
#33 0x0000000000630a0a in caml_main ()
#34 0x0000000000630a4c in main ()
Reading over the line 141 of glibc's fork.c, it seems this assert fails:
assert (THREAD_GETMEM (self, tid) != ppid);
This is triggered when the worker process PID in the namespace reaches that
of the ancestor on the host.
Cheers,
Thomas
On Mon, Jun 1, 2015 at 6:10 PM, Thomas Calderon <[email protected]>
wrote:
> OK,
>
> I am investigating further, I have some more hints that it might be
> related to the Linux PID namespace implementation.
> I am trying to reproduce the issue outside OCaml/Ocamlnet.
>
> I will keep you posted.
>
> Thanks
>
> On Mon, Jun 1, 2015 at 6:05 PM, Gerd Stolpmann <[email protected]>
> wrote:
>
>> Just a guess: There is a Unix.getpid call in Netplex_mp. This call
>> returns the PID in the new PID space, and this PID is different from the
>> PID returned by fork() (look into the sources of Netplex_mp, where both
>> is done). I do not remember for what the PID is used, but it is probably
>> a key in a management data structure. Then (and this is the unverified
>> part of my guess), some lookup fails that normally cannot fail, and the
>> controller gets confused.
>>
>> Note that you do not see the getpid() calls in the strace because it is
>> not a real syscall (afaik the kernel just writes the PID into some
>> memory location after fork/clone, where glibc expects it).
>>
>> I don't know whether this is really the problem, but if so, the fix is
>> probably not trivial. The controller would have to tell the container
>> via the control socket what the PID from the view of the controller is;
>> or via the pipe that is used inside Netplex_mp for synchronization.
>>
>> The restart_syscall thing you observed is just a poll waiting for an
>> event. strace just doesn't print it cleanly.
>>
>> Gerd
>>
>> Am Montag, den 01.06.2015, 13:03 +0200 schrieb Thomas Calderon:
>> > Hello Gerd,
>> >
>> >
>> > I do not think I reach the limit of the maximum number of processes
>> > since I have at most 3 defunct processes.
>> > I would also be likely to see some other message indicating I reached
>> > this limit (GrSecurity would leave a trace).
>> >
>> >
>> > When attaching to stalled instances, the controller and worker
>> > instances (except one) are blocked on :
>> > restart_syscall(<... resuming interrupted call ...>
>> >
>> >
>> > As mentioned, one of the worker process is blocked on:
>> > futex(0x...., FUTEX_WAIT_PRIVATE, 2, NULL
>> >
>> >
>> > You will find the strace -f as an attachment.
>> >
>> >
>> >
>> > Cheers,
>> >
>> >
>> > Thomas
>> >
>> > On Mon, Jun 1, 2015 at 12:13 PM, Gerd Stolpmann
>> > <[email protected]> wrote:
>> > Am Montag, den 01.06.2015, 11:06 +0200 schrieb Thomas
>> > Calderon:
>> > > Hi,
>> > >
>> > >
>> > > We are observing an issue when using OCamlnet netplex in
>> > combination
>> > > with VServer PID namespaces.
>> > > We are using Netplex in the multi-process mode.
>> > >
>> > >
>> > > Here is what we are doing:
>> > > - start our netplex controller
>> > > - use the post_add_hook to enter a new PID namespace
>> > > - use dynamic workload manager to spawn child workers
>> > > - configured with conn_limit=1
>> > >
>> > >
>> > > - launch a loop of client connections
>> > > - this spawns a new worker process for each connection
>> > >
>> > >
>> > > After several successful connections of the loop, clients
>> > cannot
>> > > connect anymore.
>> > > We observe some worker processes in a defunct/zombie state.
>> > > The controller and running worker processes seem deadlocked
>> > in some
>> > > condition.
>> > >
>> > >
>> > > When we do not use the post_add_hook to enter a new PID
>> > namespace, the
>> > > problem cannot be triggered anymore.
>> > >
>> > >
>> > > Do you have any hint on this?
>> >
>> >
>> > The controller runs of course waitpid() on the terminated
>> > processes to
>> > un-zombie these, and obviously this does not work. I guess you
>> > reach
>> > then the maximum number of processes after some time.
>> >
>> > You say "vServer" but there are several such technologies
>> > (Linux
>> > containers, Virtuozzo, maybe some derived products). I also
>> > don't know
>> > much about this corner of the OS.
>> >
>> > What would definitely help is an strace -f of the server.
>> >
>> > Gerd
>> >
>> >
>> >
>> > >
>> > >
>> > > Many thanks.
>> > >
>> > >
>> > > Thomas
>> >
>> > --
>> > ------------------------------------------------------------
>> > Gerd Stolpmann, Darmstadt, Germany [email protected]
>> > My OCaml site: http://www.camlcity.org
>> > Contact details: http://www.camlcity.org/contact.html
>> > Company homepage: http://www.gerd-stolpmann.de
>> > ------------------------------------------------------------
>> >
>> >
>> >
>>
>> --
>> ------------------------------------------------------------
>> Gerd Stolpmann, Darmstadt, Germany [email protected]
>> My OCaml site: http://www.camlcity.org
>> Contact details: http://www.camlcity.org/contact.html
>> Company homepage: http://www.gerd-stolpmann.de
>> ------------------------------------------------------------
>>
>>
>
--001a1136e9f44f49e30517880306
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi,<div><br></div><div>After further investigation, I can =
reproduce it on a standard Linux kernel.</div><div><br></div><div>Here'=
s the gdb backtrace when the controller is stuck (seems fork() fails at low=
-level):</div><div><br></div><blockquote style=3D"margin:0px 0px 0px 40px;b=
order:none;padding:0px"><div><div>(gdb) bt</div></div><div><div>#0 =C2=A0__=
lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevel=
lock.S:95</div></div><div><div>#1 =C2=A00x00007f8e37e91eeb in _L_lock_13840=
() from /lib/x86_64-linux-gnu/libc.so.6</div></div><div><div>#2 =C2=A00x00=
007f8e37e8ffb8 in __GI___libc_realloc (oldmem=3D0x15b2260, bytes=3D574) at =
malloc.c:3025</div></div><div><div>#3 =C2=A00x00007f8e37e7f2db in _IO_vaspr=
intf (result_ptr=3D0x7fff5a19b2f0, format=3D<optimized out>,=C2=A0</d=
iv></div><div><div>=C2=A0 =C2=A0 args=3Dargs@entry=3D0x7fff5a19b1c8) at vas=
printf.c:84</div></div><div><div>#4 =C2=A00x00007f8e37e61657 in ___asprintf=
(string_ptr=3Dstring_ptr@entry=3D0x7fff5a19b2f0,=C2=A0</div></div><div><di=
v>=C2=A0 =C2=A0 format=3Dformat@entry=3D0x7f8e37f8d830 "%s%s%s:%u: %s%=
sAssertion `%s' failed.\n%n") at asprintf.c:35</div></div><div><di=
v>#5 =C2=A00x00007f8e37e3cae2 in __assert_fail_base (fmt=3D0x7f8e37f8d830 &=
quot;%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",=C2=A0</div></div>=
<div><div>=C2=A0 =C2=A0 assertion=3Dassertion@entry=3D0x7f8e37f90a38 "=
({ __typeof (self->tid) __value; if (sizeof (__value) =3D=3D 1) asm vola=
tile (\"movb %%fs:%P2,%b0\" : \"=3Dq\" (__value) : \&qu=
ot;0\" (0), \"i\" (__builtin_offsetof (struct pthread, tid))=
); else if (sizeof (__value) =3D=3D "..., file=3Dfile@entry=3D0x7f8e37=
f90a00 "../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c",=C2=A0<=
/div></div><div><div>=C2=A0 =C2=A0 line=3Dline@entry=3D141, function=3Dfunc=
tion@entry=3D0x7f8e37f8b38d <__PRETTY_FUNCTION__.11207> "__libc_=
fork")</div></div><div><div>=C2=A0 =C2=A0 at assert.c:57</div></div><d=
iv><div>#6 =C2=A00x00007f8e37e3cc32 in __GI___assert_fail (</div></div><div=
><div>=C2=A0 =C2=A0 assertion=3D0x7f8e37f90a38 "({ __typeof (self->=
tid) __value; if (sizeof (__value) =3D=3D 1) asm volatile (\"movb %%fs=
:%P2,%b0\" : \"=3Dq\" (__value) : \"0\" (0), \&quo=
t;i\" (__builtin_offsetof (struct pthread, tid))); else if (sizeof (__=
value) =3D=3D "..., file=3D0x7f8e37f90a00 "../nptl/sysdeps/unix/s=
ysv/linux/x86_64/../fork.c", line=3D141,=C2=A0</div></div><div><div>=
=C2=A0 =C2=A0 function=3D0x7f8e37f8b38d <__PRETTY_FUNCTION__.11207> &=
quot;__libc_fork") at assert.c:101</div></div><div><div>#7 =C2=A00x000=
07f8e37ece252 in __libc_fork () at ../nptl/sysdeps/unix/sysv/linux/x86_64/.=
./fork.c:141</div></div><div><div>#8 =C2=A00x000000000062b7a6 in unix_fork =
()</div></div><div><div>#9 =C2=A00x00000000004b88b4 in camlNetplex_mp__fun_=
1577 () at <a href=3D"http://netplex_mp.ml:80">netplex_mp.ml:80</a></div></=
div><div><div>#10 0x00000000004d54ef in camlNetplex_controller__fun_3861 ()=
at <a href=3D"http://netplex_controller.ml:359">netplex_controller.ml:359<=
/a></div></div><div><div>#11 0x00000000004d5dd8 in camlNetplex_controller__=
fun_3830 () at <a href=3D"http://netplex_controller.ml:265">netplex_control=
ler.ml:265</a></div></div><div><div>#12 0x00000000004c8355 in camlNetplex_w=
orkload__fun_2015 () at <a href=3D"http://netplex_workload.ml:332">netplex_=
workload.ml:332</a></div></div><div><div>#13 0x00000000004c8b06 in camlNetp=
lex_workload__fun_1982 () at <a href=3D"http://netplex_workload.ml:230">net=
plex_workload.ml:230</a></div></div><div><div>#14 0x00000000004d445e in cam=
lNetplex_controller__fun_4158 () at <a href=3D"http://netplex_controller.ml=
:665">netplex_controller.ml:665</a></div></div><div><div>#15 0x00000000004d=
339d in camlNetplex_controller__fun_4275 () at <a href=3D"http://netplex_co=
ntroller.ml:896">netplex_controller.ml:896</a></div></div><div><div>#16 0x0=
00000000050fbdc in camlRpc_server__protect_1582 () at <a href=3D"http://rpc=
_server.ml:504">rpc_server.ml:504</a></div></div><div><div>#17 0x0000000000=
50fbdc in camlRpc_server__protect_1582 () at <a href=3D"http://rpc_server.m=
l:504">rpc_server.ml:504</a></div></div><div><div>#18 0x00000000005144fa in=
camlRpc_server__handle_incoming_message_1710 () at <a href=3D"http://rpc_s=
erver.ml:889">rpc_server.ml:889</a></div></div><div><div>#19 0x000000000053=
92ba in camlUq_multiplex__anyway_1042 () at <a href=3D"http://uq_multiplex.=
ml:20">uq_multiplex.ml:20</a></div></div><div><div>#20 0x0000000000531abe i=
n camlUq_multiplex__fun_3594 () at <a href=3D"http://uq_multiplex.ml:464">u=
q_multiplex.ml:464</a></div></div><div><div>#21 0x000000000051b33e in camlU=
nixqueue_pollset__forward_event_to_1567 () at <a href=3D"http://unixqueue_p=
ollset.ml:768">unixqueue_pollset.ml:768</a></div></div><div><div>#22 0x0000=
000000517f88 in camlEqueue__fun_1262 () at <a href=3D"http://equeue.ml:166"=
>equeue.ml:166</a></div></div><div><div>#23 0x00000000005eaea9 in camlQueue=
__iter_1048 () at <a href=3D"http://queue.ml:135">queue.ml:135</a></div></d=
iv><div><div>#24 0x0000000000518a02 in camlEqueue__run_1070 () at <a href=
=3D"http://equeue.ml:159">equeue.ml:159</a></div></div><div><div>#25 0x0000=
00000051cdc9 in camlUnixqueue_pollset__fun_3391 () at <a href=3D"http://uni=
xqueue_pollset.ml:999">unixqueue_pollset.ml:999</a></div></div><div><div>#2=
6 0x00000000004e54a4 in camlNetplex_main__run_controller_1077 () at <a href=
=3D"http://netplex_main.ml:130">netplex_main.ml:130</a></div></div><div><di=
v>#27 0x00000000004e4348 in camlNetplex_main__fun_1295 () at <a href=3D"htt=
p://netplex_main.ml:312">netplex_main.ml:312</a></div></div><div><div>---Ty=
pe <return> to continue, or q <return> to quit---</div></div><d=
iv><div>#28 0x00000000004e574d in camlNetplex_main__redirect_logger_1094 ()=
at <a href=3D"http://netplex_main.ml:187">netplex_main.ml:187</a></div></d=
iv><div><div>#29 0x00000000004e4a20 in camlNetplex_main__fun_1287 () at <a =
href=3D"http://netplex_main.ml:294">netplex_main.ml:294</a></div></div><div=
><div>#30 0x000000000043c2f4 in camlServer__entry ()</div></div><div><div>#=
31 0x00000000004063e9 in caml_program ()</div></div><div><div>#32 0x0000000=
000642934 in caml_start_program ()</div></div><div><div>#33 0x0000000000630=
a0a in caml_main ()</div></div><div><div>#34 0x0000000000630a4c in main ()<=
/div></div><div><br></div></blockquote>Reading over the line 141 of glibc&#=
39;s fork.c, it seems this assert fails:<blockquote style=3D"margin:0px 0px=
0px 40px;border:none;padding:0px">assert (THREAD_GETMEM (self, tid) !=3D p=
pid);<br><br></blockquote>This is triggered when the worker process PID in =
the namespace reaches that of the ancestor on the host.<div><br></div><div>=
Cheers,</div><div><br></div><div>Thomas</div></div><div class=3D"gmail_extr=
a"><br><div class=3D"gmail_quote">On Mon, Jun 1, 2015 at 6:10 PM, Thomas Ca=
lderon <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" t=
arget=3D"_blank">[email protected]</a>></span> wrote:<br><blockq=
uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div dir=3D"ltr">OK,<div><br></div><div>I am invest=
igating further, I have some more hints that it might be related to the Lin=
ux PID namespace implementation.</div><div>I am trying to reproduce the iss=
ue outside OCaml/Ocamlnet.</div><div><br></div><div>I will keep you posted.=
</div><div><br></div><div>Thanks</div></div><div class=3D"HOEnZb"><div clas=
s=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, =
Jun 1, 2015 at 6:05 PM, Gerd Stolpmann <span dir=3D"ltr"><<a href=3D"mai=
lto:[email protected]" target=3D"_blank">[email protected]</a>>=
;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">Just a guess: There is a =
Unix.getpid call in Netplex_mp. This call<br>
returns the PID in the new PID space, and this PID is different from the<br=
>
PID returned by fork() (look into the sources of Netplex_mp, where both<br>
is done). I do not remember for what the PID is used, but it is probably<br=
>
a key in a management data structure. Then (and this is the unverified<br>
part of my guess), some lookup fails that normally cannot fail, and the<br>
controller gets confused.<br>
<br>
Note that you do not see the getpid() calls in the strace because it is<br>
not a real syscall (afaik the kernel just writes the PID into some<br>
memory location after fork/clone, where glibc expects it).<br>
<br>
I don't know whether this is really the problem, but if so, the fix is<=
br>
probably not trivial. The controller would have to tell the container<br>
via the control socket what the PID from the view of the controller is;<br>
or via the pipe that is used inside Netplex_mp for synchronization.<br>
<br>
The restart_syscall thing you observed is just a poll waiting for an<br>
event. strace just doesn't print it cleanly.<br>
<span><font color=3D"#888888"><br>
Gerd<br>
</font></span><div><div><br>
Am Montag, den 01.06.2015, 13:03 +0200 schrieb Thomas Calderon:<br>
> Hello Gerd,<br>
><br>
><br>
> I do not think I reach the limit of the maximum number of processes<br=
>
> since I have at most 3 defunct processes.<br>
> I would also be likely to see some other message indicating I reached<=
br>
> this limit (GrSecurity would leave a trace).<br>
><br>
><br>
> When attaching to stalled instances, the controller and worker<br>
> instances (except one) are blocked on :<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0restart_syscall(<... resuming inte=
rrupted call ...><br>
><br>
><br>
> As mentioned, one of the worker process is blocked on:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0futex(0x...., FUTEX_WAIT_PRIVATE, 2, =
NULL<br>
><br>
><br>
> You will find the strace -f as an attachment.<br>
><br>
><br>
><br>
> Cheers,<br>
><br>
><br>
> Thomas<br>
><br>
> On Mon, Jun 1, 2015 at 12:13 PM, Gerd Stolpmann<br>
> <<a href=3D"mailto:[email protected]" target=3D"_blank">info@g=
erd-stolpmann.de</a>> wrote:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Am Montag, den 01.06.2015, 11:06 +020=
0 schrieb Thomas<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Calderon:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Hi,<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We are observing an issue when u=
sing OCamlnet netplex in<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0combination<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> with VServer PID namespaces.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We are using Netplex in the mult=
i-process mode.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Here is what we are doing:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- start our netplex =
controller<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- use the pos=
t_add_hook to enter a new PID namespace<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- use dynamic worklo=
ad manager to spawn child workers<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- configured =
with conn_limit=3D1<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- launch a loop of c=
lient connections<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- this spawns=
a new worker process for each connection<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> After several successful connect=
ions of the loop, clients<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cannot<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> connect anymore.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We observe some worker processes=
in a defunct/zombie state.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> The controller and running worke=
r processes seem deadlocked<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0in some<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> condition.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> When we do not use the post_add_=
hook to enter a new PID<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0namespace, the<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> problem cannot be triggered anym=
ore.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Do you have any hint on this?<br=
>
><br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The controller runs of course waitpid=
() on the terminated<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0processes to<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0un-zombie these, and obviously this d=
oes not work. I guess you<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0reach<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0then the maximum number of processes =
after some time.<br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0You say "vServer" but there=
are several such technologies<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(Linux<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0containers, Virtuozzo, maybe some der=
ived products). I also<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0don't know<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0much about this corner of the OS.<br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0What would definitely help is an stra=
ce -f of the server.<br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Gerd<br>
><br>
><br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Many thanks.<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Thomas<br>
><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-------------------------------------=
-----------------------<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Gerd Stolpmann, Darmstadt, Germany=C2=
=A0 =C2=A0 <a href=3D"mailto:[email protected]" target=3D"_blank">gerd=
@gerd-stolpmann.de</a><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0My OCaml site:=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 <a href=3D"http://www.camlcity.org" target=3D"_blank">http://=
www.camlcity.org</a><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Contact details:=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 <a href=3D"http://www.camlcity.org/contact.html" target=3D"_blank">h=
ttp://www.camlcity.org/contact.html</a><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Company homepage:=C2=A0 =C2=A0 =C2=A0=
=C2=A0<a href=3D"http://www.gerd-stolpmann.de" target=3D"_blank">http://ww=
w.gerd-stolpmann.de</a><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-------------------------------------=
-----------------------<br>
><br>
><br>
><br>
<br>
--<br>
------------------------------------------------------------<br>
Gerd Stolpmann, Darmstadt, Germany=C2=A0 =C2=A0 <a href=3D"mailto:gerd@gerd=
-stolpmann.de" target=3D"_blank">[email protected]</a><br>
My OCaml site:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.caml=
city.org" target=3D"_blank">http://www.camlcity.org</a><br>
Contact details:=C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.camlcity.=
org/contact.html" target=3D"_blank">http://www.camlcity.org/contact.html</a=
><br>
Company homepage:=C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"http://www.gerd-stol=
pmann.de" target=3D"_blank">http://www.gerd-stolpmann.de</a><br>
------------------------------------------------------------<br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
--001a1136e9f44f49e30517880306--
--===============5962687081846520949==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
--===============5962687081846520949==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Ocamlnet-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel
--===============5962687081846520949==--