Re: Netplex issue with VServer PID namespace
Thomas Calderon <[email protected]> Mon, 1 Jun 2015 18:10:19 +0200
| Newsgroups | gmane.comp.lang.ocaml.lib.net.devel |
|---|---|
| Message-ID | <CA+1ewKZx4UQwxL0oHuVOnSjGVD+qLE-9gaNHJ3S4jrZMsYac_g@mail.gmail.com> |
--===============5247919557006731244== Content-Type: multipart/alternative; boundary=047d7bd7524424eb0e0517771149 --047d7bd7524424eb0e0517771149 Content-Type: text/plain; charset=UTF-8 OK, I am investigating further, I have some more hints that it might be related to the Linux PID namespace implementation. I am trying to reproduce the issue outside OCaml/Ocamlnet. I will keep you posted. Thanks On Mon, Jun 1, 2015 at 6:05 PM, Gerd Stolpmann <[email protected]> wrote: > Just a guess: There is a Unix.getpid call in Netplex_mp. This call > returns the PID in the new PID space, and this PID is different from the > PID returned by fork() (look into the sources of Netplex_mp, where both > is done). I do not remember for what the PID is used, but it is probably > a key in a management data structure. Then (and this is the unverified > part of my guess), some lookup fails that normally cannot fail, and the > controller gets confused. > > Note that you do not see the getpid() calls in the strace because it is > not a real syscall (afaik the kernel just writes the PID into some > memory location after fork/clone, where glibc expects it). > > I don't know whether this is really the problem, but if so, the fix is > probably not trivial. The controller would have to tell the container > via the control socket what the PID from the view of the controller is; > or via the pipe that is used inside Netplex_mp for synchronization. > > The restart_syscall thing you observed is just a poll waiting for an > event. strace just doesn't print it cleanly. > > Gerd > > Am Montag, den 01.06.2015, 13:03 +0200 schrieb Thomas Calderon: > > Hello Gerd, > > > > > > I do not think I reach the limit of the maximum number of processes > > since I have at most 3 defunct processes. > > I would also be likely to see some other message indicating I reached > > this limit (GrSecurity would leave a trace). > > > > > > When attaching to stalled instances, the controller and worker > > instances (except one) are blocked on : > > restart_syscall(<... resuming interrupted call ...> > > > > > > As mentioned, one of the worker process is blocked on: > > futex(0x...., FUTEX_WAIT_PRIVATE, 2, NULL > > > > > > You will find the strace -f as an attachment. > > > > > > > > Cheers, > > > > > > Thomas > > > > On Mon, Jun 1, 2015 at 12:13 PM, Gerd Stolpmann > > <[email protected]> wrote: > > Am Montag, den 01.06.2015, 11:06 +0200 schrieb Thomas > > Calderon: > > > Hi, > > > > > > > > > We are observing an issue when using OCamlnet netplex in > > combination > > > with VServer PID namespaces. > > > We are using Netplex in the multi-process mode. > > > > > > > > > Here is what we are doing: > > > - start our netplex controller > > > - use the post_add_hook to enter a new PID namespace > > > - use dynamic workload manager to spawn child workers > > > - configured with conn_limit=1 > > > > > > > > > - launch a loop of client connections > > > - this spawns a new worker process for each connection > > > > > > > > > After several successful connections of the loop, clients > > cannot > > > connect anymore. > > > We observe some worker processes in a defunct/zombie state. > > > The controller and running worker processes seem deadlocked > > in some > > > condition. > > > > > > > > > When we do not use the post_add_hook to enter a new PID > > namespace, the > > > problem cannot be triggered anymore. > > > > > > > > > Do you have any hint on this? > > > > > > The controller runs of course waitpid() on the terminated > > processes to > > un-zombie these, and obviously this does not work. I guess you > > reach > > then the maximum number of processes after some time. > > > > You say "vServer" but there are several such technologies > > (Linux > > containers, Virtuozzo, maybe some derived products). I also > > don't know > > much about this corner of the OS. > > > > What would definitely help is an strace -f of the server. > > > > Gerd > > > > > > > > > > > > > > > Many thanks. > > > > > > > > > Thomas > > > > -- > > ------------------------------------------------------------ > > Gerd Stolpmann, Darmstadt, Germany [email protected] > > My OCaml site: http://www.camlcity.org > > Contact details: http://www.camlcity.org/contact.html > > Company homepage: http://www.gerd-stolpmann.de > > ------------------------------------------------------------ > > > > > > > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany [email protected] > My OCaml site: http://www.camlcity.org > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------ > > --047d7bd7524424eb0e0517771149 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">OK,<div><br></div><div>I am investigating further, I have = some more hints that it might be related to the Linux PID namespace impleme= ntation.</div><div>I am trying to reproduce the issue outside OCaml/Ocamlne= t.</div><div><br></div><div>I will keep you posted.</div><div><br></div><di= v>Thanks</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quot= e">On Mon, Jun 1, 2015 at 6:05 PM, Gerd Stolpmann <span dir=3D"ltr"><<a = href=3D"mailto:[email protected]" target=3D"_blank">info@gerd-stolpman= n.de</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just a guess: = There is a Unix.getpid call in Netplex_mp. This call<br> returns the PID in the new PID space, and this PID is different from the<br= > PID returned by fork() (look into the sources of Netplex_mp, where both<br> is done). I do not remember for what the PID is used, but it is probably<br= > a key in a management data structure. Then (and this is the unverified<br> part of my guess), some lookup fails that normally cannot fail, and the<br> controller gets confused.<br> <br> Note that you do not see the getpid() calls in the strace because it is<br> not a real syscall (afaik the kernel just writes the PID into some<br> memory location after fork/clone, where glibc expects it).<br> <br> I don't know whether this is really the problem, but if so, the fix is<= br> probably not trivial. The controller would have to tell the container<br> via the control socket what the PID from the view of the controller is;<br> or via the pipe that is used inside Netplex_mp for synchronization.<br> <br> The restart_syscall thing you observed is just a poll waiting for an<br> event. strace just doesn't print it cleanly.<br> <span class=3D"HOEnZb"><font color=3D"#888888"><br> Gerd<br> </font></span><div class=3D"HOEnZb"><div class=3D"h5"><br> Am Montag, den 01.06.2015, 13:03 +0200 schrieb Thomas Calderon:<br> > Hello Gerd,<br> ><br> ><br> > I do not think I reach the limit of the maximum number of processes<br= > > since I have at most 3 defunct processes.<br> > I would also be likely to see some other message indicating I reached<= br> > this limit (GrSecurity would leave a trace).<br> ><br> ><br> > When attaching to stalled instances, the controller and worker<br> > instances (except one) are blocked on :<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0restart_syscall(<... resuming inte= rrupted call ...><br> ><br> ><br> > As mentioned, one of the worker process is blocked on:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0futex(0x...., FUTEX_WAIT_PRIVATE, 2, = NULL<br> ><br> ><br> > You will find the strace -f as an attachment.<br> ><br> ><br> ><br> > Cheers,<br> ><br> ><br> > Thomas<br> ><br> > On Mon, Jun 1, 2015 at 12:13 PM, Gerd Stolpmann<br> > <<a href=3D"mailto:[email protected]">[email protected]</= a>> wrote:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Am Montag, den 01.06.2015, 11:06 +020= 0 schrieb Thomas<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Calderon:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Hi,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We are observing an issue when u= sing OCamlnet netplex in<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0combination<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> with VServer PID namespaces.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We are using Netplex in the mult= i-process mode.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Here is what we are doing:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- start our netplex = controller<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- use the pos= t_add_hook to enter a new PID namespace<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- use dynamic worklo= ad manager to spawn child workers<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- configured = with conn_limit=3D1<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0- launch a loop of c= lient connections<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0- this spawns= a new worker process for each connection<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> After several successful connect= ions of the loop, clients<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cannot<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> connect anymore.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> We observe some worker processes= in a defunct/zombie state.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> The controller and running worke= r processes seem deadlocked<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0in some<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> condition.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> When we do not use the post_add_= hook to enter a new PID<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0namespace, the<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> problem cannot be triggered anym= ore.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Do you have any hint on this?<br= > ><br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The controller runs of course waitpid= () on the terminated<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0processes to<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0un-zombie these, and obviously this d= oes not work. I guess you<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0reach<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0then the maximum number of processes = after some time.<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0You say "vServer" but there= are several such technologies<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(Linux<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0containers, Virtuozzo, maybe some der= ived products). I also<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0don't know<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0much about this corner of the OS.<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0What would definitely help is an stra= ce -f of the server.<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Gerd<br> ><br> ><br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Many thanks.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Thomas<br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-------------------------------------= -----------------------<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Gerd Stolpmann, Darmstadt, Germany=C2= =A0 =C2=A0 <a href=3D"mailto:[email protected]">[email protected]= </a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0My OCaml site:=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 <a href=3D"http://www.camlcity.org" target=3D"_blank">http://= www.camlcity.org</a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Contact details:=C2=A0 =C2=A0 =C2=A0 = =C2=A0 <a href=3D"http://www.camlcity.org/contact.html" target=3D"_blank">h= ttp://www.camlcity.org/contact.html</a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Company homepage:=C2=A0 =C2=A0 =C2=A0= =C2=A0<a href=3D"http://www.gerd-stolpmann.de" target=3D"_blank">http://ww= w.gerd-stolpmann.de</a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-------------------------------------= -----------------------<br> ><br> ><br> ><br> <br> --<br> ------------------------------------------------------------<br> Gerd Stolpmann, Darmstadt, Germany=C2=A0 =C2=A0 <a href=3D"mailto:gerd@gerd= -stolpmann.de">[email protected]</a><br> My OCaml site:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.caml= city.org" target=3D"_blank">http://www.camlcity.org</a><br> Contact details:=C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.camlcity.= org/contact.html" target=3D"_blank">http://www.camlcity.org/contact.html</a= ><br> Company homepage:=C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"http://www.gerd-stol= pmann.de" target=3D"_blank">http://www.gerd-stolpmann.de</a><br> ------------------------------------------------------------<br> <br> </div></div></blockquote></div><br></div> --047d7bd7524424eb0e0517771149-- --===============5247919557006731244== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ --===============5247919557006731244== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ocamlnet-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel --===============5247919557006731244==--