Asunto: Re: Slow fork bomb message in latest version of POE

Hi again,

 sorry! from the code below, there's a mistake as DIE signal is linked to _=
sigterm, while is really pointing to _sigdie; just to clarify it before som=
eone says "it can't work, you are pointing to the wrong method!" :D
=20
 Alberto

=20
---- Activado lun, 24 mar 2014 16:44:36 +0100 albertocurro<albertocurro@zoh=
o.com> escribi=C3=B3 ----=20

 > Hi Rocco,=20
 > =20
 >  many thanks for your quick answer! Unfortunately, the provided solution=
 only works partially. I still have some cases where the "fork bomb" messag=
e is here with us :(=20
 > =20
 >   One of the cases is this one: under some configuration, an instance of=
 nginx is started, so our product writes the configuration file and starts =
the Nginx instance pointing to that configuration file. BUT, if the configu=
ration file could not be written (directory does not exist, etc), then the =
error raises, and I've not found any way to handle it:=20
 > =20
 > DEBUG - Created nginx temporary directory /opt/tmp/pull/instance1=20
 > DEBUG - Created nginx configuration directory /opt/etc/pull/instance1=20
 > DEBUG - Created nginx log directory /opt/log/pull/instance1=20
 > DEBUG - creating nginx configfile for instance 1 in /opt/etc/pull/instan=
ce1=20
 > =3D=3D=3D 13991 =3D=3D=3D !!! Kernel has 1 child process(es).=20
 > =3D=3D=3D 13991 =3D=3D=3D !!! At least one child process is still runnin=
g when POE::Kernel->run() is ready to return.=20
 > =3D=3D=3D 13991 =3D=3D=3D !!! Be sure to use sig_child() to reap child p=
rocesses.=20
 > =3D=3D=3D 13991 =3D=3D=3D !!! In extreme cases, failure to reap child pr=
ocesses has=20
 > =3D=3D=3D 13991 =3D=3D=3D !!! resulted in a slow 'fork bomb' that has ha=
lted systems.=20
 > Could not open file: No such file or directory=20
 > =20
 >  I've added a DIE handler in the main session to try to handle this:=20
 > =20
 >  $sig_session =3D POE::Session->create(=20
 >     inline_states =3D> {=20
 >         _start =3D> sub {=20
 >             $_[HEAP]{RELOADED} =3D 0;=20
 >             $_[KERNEL]->sig(TERM =3D> '_sigterm');=20
 >             $_[KERNEL]->sig(INT =3D> '_sigterm');=20
 >             $_[KERNEL]->sig(DIE =3D> '_sigterm');=20
 >             $_[KERNEL]->sig(nginx_reload =3D> '_sig_nginx_reload');=20
 >             $_[KERNEL]->alias_set('sighandler');=20
 >         },=20
 >         _sigdie =3D> sub {=20
 >             print "Handling exception, calling stop";=20
 >             POE::Kernel->call($sig_session, '_stop');=20
 >         },=20
 >         _stop =3D> sub {=20
 >             # Reap any existing pid (# 1825119)=20
 >             print "Handling stop";=20
 >             POE::Kernel->sig_child();=20
 >             use POSIX ":sys_wait_h";=20
 >             1 while waitpid(WNOHANG, -1) > 0;=20
 > =20
 >             # Clear signal handlers...=20
 >             $_[KERNEL]->sig('TERM');=20
 > =20
 > But, as said above, it's not working. Checking POE's code, I can see the=
 message lines are generated in Resources/Signals.pm, under _data_sig_final=
ize() method (where POE is already doing the same you recommended me, waiti=
ng for the pid).=20
 > =20
 > But _data_sig_finalize() method is called in Kernel.pm just after unregi=
stered all the signals (Kernel.pm =3D> _finalize_kernel):=20
 > =20
 >  my $self =3D shift;=20
 > =20
 >   # Disable signal watching since there's now no place for them to go.=
=20
 >   foreach ($self->_data_sig_get_safe_signals()) {=20
 >     $self->loop_ignore_signal($_);=20
 >   }=20
 > =20
 >   # Remove the kernel session's signal watcher.=20
 >   $self->_data_sig_remove($self->ID, "IDLE");=20
 > =20
 >   # The main loop is done, no matter which event library ran it.=20
 >   # sig before loop so that it clears the signal_pipe file handler=20
 >   $self->_data_sig_finalize();=20
 >   $self->loop_finalize();=20
 > =20
 >  Once here, none of my signal handlers in the main session instance woul=
d work, as the signals have been unregistered. On an exception (die) while =
POE::Kernel->run(), how could I handle it then??=20
 > =20
 >  Thanks a lot=20
 >  Alberto=20
 > =20
 > =20
 > =20
 > =20
 > ---- Activado lun, 24 mar 2014 13:45:45 +0100 Rocco Caputo  escribi=C3=
=B3 ---- =20
 > =20
 > >Hi, Alberto. =20
 > > =20
 > >At program end time, POE runs a quick waitpid() check for child process=
es that may have leaked. This check was added after a bug report where POE =
locked up a server after several days of running. It turned out to be the r=
eporter's application, but it was hard to debug. =20
 > > =20
 > >Your program seems to have created two processes that it didn't reap: P=
IDs 5373 and 5374. The ideal solution is to reap those processes before exi=
ting. Your program can do this using POE::Kernel's sig_child() method. =20
 > > =20
 > >In some cases, a third-party library will create processes and not prop=
erly clean them up. It can be impossible to solve this case without modifyi=
ng other people's code. =20
 > > =20
 > >If you just want to ignore the problem, this might do the trick. Put th=
ese lines in your last _stop handler. They should reap the processes you've=
 leaked before POE's check: =20
 > > =20
 > >use POSIX ":sys_wait_h"; =20
 > >1 while waitpid(WNOHANG, -1) > 0; =20
 > > =20
 > >It's a bit of a pain, but I think it's better to explicitly ignore the =
problem than for it to go unnoticed by default. =20
 > > =20
 > >Please let me know whether that resolves your problem. It may not. For =
example, the processes may still be open until an object is destroyed at gl=
obal destruction time. =20
 > > =20
 > >-- =20
 > >Rocco Caputo  =20
 > > =20
 > >On Mar 24, 2014, at 05:46, albertocurro  wrote: =20
 > > =20
 > >> Guys, =20
 > >> =20
 > >> We have a product developed using POE as a base framework, with some =
other tool libraries as log4perl; basically is a forward proxy, composed of=
 several modules, each one of them comprising a POE::Session; all of them s=
hare an internal queue of tasks to be performed. Each module performs sever=
al tasks on initialization, and if anything goes wrong, croak() is called t=
o stop the service -> this is considered ok, since croak() is only called d=
uring initialization, when validation is being performed. =20
 > >> =20
 > >> The product is stable and works really fine, but recently I updated P=
OE to the latest version, and since then we can see this message in the log=
s: =20
 > >> =20
 > >> registering pdu failed: 263! =20
 > >> =3D=3D=3D 5267 =3D=3D=3D 5 -> on_handle (from Handler/StoreRemote.pm =
at 87) =20
 > >> =3D=3D=3D 5267 =3D=3D=3D 5 -> on_retry (from Handler/StoreRemote.pm a=
t 141) =20
 > >> =3D=3D=3D 5267 =3D=3D=3D 9 -> on_handle (from Handler/StoreRemote.pm =
at 87) =20
 > >> =3D=3D=3D 5267 =3D=3D=3D 9 -> on_retry (from Handler/StoreRemote.pm a=
t 141) =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Kernel has child processes. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Stopped child process (PID 5373) reaped =
when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Stopped child process (PID 5374) reaped =
when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! At least one child process is still runn=
ing when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Be sure to use sig_child() to reap child=
 processes. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! In extreme cases, failure to reap child =
processes has =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! resulted in a slow 'fork bomb' that has =
halted systems. =20
 > >> mkdir /mnt/nfs99: Permission denied at Handler/Store.pm line 147 =20
 > >> =20
 > >> first lines and last line above are the errors itself, but this part =
is new since the upgrading: =20
 > >> =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Kernel has child processes. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Stopped child process (PID 5373) reaped =
when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Stopped child process (PID 5374) reaped =
when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! At least one child process is still runn=
ing when POE::Kernel->run() is ready to return. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! Be sure to use sig_child() to reap child=
 processes. =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! In extreme cases, failure to reap child =
processes has =20
 > >> =3D=3D=3D 5267 =3D=3D=3D !!! resulted in a slow 'fork bomb' that has =
halted systems. =20
 > >> =20
 > >> I can see it everytime the service is stopped because of an unhandled=
 condition, even when POE's event loop has been already running for ours. I=
t was not visible before, and I can't get rid of it in any way. I've tried =
different ways to avoid it with no luck. =20
 > >> =20
 > >> Any advice or alternative approach on this? =20
 > >> =20
 > >> Many thanks =20
 > >> Alberto =20
 > >> =20
 > > =20
 > >=20
 > =20
 >=20