Re: posible bug in daemontools

On Mon, Mar 22, 2010 at 04:22:09PM +1030, Daryl Tester wrote:

> >I don't think the question is the rate at which the daemon is respawning
> >(if that were the case a simple sleep as is found in qsutils.c's nomem()
> >and pausdir() would likely be  sufficient; supervise does actually sleep
> >already at  this point I believe),  but rather the futility  of doing so
> >when  there  is  an  inimical  agent attempting  to  randomly  kill  off
> >processes.
> 
> Bear in mind that the machine that caused the original patch to be produced
> wasn't a Linux box, and the supervised process didn't didn't die due to an
> OOM killer (which that platform didn't have).

If you're referring to the patch Paul Jarc proposed a few years ago in
response to a problem I reported: you're right that it wasn't on a Linux
box (it was on AIX). (That platform certainly *does* have an OOM killer BTW,
and (in hindsight) it has caused problems earlier with supervised services.
But that's not really relevant for this particular discussion.)

Before AIX starts killing processes, the fork function will start
to fail. I reached (by chance) the point were the OOM killer hadn't kicked
in yet, but the fork function was already failing because of the OOM
situation. After the OOM situation ended, fork started functioning again,
but daemontools never retried the failed fork so the service was never
restarted. The service was a "do_something_small ; sleep 60" type of
service, so this meant the service was effectively down.


I think the OOM situation is just the way the problem was discovered, the
problem being that a failed fork is never retried and the service remains
in "want up" state. A failing fork can happen under other cicumstances than
an OOM situation, so I would consider (FWIW) that something that should be
fixed.

-- 
Jurjen Oskam

Savage's Law of Expediency:
        You want it bad, you'll get it bad.