Re: daemontools respin?

Wayne Marshall <[email protected]> Thu, 10 Jun 2010 13:34:07 +0000
Newsgroups gmane.comp.sysutils.bgware
Organization guinix international
Message-ID <[email protected]>
Hi,

I noticed your comments on the bgware mailing list and found
them very interesting!
> 
> For some years now I've hacked on a alternative - but still
> compatible - daemontools implementation.
> 
> Unfortunatly I'm not in a position to share the sources since
> my employer "owns" this piece of work - but I'm allowed to
> share my ideas ;-)
>

I would like to see your work one day!

In the meantime, I wondered if you would be interested in
looking at "perp":

  http://b0llix.net/perp/

It is a persistent process supervisor born from the same
conceptual roots as daemontools.  I think your feedback would
help us improve it a lot!

> I've also had the need to supervisors using service
> definitions from a readonly filesystem.
>

This is done, and makes perp fully FHS compatible.
 
> Also another feature that I've been adding to my own
> supervisor is start stop/crash notifications.
>

This is done.  Each runscript can respond to a "reset" event
triggered every time the supervised process exits.  The
runscript is provided with the exit code/signal number, just as
you describe below.

> Since the supervisor can obtain the exitcode and signal-code
> of its supervised child and combine it with supervisors own
> service-runstate info - it can easily conclude if the service
> have crashed or was gracefully shutdown.
> 
> All this info: timestamp, start/stop/crash, up-/down-time,
> exitcode, crash/signalcode etc is sent as cmdline args to a
> "notify" program - if present in the current service dir.
> 
> A global (unified) "notify" program is called if no local
> "notify" program exists. Notifications can be turned off per
> service by touching ".no_notify" in the current service dir.
> 
> As most of you may have guessed by now - this notification
> feature is used to do reporting to a centralized monitoring
> system.
>

Your notification framework is most interesting to me.  Most of
this functionality is within perp already, but probably not so
elegant.  For example, the perpd scanner is usually started with
its own real stdin logger (as opposed to "readproctitle"), which
in turn captures stderr logging/messages from the perpetrate
supervisors.  This could be readily adapted to a more
purpose-specific notification monitoring system.

> My supervisor also implements a per service configurable 
> crash-restart-policy. If a service crashes a given number of
> times within a given timeframe the supervisor will give up and
> permanently "down" the service in question. The previously
> mentioned notification feature is also used here to inform
> about the permanent state change (down).
>

I like this idea, too, and think it would be worthwhile.
Although the perp "reset" event is currently able to provide
such a feature, a built-in facility would be advantageous.

> 
> BTW: have you looked at Gerrit Pape's runit ?
> http://smarden.org/runit/
> 
> He did a smart thing compared to daemontools - his supervisor
> (runsv) is able to supervise both the service and the logger
> within one process. This reduces the number of supervisors and
> thus gives a simpler process tree - as far as I can see at no
> cost.
>

BTW: Each perpetrate supervisor process also monitors a
service/logger pair.  Future versions of perp may possibly
coalesce the perpd/perpetrate daemons further.

Thank you again for sharing your ideas.  Please let me know if
you have any other suggestions/questions for perp!

Best regards,

Wayne