Re: Kernel Edgeplay

   From: Chris Babcock <[email protected]>

   I'm building a little something something that spawns multiple
   asynchronous processes, each of which maintains an independent dialog
   with an external process.=20

   The main thread is a looping daemon that checks a status pipe for
   messages from daemons it has spawned, pushes data out to any waiting
   demons and spawns a new daemon child if there aren't any waiting for
   data.

   Each of the daemon children grabs data off an input pipe, does a merge
   with a template supplied at creation, runs an SMTP conversation via a
   telnet process to the local mail server and lets the parent know when
   its ready for another record. Even with pipelining to a local mail
   server the telnet process has to be watched:

   #!/bin/sh
   # cto.sh - Configurable Time Out
   # $1 - PID to timeout
   # $2 - Seconds to timeout
   sleep $2s
   kill $1

   The child starts the telnet process and calls cto.sh asynchronously with
   the PID whenever it polls the telnet process for a response. When telnet
   returns data, the child kills the current cto.sh process.

Ahhh... the old "how do you know if it's hung" problem.  A more 
up-close-and-personal version of the halting problem.

And, since the child processes get re-used rather than respawned,
you don't have the sucker processes as well.  


   The obvious problem here is that every child spawned off the main loop
   is using 6 or 9 processes at any time. When you add in the processes
   from the mail server that's 400-500 concurrent processes from this
   application... assuming that I configure the MTA to enforce the same
   limits on local connections that it applies to outside servers.

Well, are you blowing the gaskets on your kernel?  If so, then 
we can fix it.  If not, then the second rule of coding applies 
(don't fix bugs that aren't there).

   I have a little while to work on this, because the first version is
   just going to dump the merged documents to the sendmail command, but I
   do want to know if there's a better idiom for watching a process that
   I'm missing coming at the problem like a shell script. 

Not that I know of.  But I get surprised all the time.

HOWEVER - one thing I'd consider is whether keeping the
same children around for multiple uses is actually a win.
Sometimes it's better to launch each one as a "fire and 
forget" ASYNC process, which dies when it dies.  Then
you don't have to keep track of all of your cto timeout 
processes (which you keep relaunching anyway).

    Also, if there's
   room to tweak syscall to do this housekeeping since you're already
   wrapping the external process when you do an asynchronous call that
   would be Godiva.

Yes, it might be possible.

We could integrate a timeout / self-destruct into the pusher process.
(for those of you who aren't into the code, each "syscall" process-forking
actually creates two processes- the one the programmer asked for, and
a second one whose job it is to push all the data in the "send to" block
out into the actual input pipe of the syscalled process.  That
second process *might* be able to handle a timeout with proper coding
as well.  ASYNC processes also launch a third "sucker" process
to assure that they can run to completion.  

The downside of this is it means that we have to depend on some 
slightly esoteric programming to have the pusher process both push
data and not get hung waiting for the data to get sucked in.

Possibly a better idea would be to spawn a third "kira" process (name
stolen from the Japanese anime "Death Note", if you must know) that
would, at the appointed time, kill both the syscalled fork and the
pusher process, then kill itself.  That would mean you'd have three
processes rather than five as you have now ( fork, fork's pusher, 
kill script, kill script pusher, kill script sucker).  

Hmmm... 

I think I'd rather just switch to spawning a one-shot process
for each incoming email, with a cto process that kills it off 
automatically if it runs away.  That keeps it all very simple and
straightforward.

If there's some precomputation that the process carries with it
that you don't want to re-execute for each child, do that
precomputation set-up in the spawning parent first, and then
have it do the spawns.

Keeping things simple can be a powerful technique.

     - Bill Yerazunis

------------------------------------------------------------------------------