Re: Kernel Edgeplay

On Fri, 2 Jan 2009 19:56:30 -0500
Bill Yerazunis <[email protected]> wrote:
> From: Chris Babcock <[email protected]>
> >    The child starts the telnet process and calls cto.sh
> > asynchronously with the PID whenever it polls the telnet process for
> > a response. When telnet returns data, the child kills the current
> > cto.sh process.
> 
> Ahhh... the old "how do you know if it's hung" problem.  A more 
> up-close-and-personal version of the halting problem.
> 
> And, since the child processes get re-used rather than respawned,
> you don't have the sucker processes as well.  

Just for clarity... The issue is monitoring the telnet process. This is
a dialog with the SMTP server. I'd like to be able to queue up all its
data and fire it off, but the dialog breaks if the server gets a DATA
command before it has said OK to the sender and recipient.

> >    The obvious problem here is that every child spawned off the main
> > loop is using 6 or 9 processes at any time. When you add in the
> > processes from the mail server that's 400-500 concurrent processes
> > from this application... assuming that I configure the MTA to
> > enforce the same limits on local connections that it applies to
> > outside servers.
> 
> Well, are you blowing the gaskets on your kernel?  If so, then 
> we can fix it.  If not, then the second rule of coding applies 
> (don't fix bugs that aren't there).

It probably won't break my setup because I'm running Postfix with some
sane defaults on both my machines... So it will cap out at 500
processes regardless of the size of the lists. Those machines have the
capacity to spare.

The problem comes when I release this into the wild...

The Linux process limit prior to the 2.6 kernel is ~4000 process, then
there's system overhead as well as whatever else is running on that
box. When I checked to see how many process might be available, I was
finding anecdotal evidence of slackware distros with 1500 threads
available after system overhead and a Fedora install that only had 255
available. (There was a bit of code available for testing 

> HOWEVER - one thing I'd consider is whether keeping the
> same children around for multiple uses is actually a win.
> Sometimes it's better to launch each one as a "fire and 
> forget" ASYNC process, which dies when it dies.  Then
> you don't have to keep track of all of your cto timeout 
> processes (which you keep relaunching anyway).

The looping demon doesn't worry about the children unless they call.

The daemon children read the template file from a disk file (for a mail
merge) and initialize the SMTP dialog. There's some overhead involved
in the disk process and network access that would probably be worth
reusing even if the application didn't depend on a dialog. That said,
if the SMTP dialog doesn't keep up with the mail merge that might be
the time to finish assembling the the message and dump it to the
sendmail binary.

> > Also, if there's
> > room to tweak syscall to do this housekeeping since you're
> > already wrapping the external process when you do an asynchronous
> > call that would be Godiva.
> 
> Yes, it might be possible.
> 
> We could integrate a timeout / self-destruct into the pusher process.
> (for those of you who aren't into the code, each "syscall"
> process-forking actually creates two processes- the one the
> programmer asked for, and a second one whose job it is to push all
> the data in the "send to" block out into the actual input pipe of the
> syscalled process.  That second process *might* be able to handle a
> timeout with proper coding as well.  ASYNC processes also launch a
> third "sucker" process to assure that they can run to completion.  

...which is exactly why I mentioned it a couple months ahead of
implementing the design. ;-)

> The downside of this is it means that we have to depend on some 
> slightly esoteric programming to have the pusher process both push
> data and not get hung waiting for the data to get sucked in.

My C-fu is negligible. Shameful, I know, but I'll get there eventually.

> Possibly a better idea would be to spawn a third "kira" process (name
> stolen from the Japanese anime "Death Note", if you must know) that
> would, at the appointed time, kill both the syscalled fork and the
> pusher process, then kill itself.  That would mean you'd have three
> processes rather than five as you have now ( fork, fork's pusher, 
> kill script, kill script pusher, kill script sucker).  
> 
> Hmmm... 

Which is exactly what I'm doing, but because I have to start an
additional asyncronous process from the CRM script it's 3 additional
processes instead of one.

> I think I'd rather just switch to spawning a one-shot process
> for each incoming email, with a cto process that kills it off 
> automatically if it runs away.  That keeps it all very simple and
> straightforward.

That's the way the concept code works, which takes a good 30
seconds to process 900 messages without the mail merge capability.
Well, I'll get a real time on that next Newsletter.

> If there's some precomputation that the process carries with it
> that you don't want to re-execute for each child, do that
> precomputation set-up in the spawning parent first, and then
> have it do the spawns.

If it was computation, I wouldn't worry. It's network access. It's
worth paralellizing, but for maximum throughput you want to keep each
open connection and avoid repeating the opening handshake.

> Keeping things simple can be a powerful technique.

I'm going to put that on a bumpersticker... and do another one that
says, "I'm a friend of Bill Y." :-)

I'm always getting better, but never getting well.

Chris

------------------------------------------------------------------------------

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQEcBAEBAgAGBQJJXs1cAAoJEKEkGnV5bbn/KI0IAJ2BsuUTI/ob9RQWbKwOEjKp
Y8dmdF6vFOkJIl2EZDmH8H0W/v5ru74CCF5FL8DWfD1oOa/jzE8jjkFk4ir0xvMc
N2k0SCYFgikdChexH3QtTPpzeZnJCKgucxPz7kgMfjD69e8OxLGYoPwCa3t7DqqN
6lVmrJ5YzAssd94J3K0hyelC0UxKHiA9JERo9+CB4hUZT/0mygURYenemAan9PxC
Of+C6Hztrf+nVhS9wZeZLMvFyLwrhHU+iMOm4uJXXkNFPbfxN+H21wCQAkZdm0Zi
3uBUGYpQPci6PrXhGhewHLroQc68s+g6ukiyugVu15GCMpFZSEL+kgUEOww+2YM=
=FCVh
-----END PGP SIGNATURE-----