daemontools-0.76, AIX 5.2: problem restarting service

Hi everybody,

On an AIX 5.2 machine, I use daemontools-0.76. Today, because of
an out-of-memory situation, processes were killed by the VMM.
I know that one supervise process was killed, the one supervising
the service /service/enable-printers.

Now, the state of the services is:

/service/clear-errors: down 4462 seconds, normally up, want up
/service/dsmcad: up (pid 766010) 783525 seconds
/service/enable-printers: up (pid 360594) 1081 seconds
/service/nmbd: up (pid 774230) 776558 seconds
/service/qmail-send: up (pid 704574) 775375 seconds
/service/qmail-smtpd: down 1198531 seconds
/service/smbd: up (pid 823454) 776558 seconds
/service/umclient: up (pid 184372) 1198531 seconds
/service/qmail-send/log: up (pid 167974) 1198531 seconds
/service/qmail-smtpd/log: up (pid 278668) 1198531 seconds
/service/umclient/log: up (pid 274568) 1198531 seconds

The output of "ps -ef | grep super[v]ise":

    root 188608 266410   0   Oct 29      -  0:09 supervise clear-errors
    root 204830 266410   0   Oct 29      -  0:00 supervise log
    root 221268 266410   0   Oct 29      -  0:00 supervise log
    root 241896 266410   0   Oct 29      -  0:00 supervise umclient
    root 245928 266410   0   Oct 29      -  0:00 supervise log
    root 250042 266410   0   Oct 29      -  0:00 supervise qmail-smtpd
    root 258228 266410   0   Oct 29      -  0:00 supervise dsmcad
    root 290982 266410   0   Oct 29      -  0:00 supervise qmail-send
    root 323614      1   0   Oct 29      -  0:00 /command/readproctitle
service errors: ...e: warning: unable to fork for clear-errors, sleeping 60 seconds: out of memory?/usr/local/bin/enable-printers[56]: 585852 Killed?/usr/local/bin/enable-printers[44]: 0403-031 The fork function failed. There is not enough memory available.?supervise: warning: unable to fork for enable-printers, sleeping 60 seconds: out of memory?svscan: warning: unable to fork for enable-printers: out of memory?
    root 528562 266410   0 10:38:39      -  0:00 supervise enable-printers
    root 602298 266410   0   Nov 03      -  0:00 supervise smbd
    root 622728 266410   0   Nov 03      -  0:00 supervise nmbd

The output of "ps -ef | grep command.sv[s]can":

    root 266410 323614   0   Oct 29      -  1:39 /command/svscan /service

The output of "cat /service/clear-errors/run":

#!/bin/ksh
print -n '..........' >&2
sleep 300


I did not expect /service/clear-errors to be in "want up" state. I expected
it to be "up". Prior to the memory exhaustion, this service has worked
fine.

No process on the system has 188608 as PPID:

# ps -ef | grep 1[8]8608
    root 188608 266410   0   Oct 29      -  0:09 supervise clear-errors

While, for example, process 241896 does appear as a PPID:

# ps -ef | grep 2418[9]6
    root 184372 241896   0   Oct 29      -  7:57 /usr/upsman/umclient/umc -v
    root 241896 266410   0   Oct 29      -  0:00 supervise umclient

A truss of the complaining supervise process only showed one line, and I
interrupted it after 5 minutes:

# truss -p 188608
_poll(0x00000000, 0, 0)         (sleeping...)
Pstatus: process is not stopped

What can I do to further diagnose this? I didn't do "svc -t
/service/clear-errors" or "svc -dx /service/clear-errors" yet.

-- 
Jurjen Oskam

PGP Key available at http://www.stupendous.org/