re: port held open and killing processes

=> I was able to get things in a state where I killed off the 
=> rsh commands on the local node, but the remote node
=> still had the gm_allsize process holding open the port.
=> This is exactly what we have seen with the mpirun 
=> process not being able to kill off the remote mpi job.
=> 
=> I'm trying a bit more investigation.


It appears that a 
    kill -9 <pid>
of the rsh on the local node leaves the remote job still running.
Using
    kill <pid>
on the local node seems to kill off the remote process cleanly.

Does that make sense to anyone?

It looks to me like our stock mpirun.ch_gm
uses kill 'KILL' which is just kill -9

frisbee.myri.com 31% grep kill mpirun.ch_gm.in 
   print STDERR " --gm-kill <n>  n secs after first process exits, kill all other processes \n";
     if (kill 0 => $p) {
       print "killing process $p\n" if $verbose;
       kill 'KILL', $p;
    } elsif ($_ eq '--gm-kill') {
        print "option kill procs after $ARGV[0] seconds\n" if $verbose;
        $kill_after_t = 1;
        printf("kill processor %d seconds after first exits\n",$time) if $verbose;
if ($kill_after_t){
    if (kill 0 => $p){
      print "1 process still alive: wait $time sec, then kill it\n" if $verbose;
      print "$still_alive processes still alive: wait $time sec, then kill them\n" if $verbose;
      if (kill 0 => $p) {
        print "killing process $p\n" if $verbose;
        kill 'KILL', $p;