Re: [myrinet] Re: mpich PBS etc. and gm_ports not closing

Hi,

Here's a patch that catches the C-c and calls MPID_Abort. This is just an
example, the patch is completely untested (I think it will compile). This
is against the prerelease of mpich-1.2..4. Does anyone want to volunteer
to test this, Or program in more sophisticated signal handling? The SIGINT
handling should be returned to normal in the MPI_Finalize code, just in
case anyone does nontrivial things after calling MPI_Finalize.

Doug



Index: mpid/ch_gm/adi2init.c
===================================================================
RCS file: /ufs/src/repositories/mpich/mpid/ch_gm/adi2init.c,v
retrieving revision 1.5
diff -u -r1.5 adi2init.c
--- mpid/ch_gm/adi2init.c	2000/09/13 22:16:48	1.5
+++ mpid/ch_gm/adi2init.c	2000/09/20 15:19:55
@@ -14,6 +14,7 @@
 #include "reqalloc.h"
 #include "gmpi.h"
 #include <unistd.h>
+#include <signal.h>
 
 /* Home for these globals */
 int MPID_MyWorldSize, MPID_MyWorldRank;
@@ -51,6 +52,12 @@
 MPID_Config * MPID_GetConfigInfo ANSI_ARGS((int *, char ***, void * ,int *));
 #endif
 
+static void exitall()
+{
+  struct MPIR_COMMUNICATOR *comm = 0;
+
+  MPID_Abort(comm ,0 ,NULL, NULL);
+}
 
 void MPID_Init( argc, argv, config, error_code )
 int  *argc, *error_code;
@@ -62,7 +69,8 @@
     int mypid;
     struct MPIR_COMMUNICATOR *comm = 0;
     struct gm_global *gm_intermediate, *g;
+    sctruct act *sigendall;
 
 #ifdef SMP_PLUG
     /* here we bypass the default fake config function. Instead, we call 
@@ -194,6 +202,18 @@
 	MPID_SendContig(comm, &mypid, sizeof(int), MPID_MyWorldRank, 
 			0, 0, 0, msgrp, &err);
       }
+
+    /* Signal handling to properly exit all MPI processes if C-c
+       is used to exit the interactive parallel job. Relies on
+       MPID_Abort doing the 'right' thing. See also src/env/init.c
+    */
+
+    sigendall->sa_handler = exitall;
+    sigemptyset(&sigendall->sa_mask);
+    sigendall->sa_flags = 0;
+
+    sigaction(SIGINT, sigendall, NULL);
+
 }
 
 /* Created a routine for remote PID's for TotalView debugger */
Index: src/env/init.c
===================================================================
RCS file: /ufs/src/repositories/mpich/src/env/init.c,v
retrieving revision 1.1.1.3
diff -u -r1.1.1.3 init.c
--- src/env/init.c	1999/12/29 18:27:24	1.1.1.3
+++ src/env/init.c	2000/09/20 15:20:05
@@ -125,6 +125,9 @@
    ch_p4 device:
    SIGUSR1
 
+   ch_gm device:
+   SIGQUIT, SIGVTALRM
+
    The ch_p4 device also catches SIGINT, SIGFPE, SIGBUS, and SIGSEGV; this
    helps the p4 device (and MPICH) more gracefully abort a failed program.
 


On Wed, 20 Sep 2000, Douglas Johnson wrote:

> I think this has everything to do with how the parallel job was
> started. The rsh may be terminated by the C-c but the process on the other
> end may not die (leaving the port in use). Is this the problem? Or are
> people seeing open ports with no process? 
> 
> I think _some_ of these problems will be fixed when MPI_Abort works
> properly (next mpich). The other solution is to change how the parallel
> program is started. At OSC we've created a c program that uses the task
> manager primitives provided by PBS to start the mpi processes on the nodes
> that have been allocated for that job. The problem with that is that the
> TM layer in PBS is not yet reliable enough to rely on it for high
> processor count jobs. This program can be downloaded from,
> 
> http://www.osc.edu/~djohnson/mpiexec
> 
> We're at version 0.4 now. There's a diff for PBS included in that for
> version 2.2p11.
> 
> There are lots of other approaches that could be taken to fix this
> problem, but none of the shell script versions are particularly elegant.
> At the MPICH layer there should be signal handling that catches the C-c
> and calls the new and improved MPI_Abort. Should be a pretty simple fix...
> 
> Doug