MPICH/ch_gm and the LAM test suite
Troy Baer <[email protected]>
| Newsgroups | gmane.network.myrinet.general |
|---|---|
| Message-ID | <[email protected]> |
Hi, I've spent the last few hours adapting the LAM test suite (http://www.mpi.nd.edu/downloads/lam/lamtests-6.3.2.tar.gz) to run with MPICH/ch_gm on our quad-Xeon cluster. I've attached the short patch I've used, as well as the full output from running the suite. Here's the summary of what I've found: * ccl/op gives a warning that attempting to free MPI_SUM returns something other than MPI_ERR_OP. * comm/attr gives an error related to the precision of int and bit-shifting. This does not appear to be specific to MPICH/ch_gm, however. * dtyp/bakstr has a total of 52 errors where unpacking a packed variable does not give back the original result. I tried to figure out what might be causing this, but the code is sufficiently weird that I only succeeded in giving myself a headache. :) * dtyp/getel hangs with the message "Error: blocking send or recv on a self communication" and has to be killed manually. However, comments in the code do note that it is written in a way that can cause deadlock. * dtyp/pptransp hangs in MPI_Send() and must be killed manually. This code does have an implicit assumption that MPI_Send is either buffering or non-blocking (two MPI processes do MPI_Send()s to the other at the same time, so if MPI_Send blocks waiting for a corresponding MPI_Recv to start, the code will hang). * env/abort demonstrates that MPI_Abort() only kills the process in which it was called. This is not desirable behavior IMHO. How do these compare with the developers' lists of currently outstanding bugs with MPICH/ch_gm? --Troy -- Troy Baer email: [email protected] Science & Technology Support phone: 614-292-9701 Ohio Supercomputer Center web: http://oscinfo.osc.edu
mpich-patch
(text/plain, 1020 B)
--- ../lamtests-6.3.2-orig/make-defs Tue Nov 16 02:36:18 1999 +++ make-defs Mon Aug 28 15:02:08 2000 @@ -55,17 +55,16 @@ testing-lamd: $(BIN) @echo "*** Testing -lamd mode ***" - @$(MAKE) MODE="-lamd" LAMPWD="`pwd`" runtests + @$(MAKE) MODE="" LAMPWD="`pwd`" runtests testing-c2c: $(BIN) - @echo "*** Testing -c2c mode ***" - @$(MAKE) MODE="-c2c" LAMPWD="`pwd`" runtests +# @echo "*** Testing -c2c mode ***" +# @$(MAKE) MODE="" LAMPWD="`pwd`" runtests # Use the "cat" trick to ensure that "make" doesn't bail on the # "abort" test runtests: $(BIN) @ for bin in $(BIN) ; do \ - lamclean ; \ case $$bin in \ range) np=8 ;; \ cart) np=6 ;; \ @@ -73,6 +72,6 @@ graph) np=4 ;; \ *) np=2 ;; \ esac ; \ - echo mpirun -np $$np -s h $(MODE) -O $(LAMPWD)/$$bin ; \ - mpirun -np $$np -s h $(MODE) -O $(LAMPWD)/$$bin | cat; \ + echo mpiexec -n $$np $(LAMPWD)/$$bin ; \ + /usr/local/bin/mpiexec.new -n $$np $(LAMPWD)/$$bin | cat; \ done
MPIregress.o20664
(text/plain, 15.9 KB)
ccl: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/allgather mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/allreduce mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/alltoall mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/barrier mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/bcast mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/bcast_struct mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/gather mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/op WARNING in MPI_Op_free: error on free MPI_SUM not MPI_ERR_OP WARNING in MPI_Op_free: error on free MPI_SUM not MPI_ERR_OP error returned was Cannot free permanent MPI_Op [844(12)] error returned was Cannot free permanent MPI_Op [844(12)] mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/reduce mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/reduce_scatter mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/scan mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/scatter mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl/struct_gatherv make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/ccl' comm: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/attr ERROR: tag_ub is 1073741823, expected 32767 ERROR: tag_ub is 1073741823, expected 32767 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/commdup mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/commfree mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/compare mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/intercomm mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm/mpisplit make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/comm' dtyp: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/bakstr Found 52 errors idx 0; original 1; packval 49 (49); unpackval 0 idx 1; original 2; packval 50 (50); unpackval 0 idx 2; original 3; packval 45 (45); unpackval 3 idx 3; original 4; packval 46 (46); unpackval 4 idx 4; original 5; packval 41 (41); unpackval 5 <- idx 5; original 6; packval 42 (42); unpackval 6 <- idx 6; original 7; packval 37 (37); unpackval 7 idx 7; original 8; packval 38 (38); unpackval 8 idx 8; original 9; packval 33 (33); unpackval 9 <- idx 9; original 10; packval 34 (34); unpackval 10 <- idx 10; original 11; packval 29 (29); unpackval 0 idx 11; original 12; packval 30 (30); unpackval 0 idx 12; original 13; packval 25 (25); unpackval 13 <- idx 13; original 14; packval 26 (26); unpackval 14 <- idx 14; original 15; packval 21 (21); unpackval 0 idx 15; original 16; packval 22 (22); unpackval 0 idx 16; original 17; packval 17 (17); unpackval 17 <- idx 17; original 18; packval 18 (18); unpackval 18 <- idx 18; original 19; packval 13 (13); unpackval 0 idx 19; original 20; packval 14 (14); unpackval 0 idx 20; original 21; packval 9 ( 9); unpackval 21 <- idx 21; original 22; packval 10 (10); unpackval 22 <- idx 22; original 23; packval 5 ( 5); unpackval 0 idx 23; original 24; packval 6 ( 6); unpackval 0 idx 24; original 25; packval 7 (95); unpackval 25 <- idx 25; original 26; packval 8 (96); unpackval 26 <- idx 26; original 27; packval 3 (91); unpackval 0 idx 27; original 28; packval 4 (92); unpackval 0 idx 28; original 29; packval 0 (87); unpackval 29 <- idx 29; original 30; packval 0 (88); unpackval 30 <- idx 30; original 31; packval 0 (83); unpackval 0 idx 31; original 32; packval 0 (84); unpackval 0 idx 32; original 33; packval 0 (79); unpackval 33 <- idx 33; original 34; packval 0 (80); unpackval 34 <- idx 34; original 35; packval 0 (75); unpackval 0 idx 35; original 36; packval 0 (76); unpackval 0 idx 36; original 37; packval 0 (71); unpackval 37 <- idx 37; original 38; packval 0 (72); unpackval 38 <- idx 38; original 39; packval 0 (67); unpackval 0 idx 39; original 40; packval 0 (68); unpackval 0 idx 40; original 41; packval 0 (63); unpackval 41 <- idx 41; original 42; packval 0 (64); unpackval 42 <- idx 42; original 43; packval 0 (59); unpackval 0 idx 43; original 44; packval 0 (60); unpackval 0 idx 44; original 45; packval 0 (55); unpackval 45 <- idx 45; original 46; packval 0 (56); unpackval 46 <- idx 46; original 47; packval 0 (51); unpackval 0 idx 47; original 48; packval 0 (52); unpackval 0 idx 48; original 49; packval 0 ( 0); unpackval 49 <- idx 49; original 50; packval 0 ( 0); unpackval 50 <- idx 50; original 51; packval 0 ( 0); unpackval 0 <- idx 51; original 52; packval 0 ( 0); unpackval 0 <- idx 52; original 53; packval 0 ( 0); unpackval 0 idx 53; original 54; packval 0 ( 0); unpackval 0 idx 54; original 55; packval 0 ( 0); unpackval 0 <- idx 55; original 56; packval 0 ( 0); unpackval 0 <- idx 56; original 57; packval 0 ( 0); unpackval 0 idx 57; original 58; packval 0 ( 0); unpackval 0 idx 58; original 59; packval 0 ( 0); unpackval 0 <- idx 59; original 60; packval 0 ( 0); unpackval 0 <- idx 60; original 61; packval 0 ( 0); unpackval 0 idx 61; original 62; packval 0 ( 0); unpackval 0 idx 62; original 63; packval 0 ( 0); unpackval 0 <- idx 63; original 64; packval 0 ( 0); unpackval 0 <- idx 64; original 65; packval 0 ( 0); unpackval 0 idx 65; original 66; packval 0 ( 0); unpackval 0 idx 66; original 67; packval 0 ( 0); unpackval 0 <- idx 67; original 68; packval 0 ( 0); unpackval 0 <- idx 68; original 69; packval 0 ( 0); unpackval 0 idx 69; original 70; packval 0 ( 0); unpackval 0 idx 70; original 71; packval 0 ( 0); unpackval 0 <- idx 71; original 72; packval 0 ( 0); unpackval 0 <- idx 72; original 73; packval 0 ( 0); unpackval 0 idx 73; original 74; packval 0 ( 0); unpackval 0 idx 74; original 75; packval 0 ( 0); unpackval 0 <- idx 75; original 76; packval 0 ( 0); unpackval 0 <- idx 76; original 77; packval 0 ( 0); unpackval 0 idx 77; original 78; packval 0 ( 0); unpackval 0 idx 78; original 79; packval 0 ( 0); unpackval 0 <- idx 79; original 80; packval 0 ( 0); unpackval 0 <- idx 80; original 81; packval 0 ( 0); unpackval 0 idx 81; original 82; packval 0 ( 0); unpackval 0 idx 82; original 83; packval 0 ( 0); unpackval 0 <- idx 83; original 84; packval 0 ( 0); unpackval 0 <- idx 84; original 85; packval 0 ( 0); unpackval 0 idx 85; original 86; packval 0 ( 0); unpackval 0 idx 86; original 87; packval 0 ( 0); unpackval 0 <- idx 87; original 88; packval 0 ( 0); unpackval 0 <- idx 88; original 89; packval 0 ( 0); unpackval 0 idx 89; original 90; packval 0 ( 0); unpackval 0 idx 90; original 91; packval 0 ( 0); unpackval 0 <- idx 91; original 92; packval 0 ( 0); unpackval 0 <- idx 92; original 93; packval 0 ( 0); unpackval 0 idx 93; original 94; packval 0 ( 0); unpackval 0 idx 94; original 95; packval 0 ( 0); unpackval 0 <- idx 95; original 96; packval 0 ( 0); unpackval 0 <- mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/bottom mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/getel Error: blocking send or recv on a self communication mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/lbub mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/lbub2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/loop mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/paktest mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/pptransp mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/strangest1 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/structsr mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/structsr2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/transp mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/transp2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/transp3 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/transpa mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/zero1 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/zero2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/zero3 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/zero5 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp/zero6 make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/dtyp' env: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/abort --> This program tests MPI_ABORT and generates error messages [0] MPI Abort by user Aborting program ! [0] Aborting program! mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/err ERROR: rank 0: expected errcode 2, got 4162 ERROR: rank 1: expected errcode 2, got 4162 ERROR: rank 0: expected errcode 7, got 8263 ERROR: NULL function not detected ERROR: rank 1: expected errcode 7, got 8263 ERROR: NULL function not detected mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/final --> Should generate a "process not initialized" error message Fatal error; unknown error handler May be MPI call before MPI_INIT. Error message is MPI_BARRIER and code is 197 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/initialized mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/pcontrol mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/procname mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env/wtime resolution = 0.0000009537, wtick = 0.0000009537 make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/env' group: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group/group mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group/groupfree mpiexec -n 8 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group/range make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/group' pt2pt: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt' mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/badbuf mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/bsend mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/bsend_free mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/buffer mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/free [0] MPI internal Aborting program Unimplemented operation - persistent recv free [1] MPI internal Aborting program Unimplemented operation - persistent recv free [1] Unimplemented operation - persistent recv free [0] Unimplemented operation - persistent recv free mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/getcount mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/interf mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/iprobe mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/isend mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/probe mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/rsend mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/rsend2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/sendrecv mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/sendrecv_rep mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/seq mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/ssend mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/start mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/startall mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/test1 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/test2 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/test3 mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/testall mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/testany mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/testsome mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/waitall mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/waitany mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/waitnull mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/waitsome mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt/wildcard make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/pt2pt' topo: make[1]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo' *** Testing -lamd mode *** make[2]: Entering directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo' mpiexec -n 6 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo/cart mpiexec -n 2 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo/dimscreate mpiexec -n 4 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo/graph mpiexec -n 6 /misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo/sub make[2]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo' make[1]: Leaving directory `/misc/c/troy/Beowulf/tools/mpi-regression/lamtests-6.3.2/topo'