re: gm_board_info hang in gm-1.2.3
Bob Felderman <[email protected]>
| Newsgroups | gmane.network.myrinet.general |
|---|---|
| Message-ID | <[email protected]> |
This patch will fix the gm_board_info
hang that people see in the gm-1.2.3
release. We'll be releasing gm-1.2.4
with this patch and some others, but I wanted to
get people the real fix for gm_board_info
in isolation.
The modifies the locking the gm_board_info is doing
in the driver, so you'll need to recompile, then
re-load the GM driver.
Related to this issue I tried a simple test with
rsh calling gm_allsize on a remote node.
20% rsh rwcp /ufs/feldy/ruth/gm-1.2.3/binary/bin/gm_allsize --blocking --slave
gm_board_info on that remote node shows the process
Port: Status PID
0: BUSY 14388 (this process [gm_board_info])
1: free
2: BUSY 14387
3: free
4: free
5: free
6: free
7: free
And on the local node
frisbee.myri.com 8% ps -aux | grep rsh
feldy 2464 0.0 0.5 1576 720 ? S Sep20 0:00 rsh -l help-handl
feldy 2465 0.0 0.0 0 0 ? Z Sep20 0:00 [rsh <defunct>]
feldy 6883 0.0 0.5 1576 720 pts/20 S 15:41 0:00 rsh rwcp /ufs/fel
feldy 6884 0.0 0.5 1576 720 pts/20 S 15:41 0:00 rsh rwcp /ufs/fel
I see "two" rsh commands to that node. Not sure what that is about, but...
I was able to get things in a state where I killed off the
rsh commands on the local node, but the remote node
still had the gm_allsize process holding open the port.
This is exactly what we have seen with the mpirun
process not being able to kill off the remote mpi job.
I'm trying a bit more investigation.
Index: gm/libgm/_gm_get_opener_pids.c
diff -u gm/libgm/_gm_get_opener_pids.c:1.8 gm/libgm/_gm_get_opener_pids.c:1.8.26.1
--- gm/libgm/_gm_get_opener_pids.c:1.8 Wed Mar 22 11:01:56 2000
+++ gm/libgm/_gm_get_opener_pids.c Thu Sep 21 14:58:05 2000
@@ -30,7 +30,8 @@
other_ps = ((i == ps->id)
? ps
- : gm_acquire_port_state_for_id (is, i));
+ : __gm_port_state_for_id (is, i));
+ /* : gm_acquire_port_state_for_id (is, i)); */
if (!other_ps) {
GM_PRINT (2, ("get_opener_pids: ps for id = %d not found\n",i));
continue;
@@ -46,7 +47,7 @@
/* release exclusive access to the port. */
if (other_ps != ps) {
- gm_release_port_state (other_ps);
+ /* gm_release_port_state (other_ps); */
}
}