rexecv, ssh, nohup, reop_export_path
Jack de Valpine <[email protected]>
| Newsgroups | gmane.linux.cluster.ssic.user |
|---|---|
| Message-ID | <[email protected]> |
Hi,
I have been through the archives a bit and seen that this topic has come
up in part but it is not clear if there is any resolution. Following is
my description of what we are seeing.
I have added rexecv into some code to start jobs on selected nodes.
Normally I login into the openSSI (1.2.2 as reported by cluster -r)
cluster via ssh. The job is started as follows:
/usr/bin/nohup my_distrubuted_job &
We use nohup on non ssi nodes so that the job will continue to process
even if the user logs out of their ssh session or the ssh session goes
down for some reason. So the theory is that it should work on the
cluster. However, after exiting my ssh session and having to kill the
xterm the session was running in as the exit just hangs on execution,
upon logging back in I have discovered the following:
* processes formerly running on other nodes are gone
* only process running on node 1 continues
* dmesg reports - reop_export_path: Can't export unlinked file
/cluster/dev/pts1 (deleted)
* lsof | grep pts - shows /dev/pts/1 (deleted) - repeated 4 times
* ls /proc/<pid of main process>/fd - shows that:
o 0 -> /dev/pts/1 (deleted)
o 1 -> nohup.out
o 2 -> nohup.out
o 3 -> nohup.out
o 4 -> pipe:[17098126]
o 5 -> input_file
o 6 -> pipe:[17098129]
where fd's 0, 4 and 6 are shown in red, which I think means they are
broken links.
In an earlier thread on "Migration pains," it seems the problem relates
to sshd disconnecting the pty which then frees up the pty when in fact
it should not be. It sounded like there was a potential patch/resolution
to this. Did this make it into 1.2.2? Does it need to be applied by hand?
Note that if I do note logout of the ssh session then the job runs fine
to completion.
Thanks for your time and please let me know if there is any additional
information that I can add to clarify the issue.
-Jack de Valpine
--
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction