Re: rexecv, ssh, nohup, reop_export_path

Hi,

Ok, I think that I may have come up with at least one work around for 
this myself. I did a search for ssh, background jobs and hanging on exit 
and came up with a couple of solutions. The one that seems simplest is 
to wrap the whole call in a shell call:

    sh -c "nohup my_program &"

Then it is possible to exit cleanly without hangup and tests that I have 
run continue to completion on all nodes.

To confirm this I have also watched top in a seperate ssh session. Some 
other possible solutions are as follows, though not really tested:

    * redirect stdin - nohup my_program </dev/null & - I did do a
      partial test on this and it appears to work however by redirecting
      stdin I think behavior of nohup is effected as it no longer
      generates nohup.out
    * use at - at -f <job_script> now

I hope this helps anyone else who may have encountered similar issues.

Regards,

-Jack


Jack de Valpine wrote:
> Hi,
>
> I have been through the archives a bit and seen that this topic has 
> come up in part but it is not clear if there is any resolution. 
> Following is my description of what we are seeing.
>
> I have added rexecv into some code to start jobs on selected nodes. 
> Normally I login into the openSSI (1.2.2 as reported by cluster -r) 
> cluster via ssh. The job is started as follows:
>
>     /usr/bin/nohup my_distrubuted_job &
>
> We use nohup on non ssi nodes so that the job will continue to process 
> even if the user logs out of their ssh session or the ssh session goes 
> down for some reason. So the theory is that it should work on the 
> cluster. However, after exiting my ssh session and having to kill the 
> xterm the session was running in as the exit just hangs on execution, 
> upon logging back in I have discovered the following:
>
>     * processes formerly running on other nodes are gone
>     * only process running on node 1 continues
>     * dmesg reports - reop_export_path: Can't export unlinked file
>       /cluster/dev/pts1 (deleted)
>     * lsof | grep pts - shows /dev/pts/1 (deleted) - repeated 4 times
>     * ls /proc/<pid of main process>/fd - shows that:
>           o 0 -> /dev/pts/1 (deleted)
>           o 1 -> nohup.out
>           o 2 -> nohup.out
>           o 3 -> nohup.out
>           o 4 -> pipe:[17098126]
>           o 5 -> input_file
>           o 6 -> pipe:[17098129]
>
>     where fd's 0, 4 and 6 are shown in red, which I think means they
>     are broken links.
>
> In an earlier thread on "Migration pains," it seems the problem 
> relates to sshd disconnecting the pty which then frees up the pty when 
> in fact it should not be. It sounded like there was a potential 
> patch/resolution to this. Did  this make it into 1.2.2? Does it need 
> to be applied by hand?
>
> Note that if I do note logout of the ssh session then the job runs 
> fine to completion.
>
> Thanks for your time and please let me know if there is any additional 
> information that I can add to clarify the issue.
>
> -Jack de Valpine
> -- 
> # Jack de Valpine
> # president
> #
> # visarc incorporated
> # http://www.visarc.com
> #
> # channeling technology for superior design and construction
>   

-- 
# Jack de Valpine
# president
#
# visarc incorporated
# http://www.visarc.com
#
# channeling technology for superior design and construction