Re: remapping myrinet while jobs are running

On Fri, 24 Jun 2005, Loic Prylli wrote:

> From: Loic Prylli <[email protected]>
> Date: Fri, 24 Jun 2005 17:17:24 -0400
> Subject: Re: [Myrinet] remapping myrinet while jobs are running
> 
> John C. Wright wrote:
> > Hello list, I'm running gm-2.0.16_Linux on Linux version
> > 2.4.20-28.7sm (RH7.3) on a 24  node cluster with dual athlon
> > cpus on a Tyan board.
> > I understand that gm now remaps from the initialization script
> > and "negotiates" with the other nodes for the best network.  My
> > question is  how can I set if a node needs to be rebooted while
> > a program is running on  other nodes, that the remapping
> > doesn't change the existing architecture?   I just what the
> > node to reboot and "plug" itself back into the network map
> > without recreating it, so the the existing job doesn't loose
> > it's  connectivity and have to be restarted.
> >
> > I think this must be simple but I haven't been able to
> > determine how to do  it from my reading of the documentation.
> > Can anyone point me in the right direction?
> >
> 
> 
> You do not need to do anything special for this to work. The gm
> "mapper" runs constantly and if you reboot a node or plug a new
> one, it will be added to the configuration of everyone without
> disturbing existing jobs (at worst there can be a very small
> performance glitch at the time of the transition to the new
> configuration because some packets might be lost. But that should
> be completely transparent for the application. And the transition
> will be very quick because the old network map will continue to be
> used  while the "negotiation" to find the new one is ongoing. Only
> once everybody has agreed on the new map, it is installed everywhere).
> 
> 
> Loic

Unfortunately, this is not our experience.  Rebooting a node kills all 
running jobs.  I thought it should work this way, but it 'hangs' any job 
running on the network when a node (not involved in one of the running 
jobs) is rebooted.  Are there certain mapping settings that are needed or 
to be avoided for this to work as you describe?
-john