Myrinet job hangs

Hi,

I have been performing some calculations on a cluster of 4 pc's, each of
which is a dual PIII box. The boxes are connected both by fast ethernet
and myrinet. Red Hat linux is used with a 2.4 kernel.

For some jobs we observe the following problem. The job runs correctly for
a short while, and then suddenly the /proc/pid directory for the job
becomes unreadable. This prevents top and ps from working. The job keeps
running, but makes no progress, and cannot be killed. We end up having to
perform a hard reboot on the system, as soft reboots do not work.

Does anyone know what the problem might be, and what can be done about it?

Cheers,

Andrew

  +----------------------------------------------------+
   Andrew Horsfield       e-mail: [email protected] 
    FECIT, Hayes Park Central, Hayes End Road, Hayes,
           Middlesex UB4 8FE, United Kingdom.           
   phone: +44-(0)20-8606-4653  FAX: +44-(0)20-8606-4422 
  +----------------------------------------------------+