Myrinet NICs and temperature(?) problems

Hello everyone,

We're having some trouble at "the other OSC", the Oxford Supercomputing
Centre, with our Myrinet NICs. Specifically, we have what we think must be
a *very* high failure rate. We have 64 dual CPU PCs (two racks of 32) each
with a single Myrinet interface card. Our failure rate is at least 1-2 per
month every month.

By "failure" I mean that the green status LED is off, the card is not
pinging and the loopback tests described on the Myricom web site indicate
a card failure. I have (in most cases) ruled out cable and switch faults
by changing those components/ports and remapping. I've also tried
reseating the cards without success.

My current suspicion is that the cards are getting too hot - from the
service processor and the temperature of the exhausted air I'd guess the
cards are in air at about 40-45C depending on rack position.  Frankly I
don't think our racks are ventilated sufficiently and have far too much
kit in them. However, before I get onto our supplier/installer (IBM) and
try to get them to do something about it, I'm trying to find out if
anybody else has had any firmer evidence or experience of temperature
related problems.

Of course, if anybody has any alternative explanations or suggestions, I'm
very happy to hear from them.

Thanks in adavance,

Jon Lockley

------------------------------------------------------------------------
| Dr Jon Lockley                   |                                   |
| Oxford Supercomputing Centre     | Email [email protected] |
| Oxford University Computing Lab. | Tel +44 (0)1865 283569            |
| Wolfson Building                 | Fax +44 (0)1865 273839            |
| Parks Rd.                        | www.osc.ox.ac.uk                  |
| Oxford, OX1 3QD                  | "Out of Darkness Cometh Light"    |
| England                          |                                   |
------------------------------------------------------------------------