Re: myrinet backwards compatibility

John C. Wright wrote:
> Hi, we sent this message to [email protected], but thought we'd post it 
> here in case any users can chime in on their experience/attempts.
> 

In the past we mixed different Myrinet generations.  I don't remember
the specific part numbers, but I will try and describe what we did.

- Combo Myri-1280 and Myri-2000

The original system was a 276 node cluster with
a Myri-1280 network.  The switches were some of
the earliest 128 port switches.  Copper cables connected
the cards and switches (DB25 like connectors).
Ribbon cables were used to connect the CLOS switches
to the spines for a full-bisection bandwdith network.

The second system was Myri-2000.  It used some of
the early M3M-128 switches (replaceable line cards).
I don't remember the specific NIC model.  It used
copper connectors (narrow, not like the DB connectors),
right before the Myricom's push to go all fibre.

We connected the two networks together.  The only
trick was that the ribbon spine cards for the 2nd system
had to be set to 1280 speed (jumper on the board).
This setup worked ok, but we never ran codes (except HPL)
across different node (and network) types.  The single network
allowed our reliability system to work.  We eventually
broke it apart to isolate instances of the batch system
to make maintenance easier.  We didn't break them apart
because of network issues.

A 3rd system came in that consisted of 768 nodes, PCIX-B
NICs, M3M-128 switches, M3-SW16-8F and M3-SPINE-8F cards.
To save money, we took the extra ribbon-cable spine cards
used in the second system and used those in the new system
to connect the switches.  This system has been in production
since 2002.

The ribbon cable equipment has been more reliable
than the fibre connectors (because of the reliability of the
fibre transeivers).  However, the ribbon cables have a tendency
to slip.  They will come lose and cause CRC errors on the
network.  Every couple of months during downtime I would go through
and check ports.  After a few hours I could generally get the network
to no longer generate CRC errors under heavy load.  Even with the
CRC errors, we could never correlate job failures or job slowdowns
because of this.

I am not sure if this answers your question about mixing hardware,
but I hope it provides some confidence that it has been done.

Craig




> Thanks.
> 
>> Date: Thu, 1 Jun 2006 15:39:41 -0400 (EDT)
>> Subject: compatibility of older cards
>>
>>
>> Hi --
>>
>> This is a question about whether older cards will work with
>> newer cards in the same network:
>>
>> We have a setup consisting of
>>
>> (25)    M3F-PCI64B-2    fiber 2000 cards
>> (25)    M3F-CB-*M       fiber cables
>> (4)     M3-SW16-8F      8 port fiber line cards
>> (4)     M3-BLANK        blank line card
>> (1)     M3-E64          5U enclosure
>> (1)     M3-M            monitoring line card
>>
>> being used in Tyan Tiger MPX motherboards with Althon MP2000
>> processors.
>>
>>
>> We recently obtained extra NIC cards and switch cards
>> from another lab:
>>
>>
>> 3rd-generation, Myrinet-2000, serial-port (M3S) components:
>>
>> (17)    M3S-PCI64B-2    cards, round cable 2001
>> (16)    round cables
>> (1)     M3-SW16-8S      switch module, round cables, 8 ports
>> (1)     M3-SPINE-8S     switch module, round cables, 8 ports
>> (1)     M3-E16          2 module switch chassis  $1600
>>
>>
>> 2nd-generation, Myrinet-1280, SAN (M2M) components:
>>
>> (4)     M3M-PCI64B-2    cards, ribbon cable, 2000
>> (10)    M2M-PCI64A-2    cards, ribbon cable
>> (17)    flat ribbon cables
>> (1)     M2M-SW16 16     port switch, flat ribbon, not modular
>>
>>
>> We would like to move our fiber based network to a new cluster
>> with PCI-X slots, and hopefully integrate the equipment we
>> obtained from the other lab into this new network.
>>
>> So here are the questions:
>>
>> 1. Can we move M3F-PCI64B-2 NICs to a new cluster with PCI-X
>> slots and AMD opteron processors?
>>
>> 2. Which of the equipment listed above, that we obtained from
>> the other lab, can we integrate into this network, based on the
>> M3-E64 chassis?  i.e., can we mix M3F and M3S in the same
>> network?  How about adding in M3M-PCI64B-2 or M2M cards?
>>
>> 3. Can we use the most recent gm drivers, which have dynamic
>> mapping, with all of the cards?  Are any of the older cards
>> restricted to using older drivers?
>>
>> 4. Can you send us detailed information on the older cards
>> and line cards?
>>
>> 5. Do you have any type of trade-in upgrade program?
> 
> 
> Darin Ernst and John Wright
> 
> _______________________________________________
> Myrinet mailing list
> [email protected]
> http://email.osc.edu/mailman/listinfo/myrinet