Re: 10-Gigabit ethernet questions

Hi Dan,

Just to complete Susan's reply.

Susan Blackford wrote:
>> 1) Myricom is selling the cards, but what switches are these cards for?
> 
> 
> Myri-10G NICs are dual-protocol NICs.

The 10G nics use the same physical link as 10 Gigabit Ethernet (true 
10Gb/s data rate). There are two differences between Ethernet and 
Myrinet at the protocol level:

* Flow control: Myrinet has byte-level backpressure flow control whereas 
Ethernet as Packet-level flow control. In short, Myrinet has little 
buffering in the switch and the sender can be stopped in the middle of 
sending a packet if the packet is blocked somewhere. Ethernet needs to 
be able to buffer a full packet (ie 9000 Bytes jumbo frame) and it uses 
special PAUSE packets to stop the sender.

* Routing: Myrinet routing is source-based and Ethernet routing is 
destination-based. The Ethernet destination MAC address is looked up by 
each crossbar to find the exit port. The Myrinet packet contain the exit 
port to take for each crossbar.

There are 3 types of usage for Myri-10G NICs:

* IT/Storage: Using it as an Ethernet NIC (typically running IP on it), 
plugged in a 10GigE switch from any vendor and an Ethernet NIC. Today, 
this is the only PCI-Express 10GigE NIC available in volume. This is 
pure Ethernet, not an encapsulation.

* HPC: You can also plug it into a Myri-10G switch and get lower latency 
and/or much higher port count than Ethernet switches. The links are used 
in Myrinet modes. This is like Myrinet-2000 (Myri-2G) but 5 times more 
link bandwidth.

* Hybrid: You can use a low-latency Ethernet switch (same switch latency 
as Myrinet switches but somewhat limited port count). Such switches 
start to be announced. In this configuration, MX will use the NIC in 
Ethernet mode. You can also use a Myri-10G switch with Ethernet line 
cards on it. These linecards have a chip that convert routing and flow 
control between Ethernet and Myrinet at line rate.


It's easier to understand if you separate the link protocol (Ethernet vx 
Myrinet) and the host protocol (native IP vs MX).

>> 2) The software to support these cards currently is released as a beta 
>> test. Is the software stable for long-term performance?

The PCI-Express spec has some room for interpretation, so we had to test 
compatibility with various PCI-Express chipsets (and sometimes work 
around their bugs in firmware). For the Ethernet firmware, we also had 
to test interoperability with 10GigE switches.

The Ethernet firmware is ready for stable release. There will be update 
for added features or new funky PCI-Express chipsets, but that's it.

For MX-10G, it already shares the library and the driver code with the 
MX-2G version, and we are backporting some firmware goodies to the MX-2G 
firmware. Once that's done, the MX-2G and MX-10G versions will share 95% 
of the lib/driver/firmware code.

>> 3) The performance tests were executed using two nodes, each with two 
>> dual-core Opterons. Have any performance tests been performed in any 
>> topology other than two nodes hooked directly to each other?

Several 10G clusters are being deployed. Largest one installed to date 
is 128 nodes. We will publish full Pallas results on it as soon as I 
enable route dispersion in MX-10G.

Patrick
-- 
Patrick Geoffray
Myricom, Inc.
http://www.myri.com