[Myrinet] Re: [Myricom help #8736] problem about the bandwidth of myrinet2k

chyyuu wrote:

> But we have some environments we did not understand. Our systems are two nodes
> connected by myrinet 2k with lanai9.2 and a switch with 32 ports. one node
> includes 2 AMD athlon CPU, 512M DDR mem and a lanai9.2 card inserted in a
> 64bit/66Mhz PCI slot.

When you say "one" node is a dual athlon, that means the other node is something 
else ?

> From our test, if our software in one node sends messages to other node that
> is not connected on the switch(send message to a not existed node), the max
> bandwidth of sending 4KB messages will be 250-255MB/s(very exciting).

This is the link speed, looks normal. Do you send from the host or just 
from the Lanai memory ?

> But if our software in a node sends messages to a node that is connected on
> the switch(send message to a existed and connected node), the max bandwidth of
> sending 4KB message only be 104MB/s.

> We don't know why? Can you give us some reasons?

Well, it's hard to say if you use your own MCP. 
Are the 4KB buffers aligned ?
You can try to connect the 2 hosts directly (without any switch) 
if you want, but I expect the results to be the same than with 
the switch (to go through a crossbar is in order of 100 ns).
You can also try to receive packets on the NIC and drop them, 
counting them to estimaste the incoming bandwidth. If you get 
the link speed, that means the bottleneck is likely to be in 
your receiving MCP (PCI Write used in the receiving side yields 
usually better performance).

Hope it helps.

Patrick

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      [email protected] 
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------