Re: IP Multicast over Myrinet

Hi,

One of our software developers replied the following:

 > > UDP Packet size is 1425 bytes ( assuming MTU of 1500 bytes )
 > > MYRINET : Sender sends 3.5 Mbytes in 0.02 seconds ( 1400 Mbits/sec
)
 > > GIGABIT : Sender sends 3.5 Mbytes in 0.04 seconds ( 700 Mbits/sec 
)
 > > 
 > > These numbers roughly match the netperf tests below.

Assuming you're sending using multicast on a 3 node network, the
numbers you quote for Myrinet are impossible.  Something is dropping
the packets before they even leave your host, or your measurements are
in error.

Myrinet does multicast through repeated unicast.  This means that if
you have 3 hosts on the network (A, B, C), and you're sending from
host C, 1 packet will go to host A, and one to host B.  The link speed
is 2Gb/sec, so the fastest you can multicast in that scenario is
1Gb/sec.

 > > Multicast Receiver ( Host B ): 
 > > 
 > > MYRINET : Receiver reads 5 packets in 0.0018 seconds.
 > > GIGABIT : Receiver reads 5 packets in 0.00011 seconds.
 > > 
 > > As a result while using myrinet, data is lost ( UDP packets are
dropped or
 > > overwritten in the sender buffer since the receiver is much slower
 > > compared to the sender.
 > > 
 > > 
 > > So I think if the problem of the slow read on the receiver is
fixed, the
 > > problem of packet loss will automatically be fixed.
 > > 
 > > 
 > > I will be really glad if someone can tell me how to fix this
problem,
 > > Thanks in advance,
 > > Sudipta

A) Use gm-2  It does interrupt coalescing, and will do a better job on
receiving things faster.

B) Use a larger MTU.  There's no point in restricting the mtu to 1400
on a private network.  The less trips through the socket layer, IP
stack, etc, the less overhead you will see, thus giving you more
bandwidth. 

You really don't want to do high bandwidth multicast on GM/Myrinet
if you're using more than a handful of hosts.  It won't scale.  Real
ethernet is a much better choice for this.

Hope this helps.
Susan

--

------------------------------------------
Susan Blackford
Member of Technical Staff
Myricom Inc.
------------------------------------------