[Myrinet] Re:Re: [Myricom help #8736] problem about the bandwidt
"chyyuu" <[email protected]>
| Newsgroups | gmane.network.myrinet.general |
|---|---|
| Message-ID | <3C4EC39E.000032.10411@mta1> |
aCBvZiBteXJpbmV0Mms=?= X-Priority: 3 X-Originating-IP: [166.111.68.179] X-Mailer: Coremail2.0 Copyright Tebie Ltd., 2001 Content-Type: Multipart/Alternative; boundary="Boundary-=_wesXcwsBQEhqenMfpjTHLUBHijUb" --Boundary-=_wesXcwsBQEhqenMfpjTHLUBHijUb Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 8bit hi, We solve the problem. Our project is a VIA(Virtual Interface Architecure) sofware for Myrinet. we found when node send to other node(which don't want to receive messages), the bandwidth is be not very fast. The reason is when the node that got the package in LANAI, it will interrupt the host to print a message, so the bandwidth is not very fast. We disble MCP in LANAI to interrupt host. then the max bandwidth(one node continue sending 100k messages from host to card, and another node just discard the messages) will go to 255MB-261MB(2089Mb)/s. When we send 4k package from host to card, and another node gets the messages to host memory, the max bandwidth is 248MB. We also found the bandwidth is not very stable, sometimes the bandwidth will drop to 90~120MB/s. Maybe it's the results of other device interrupts or kernel acitons. Is the max bandwidth we got reasonable(I mean 261MB/s for 100KB package or 248MB/s for 4kB package)? We want to know the exact max bandwidth for myrinet2k&lanai9.2 in 64bit/66MHZ PCI slot. Wait your answers and Thanks a lot. chen yu >chyyuu wrote: > >> But we have some environments we did not understand. Our systems are two nodes >> connected by myrinet 2k with lanai9.2 and a switch with 32 ports. one node >> includes 2 AMD athlon CPU, 512M DDR mem and a lanai9.2 card inserted in a >> 64bit/66Mhz PCI slot. > >When you say "one" node is a dual athlon, that means the other node is something >else ? no, other node is a dual athlon. > >> From our test, if our software in one node sends messages to other node that >> is not connected on the switch(send message to a not existed node), the max >> bandwidth of sending 4KB messages will be 250-255MB/s(very exciting). > >This is the link speed, looks normal. Do you send from the host or just >from the Lanai memory ? from host to lanai to network > >> But if our software in a node sends messages to a node that is connected on >> the switch(send message to a existed and connected node), the max bandwidth of >> sending 4KB message only be 104MB/s. > >> We don't know why? Can you give us some reasons? > >Well, it's hard to say if you use your own MCP. >Are the 4KB buffers aligned ? We use our own MCP, and the buffers ar aligned. >You can try to connect the 2 hosts directly (without any switch) >if you want, but I expect the results to be the same than with >the switch (to go through a crossbar is in order of 100 ns). >You can also try to receive packets on the NIC and drop them, >counting them to estimaste the incoming bandwidth. If you get >the link speed, that means the bottleneck is likely to be in >your receiving MCP (PCI Write used in the receiving side yields >usually better performance). > >Hope it helps. > >Patrick > >---------------------------------------------------------- >| Patrick Geoffray, Ph.D. [email protected] >| Myricom, Inc. http://www.myri.com >| Cell: 865-389-8852 685 Emory Valley Rd (B) >| Phone: 865-425-0978 Oak Ridge, TN 37830 >---------------------------------------------------------- > __________________________________________ È«³¡ÌؼۣºÊýÂëÏà»ú¡¢ÕÆÉϵçÄÔ¡¢MP3 http://shopping.263.net/category21.htm ÌìÓ®³´¹É,ÌìÉúÎÒÓ®! http://stock19.263.net/download/borntowin.htm --Boundary-=_wesXcwsBQEhqenMfpjTHLUBHijUb Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: 8bit hi, <br> We solve the problem. Our project is a VIA(Virtual Interface Architecure) sofware for Myrinet. we found when node send to other node(which don't want to receive messages), the bandwidth is be not very fast. The reason is when the node that got the package in LANAI, it will interrupt the host to print a message, so the bandwidth is not very fast. We disble MCP in LANAI to interrupt host. then the max bandwidth(one node continue sending 100k messages from host to card, and another node just discard the messages) will go to 255MB-261MB(2089Mb)/s. <br> When we send 4k package from host to card, and another node gets the messages to host memory, the max bandwidth is 248MB. <br> We also found the bandwidth is not very stable, sometimes the bandwidth will drop to 90~120MB/s. Maybe it's the results of other device interrupts or kernel acitons. <br> Is the max bandwidth we got reasonable(I mean 261MB/s for 100KB package or 248MB/s for 4kB package)? We want to know the exact max bandwidth for myrinet2k&lanai9.2 in 64bit/66MHZ PCI slot.<br><br> Wait your answers and Thanks a lot.<br><br>chen yu<br><br>>chyyuu wrote:<br>><br>>> But we have some environments we did not understand. Our systems are two nodes<br>>> connected by myrinet 2k with lanai9.2 and a switch with 32 ports. one node<br>>> includes 2 AMD athlon CPU, 512M DDR mem and a lanai9.2 card inserted in a<br>>> 64bit/66Mhz PCI slot.<br>><br>>When you say "one" node is a dual athlon, that means the other node is something <br>>else ?<br> no, other node is a dual athlon.<br>><br>>> From our test, if our software in one node sends messages to other node that<br>>> is not connected on the switch(send message to a not existed node), the max<br>>> bandwidth of sending 4KB messages will be 250-255MB/s(very exciting).<br>><br>>This is the link speed, looks normal. Do you send from the host or just <br>>from the Lanai memory ?<br> from host to lanai to network<br>><br>>> But if our software in a n ode sends messages to a node that is connected on<br>>> the switch(send message to a existed and connected node), the max bandwidth of<br>>> sending 4KB message only be 104MB/s.<br>><br>>> We don't know why? Can you give us some reasons?<br>><br>>Well, it's hard to say if you use your own MCP. <br>>Are the 4KB buffers aligned ?<br> We use our own MCP, and the buffers ar aligned. <br><br>>You can try to connect the 2 hosts directly (without any switch) <br>>if you want, but I expect the results to be the same than with <br>>the switch (to go through a crossbar is in order of 100 ns).<br>>You can also try to receive packets on the NIC and drop them, <br>>counting them to estimaste the incoming bandwidth. If you get <br>>the link speed, that means the bottleneck is likely to be in <br>>your receiving MCP (PCI Write used in the receiving side yields <br>>usually better performance).<br>><br>>Hope it helps.<br>><br>>Patrick<br>><br>>---------------------------------------------- ------------<br>>| Patrick Geoffray, Ph.D. [email protected] <br>>| Myricom, Inc. http://www.myri.com<br>>| Cell: 865-389-8852 685 Emory Valley Rd (B)<br>>| Phone: 865-425-0978 Oak Ridge, TN 37830<br>>----------------------------------------------------------<br>><br><br><br><br><br>__________________________________________<br><br>È«³¡ÌؼۣºÊýÂëÏà»ú¡¢ÕÆÉϵçÄÔ¡¢MP3 http://shopping.263.net/category21.htm<br>ÌìÓ®³´¹É,ÌìÉúÎÒÓ®! http://stock19.263.net/download/borntowin.htm<br> --Boundary-=_wesXcwsBQEhqenMfpjTHLUBHijUb--