RE: Please read - proposed WG termination

 

-----Original Message-----
From: Michael Krause [mailto:[email protected]]
Sent: Thursday, September 01, 2005 10:09 PM 
 

At 10:51 AM 9/1/2005, Vivek Kashyap wrote:


On Thu, 1 Sep 2005, Roland Dreier wrote:



   Bernard> In the HPC world most clusters use the cluster fabric (
   Bernard> and IB is the future direction ) for both MPI and IP
   Bernard> traffic. The IP traffic is usually for parallel file
   Bernard> systems and system management and control. This high
   Bernard> bandwidth IP network is required in most production HPC
   Bernard> clusters.  With the current IPoIB only using UD, the
   Bernard> performance is dismal. Our simulations using the small
   Bernard> packet MTU of IB says that the parallel file systems (
   Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
   Bernard> today and at 12X it will be about 10%.

It's not clear to me that IPoIB-CM is really the answer here.  First
of all, for Linux to take advantage of a 64K MTU, major surgery to the
network stack will be required.  Production systems have trouble even
enabling 8K jumbo frames, because the kernel can't allocate two
physically contiguous buffers to use for receive.  Given that no


That is an implementation issue in Linux and shouldn't effect the protocol.


Linux does support large send off-load today so it is not clear what the
issue is you are speaking of above. 
 
[dg]  Large send offload solves the send side, but doesn't help on the
receive side. ipoib-cm solves both ends. 
Roland has a point that in Linux, it might be more complicated to support
>4KB frames. But I agree that this is an OS specific issue. In this specific
case, solving it may also be beneficial for other applications such as
Ethernet jumbo frames (if it makes any difference). 




current IB hardware can offload TCP/IP checksums for multi-packet RC
messages, it's not clear how feasible this network stack surgery is.


Checksum offload is not part of the specification but an HCA feature.


It is also not that hard to implement.
 
[dg] Sounds right for ipoib-ud. For ipoib-cm csum offload implementation is
complicated. By the time you already transmitted 64KB, you need to go and
find the first packet (which is probably already in your peer' memory) and
place the csum there...  
 Anyway , I still see great value in ipoib-cm even without the csum offload.
The main reason is that to my knowledge, in Linux, large send and zero copy
are only supported for sendfile(). The rest of the operations are on the
copy path anyway. Calculating csum while copying, is not a big deal.



 -Dror

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib