Re: Please read - proposed WG termination

At 10:51 AM 9/1/2005, Vivek Kashyap wrote:
>On Thu, 1 Sep 2005, Roland Dreier wrote:
>
>>    Bernard> In the HPC world most clusters use the cluster fabric (
>>    Bernard> and IB is the future direction ) for both MPI and IP
>>    Bernard> traffic. The IP traffic is usually for parallel file
>>    Bernard> systems and system management and control. This high
>>    Bernard> bandwidth IP network is required in most production HPC
>>    Bernard> clusters.  With the current IPoIB only using UD, the
>>    Bernard> performance is dismal. Our simulations using the small
>>    Bernard> packet MTU of IB says that the parallel file systems (
>>    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
>>    Bernard> today and at 12X it will be about 10%.
>>
>>It's not clear to me that IPoIB-CM is really the answer here.  First
>>of all, for Linux to take advantage of a 64K MTU, major surgery to the
>>network stack will be required.  Production systems have trouble even
>>enabling 8K jumbo frames, because the kernel can't allocate two
>>physically contiguous buffers to use for receive.  Given that no
>
>That is an implementation issue in Linux and shouldn't effect the protocol.

Linux does support large send off-load today so it is not clear what the 
issue is you are speaking of above.


>>current IB hardware can offload TCP/IP checksums for multi-packet RC
>>messages, it's not clear how feasible this network stack surgery is.
>
>Checksum offload is not part of the specification but an HCA feature.

It is also not that hard to implement.



>>Second, if the justification for IPoIB-CM is really for parallel file
>
>The justification is better performance through larger MTU, and leveraging 
>connectd modes's apm capability.  I'd view parallel fs is just one 
>application that can use it.

Just to clarify, there is no change in physical MTU.  The IB hardware is 
unchanged w.r.t. the MTU.  All that is done is the same concept as for 
large send off-load - tell the network stack there is a larger MTU and let 
the hardware SAR the segment / datagram into physical units of transfer.

Mike


>Vivek
>
>>systems, then it seems much more promising to run the file system
>>natively on InfiniBand.  For example, since Lustre is implemented on
>>top of the Portals abstraction, native implementations of Lustre have
>>been in use on IB, Myrinet, Quadrics, etc. for quite some time.
>>
>>- R.
>>
>
>_______________________________________________
>IPoverIB mailing list
>[email protected]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib