Re: Please read - proposed WG termination

On Thu, 1 Sep 2005, Michael Krause wrote:

> At 10:51 AM 9/1/2005, Vivek Kashyap wrote:
>> On Thu, 1 Sep 2005, Roland Dreier wrote:
>> 
>>>    Bernard> In the HPC world most clusters use the cluster fabric (
>>>    Bernard> and IB is the future direction ) for both MPI and IP
>>>    Bernard> traffic. The IP traffic is usually for parallel file
>>>    Bernard> systems and system management and control. This high
>>>    Bernard> bandwidth IP network is required in most production HPC
>>>    Bernard> clusters.  With the current IPoIB only using UD, the
>>>    Bernard> performance is dismal. Our simulations using the small
>>>    Bernard> packet MTU of IB says that the parallel file systems (
>>>    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
>>>    Bernard> today and at 12X it will be about 10%.
>>> 
>>> It's not clear to me that IPoIB-CM is really the answer here.  First
>>> of all, for Linux to take advantage of a 64K MTU, major surgery to the
>>> network stack will be required.  Production systems have trouble even
>>> enabling 8K jumbo frames, because the kernel can't allocate two
>>> physically contiguous buffers to use for receive.  Given that no
>> 
>> That is an implementation issue in Linux and shouldn't effect the protocol.
>
> Linux does support large send off-load today so it is not clear what the 
> issue is you are speaking of above.

Let us say that even if there is an issue with 8K frames/contiguous memory
allocation in Linux, it is a Linux specific issue.

>
>
>>> current IB hardware can offload TCP/IP checksums for multi-packet RC
>>> messages, it's not clear how feasible this network stack surgery is.
>> 
>> Checksum offload is not part of the specification but an HCA feature.
>
> It is also not that hard to implement.

agreed and is not relevant to developing the protocol.

>
>
>
>>> Second, if the justification for IPoIB-CM is really for parallel file
>> 
>> The justification is better performance through larger MTU, and leveraging 
>> connectd modes's apm capability.  I'd view parallel fs is just one 
>> application that can use it.
>
> Just to clarify, there is no change in physical MTU.  The IB hardware is 
> unchanged w.r.t. the MTU.  All that is done is the same concept as for large 
> send off-load - tell the network stack there is a larger MTU and let the 
> hardware SAR the segment / datagram into physical units of transfer.

yes, it is for a higher IP MTU and brings large sendoff benefits to IPoIB.

Vivek
>
> Mike
>
>
>> Vivek
>> 
>>> systems, then it seems much more promising to run the file system
>>> natively on InfiniBand.  For example, since Lustre is implemented on
>>> top of the Portals abstraction, native implementations of Lustre have
>>> been in use on IB, Myrinet, Quadrics, etc. for quite some time.
>>> 
>>> - R.
>>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email protected]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>