Re: Please read - proposed WG termination

On Thu, 1 Sep 2005, Roland Dreier wrote:

>    Bernard> In the HPC world most clusters use the cluster fabric (
>    Bernard> and IB is the future direction ) for both MPI and IP
>    Bernard> traffic. The IP traffic is usually for parallel file
>    Bernard> systems and system management and control. This high
>    Bernard> bandwidth IP network is required in most production HPC
>    Bernard> clusters.  With the current IPoIB only using UD, the
>    Bernard> performance is dismal. Our simulations using the small
>    Bernard> packet MTU of IB says that the parallel file systems (
>    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
>    Bernard> today and at 12X it will be about 10%.
>
> It's not clear to me that IPoIB-CM is really the answer here.  First
> of all, for Linux to take advantage of a 64K MTU, major surgery to the
> network stack will be required.  Production systems have trouble even
> enabling 8K jumbo frames, because the kernel can't allocate two
> physically contiguous buffers to use for receive.  Given that no

That is an implementation issue in Linux and shouldn't effect the protocol.

> current IB hardware can offload TCP/IP checksums for multi-packet RC
> messages, it's not clear how feasible this network stack surgery is.

Checksum offload is not part of the specification but an HCA feature.

>
> Second, if the justification for IPoIB-CM is really for parallel file

The justification is better performance through larger MTU, and leveraging 
connectd modes's apm capability.  I'd view parallel fs is just one application 
that can use it.

Vivek

> systems, then it seems much more promising to run the file system
> natively on InfiniBand.  For example, since Lustre is implemented on
> top of the Portals abstraction, native implementations of Lustre have
> been in use on IB, Myrinet, Quadrics, etc. for quite some time.
>
> - R.
>
>