Re: Please read - proposed WG termination

    Bernard> In the HPC world most clusters use the cluster fabric (
    Bernard> and IB is the future direction ) for both MPI and IP
    Bernard> traffic. The IP traffic is usually for parallel file
    Bernard> systems and system management and control. This high
    Bernard> bandwidth IP network is required in most production HPC
    Bernard> clusters.  With the current IPoIB only using UD, the
    Bernard> performance is dismal. Our simulations using the small
    Bernard> packet MTU of IB says that the parallel file systems (
    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
    Bernard> today and at 12X it will be about 10%.

It's not clear to me that IPoIB-CM is really the answer here.  First
of all, for Linux to take advantage of a 64K MTU, major surgery to the
network stack will be required.  Production systems have trouble even
enabling 8K jumbo frames, because the kernel can't allocate two
physically contiguous buffers to use for receive.  Given that no
current IB hardware can offload TCP/IP checksums for multi-packet RC
messages, it's not clear how feasible this network stack surgery is.

Second, if the justification for IPoIB-CM is really for parallel file
systems, then it seems much more promising to run the file system
natively on InfiniBand.  For example, since Lustre is implemented on
top of the Portals abstraction, native implementations of Lustre have
been in use on IB, Myrinet, Quadrics, etc. for quite some time.

 - R.