RE: Please read - proposed WG termination

At 01:02 PM 9/2/2005, Dror Goldenberg wrote:
>
>-----Original Message-----
>From: Michael Krause [mailto:[email protected]]
>Sent: Thursday, September 01, 2005 10:09 PM
>
>At 10:51 AM 9/1/2005, Vivek Kashyap wrote:
>>On Thu, 1 Sep 2005, Roland Dreier wrote:
>>
>>>    Bernard> In the HPC world most clusters use the cluster fabric (
>>>    Bernard> and IB is the future direction ) for both MPI and IP
>>>    Bernard> traffic. The IP traffic is usually for parallel file
>>>    Bernard> systems and system management and control. This high
>>>    Bernard> bandwidth IP network is required in most production HPC
>>>    Bernard> clusters.  With the current IPoIB only using UD, the
>>>    Bernard> performance is dismal. Our simulations using the small
>>>    Bernard> packet MTU of IB says that the parallel file systems (
>>>    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
>>>    Bernard> today and at 12X it will be about 10%.
>>>
>>>It's not clear to me that IPoIB-CM is really the answer here.  First
>>>of all, for Linux to take advantage of a 64K MTU, major surgery to the
>>>network stack will be required.  Production systems have trouble even
>>>enabling 8K jumbo frames, because the kernel can't allocate two
>>>physically contiguous buffers to use for receive.  Given that no
>>
>>That is an implementation issue in Linux and shouldn't effect the protocol.
>
>Linux does support large send off-load today so it is not clear what the 
>issue is you are speaking of above.
>
>[dg]  Large send offload solves the send side, but doesn't help on the 
>receive side. ipoib-cm solves both ends.
>Roland has a point that in Linux, it might be more complicated to 
>support >4KB frames. But I agree that this is an OS specific issue. In 
>this specific case, solving it may also be beneficial for other 
>applications such as Ethernet jumbo frames (if it makes any difference).

I think there is confusion perhaps here.  Most OS can map arbitrarily large 
buffers and provide V-to-P mappings that enable an I/O device to DMA large 
and variable amounts of data.  The 4KB limit is generally associated with 
the physical page size of many OS (though that too varies with some OS 
supporting up 1GB) as well as the physical maximum unit of DMA which is 4KB 
(PCI / PCI-X / PCIe).   Most device drivers / devices can also handle a SGL 
for a given target DMA address and sans separating on 4K address 
boundaries, can handle large DMA quite nicely.

The more interesting issues are:
- Must a large receive be handled as a single unit of work and thus must 
progress up the network stack as a single "message".  This has QoS and 
performance implications.  Obviously, iWARP and IB can perform such 
operations since that is the basis for RDMA as well as basic SEND 
operations so the basic hardware infrastructure is present.  The same can 
be said for various Ethernet implementations which support jumbo frames.

- Is the network stack limited in anyway in terms of message buffer size 
and therefore requires the implementation to post a set of message buffers 
which may be also SGL in order to receive the larger network sends?  This 
is where some of the complexity lies.

- Is there any impact to the recovery paths when a checksum, etc. requires 
only a portion of the large message to be retransmitted?  SACK, etc. are 
generally implemented but some implementations may make assumptions as the 
granularity of how the buffers are handled.

- etc.



>>>current IB hardware can offload TCP/IP checksums for multi-packet RC
>>>messages, it's not clear how feasible this network stack surgery is.
>>
>>Checksum offload is not part of the specification but an HCA feature.
>
>It is also not that hard to implement.
>
>[dg] Sounds right for ipoib-ud. For ipoib-cm csum offload implementation 
>is complicated. By the time you already transmitted 64KB, you need to go 
>and find the first packet (which is probably already in your peer' memory) 
>and place the csum there...
>  Anyway , I still see great value in ipoib-cm even without the csum 
> offload. The main reason is that to my knowledge, in Linux, large send 
> and zero copy are only supported for sendfile(). The rest of the 
> operations are on the copy path anyway. Calculating csum while copying, 
> is not a big deal.

It is true that copy-n-checksum is a de facto method to mitigate the 
performance impacts of not having checksum off-load support. Yet, there are 
many workloads where copy operations do not occur thus the performance hit 
is a big deal.  I don't think one can simply say IB connected provides 
superior performance compared to implementing something like a large send 
off-load, etc.

BTW, the new Sockets API extensions (async execution, event management, 
explicit memory management, etc.) enable more zero-copy opportunities as 
well as OS bypass.  It is important to keep in mind that an evaluation of 
whether something is of value needs a bit more context and future usages in 
mind.  In general, it is not clear why we are debating this so 
much.  People can define a spec and as we have seen  on multiple occasions 
either all, only a subset or none of the industry may implement even if the 
technology is superior in some facet.  If people really want to do this, 
then feel free to define but I suspect it might only appear on one OS at 
the end of the day and while that OS is relatively high volume for some 
usage models, that does not mean the spec will be deployed in practice or 
that it will be viewed as successful.  I personally do not believe the 
industry needs this problem solved as there are sufficiently good methods 
already available to solve the performance issues including using native 
RDMA ULP where performance is critical.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib