Re: DISCUSS: draft-ietf-ipoib-connected-mode

[Cc'ing the IPoIB list for more comments.]

>Date: Thu, 02 Mar 2006 16:23:53 +0100
>From: Brian E Carpenter <[email protected]>

>
>This certainly a classical problem with LFNs and at least
>in some cases IB may look like an LFN. I agree with Allison

Not sure about the "L" part. IB is high-bandwidth but "low-latency".

>that this needs to be stated better. (In fact, people using
>this sort of technology for real firehose applications are
>often TCP-averse for this reason.)
>
>    Brian
>
>Allison Mankin wrote:
>> Discuss:
>> The IB folks won't be surprised by my popping up with this :)

That's right. It's been overdue :)

>> 
>>   The RC mode of InfiniBand guarantees in-order delivery of
>>   packets. Every message transmitted over the RC connection is
>>   broken into physical MTU sized packets by the RC connection. If
>>   any packet is lost, it is retransmitted until the complete
>>   message is exchanged. Therefore, there is a possibility of a
>>   reliable transport layer, such as TCP, retransmitting due to a
>>   shorter timeout, while the RC layer is still in the process of
>>   transferring the complete message. A retransmission at the upper
>>   layer will add to the already existing congestion.
>> 
>> This paragraph is important.  But the problem is not stated
>> quite right.  When a well-designed reliable transport retransmits
>> due to the RC layer, it will go into slow-start.  This means
>> it will take an unnecessary *enormous* performance hit.  There
>> will be a small amount of redundant data, but a very big loss of
>> throughput for the TCP, and long recovery.  And this pattern will
>> repeat.  The opposite will be true for shotgun-like retransmission
>> approaches such as the Skype UDP retries.  Those will send a lot of
>> extra data while the RC is still doing its transfer.  The traffic
>> over the link is heterogeneous and there's no way for the IB to
>> know which is there.

My understanding of your original concern is having more than one
layer doing retransmission at the same time can lead to more congestion
when there is a big gap of msg sizes between layers (e.g., ~64KB IPv4
datagram getting chopped up into 32 2048-byte IB datagrams, although
this case is not nearly as bad as IPoverATM).

>> 
>>   Therefore, the RC timers as well as the maximum message size
>>   supported at the IPoIB-RC connection must be set judiciously.
>> 
>> The links should minimize situations where they can cause a long delay
>> due to their retransmission.  They should use measurement-based

This is by design. The default IB RC timers operate at a value way
below TCP (or any ULP) timers, at micro second to sub milli-second level.
The belief (by the IB folks) is any pkt loss will most likely be
recovered by the link layer long before any transport level timer fires.

>> timers with starting minimum values that are short compared
>> with the starting minimum timers of the end-to-end transports. It's
>> too vague to say "set judiciously".  The guidance needs to be as
>> specific as the first two sentences here.

I agree the issue can be stated more generally, not just for TCP.

Jerry

IPoIB co-chair
Sr. Staff Engineer
Solaris Core Networking
Sun Microsystems, Inc.
U.S.A
(650) 786-5146

>> 
>> 
>> 
>