RE: A Couple of IPoIB Questions

At 01:36 PM 11/22/2004, Dror Goldenberg wrote:
>Hi Mike,
>
>Please see below.
>
>Thanks
>Dror
>-----Original Message-----
>From: Michael Krause [mailto:[email protected]]
>Sent: Monday, November 22, 2004 8:37 PM
>To: IPoverIB
>Subject: RE: [Ipoverib] A Couple of IPoIB Questions
>
>At 08:49 AM 11/20/2004, Dror Goldenberg wrote:
>>-----Original Message-----
>>From: Michael Krause [mailto:[email protected]]
>>Sent: Friday, November 19, 2004 3:45 AM
>>To: Vivek Kashyap
>>Cc: IPoverIB
>>Subject: Re: [Ipoverib] A Couple of IPoIB Questions
>>
>>At 05:14 PM 11/18/2004, Vivek Kashyap wrote:
>>
>>
>>>RC and UC both have benefits. There is almost no difference other than
>>>the connection flag between the two.
>>Many host OS implementations do not support UC as RC and UD are all that 
>>is really required within the industry.  The ACK overhead associated with 
>>RC is truly noise and the end-to-end credits are very nice as IB now 
>>supports three signaling rates combined with 4 link widths (though only 
>>three are really being implemented).  Such a permutation in bandwidth 
>>capability makes RC a more tenable / good citizen as we designed it to be 
>>so I'd prefer RC.
>>[DG] Mike,
>>  A few reasons I think that the end to end credits / RNR  in an RC 
>> connection is a problem.
>>It may be worth discussing it:
>>1) Lack of receive WQEs in the responder implies a slow responder. 
>>Getting the messaged dropped
>>     in this case is desirable from protocols that have injection control 
>> such as TCP.  In this case it
>>     is supposed to back off and restart slowlier. While UC/UD result  in 
>> a similar behavior of messages
>>     being dropped at the receiver when it's slow, RC does not. Instead, 
>> there is persistence in getting the
>>     message transmitted and the receiver won't be able to tell the 
>> requester that it's being slow.
>
>TCP on the sending side will regulate due to lack of update 
>window  credits.   Hence, there is no need to restart the large messages 
>that are put forth as the reason for using *C instead of UD.
>
>[dg] I think it'll be common to find very large TCP windows being advertised.

A TCP window that is advertised is required to have the associated 
buffering available.  While some implementations assume statistical 
provisioning in the kernel, they assume that the application buffers are 
available and the only problem is being able to move kernel buffers quickly 
enough to application buffers which is a transient issue.

>  Therefore, when you work against a very slow receiver, I think that it 
> makes sense to activate the TCP congestion mechanism rather than to rely 
> on the TCP window which is not intended to take care of congestion. 
> Typically, the overall advertised TCP windows (from all connections 
> together) is much more than actually being posted on the IPoIB QP receive 
> queue. In a slow receiver, the replenishment pace on receive WQEs is 
> slow, and you'd want remote senders to slow down when trying to fill its 
> TCP Windows.

Dropping a buffer is fine but that should be at the TCP/IP level and not a 
driver decision.  A driver should have sufficient buffers to avoid having 
wasted the network bandwidth.  Hence, the driver should be posting 
sufficient buffers to keep up with the workload which may span multiple 
connections / datagrams.  Use of UC or RC does not change anything in this 
regard.  A drop using UC would simply waste IB network bandwidth, consume 
HCA resources flushing the work (the transmitter would continue to transmit 
so nothing is saved there), etc. and only impact one connection at a 
time.  It does nothing for the rest of the connections.  So while one might 
get a bit of benefit akin to a RED scheme, if the endnode pairs are 
operating at a high workload, all one gets with UC is the ability of one 
endnode to flood another with no push back except on random 
connections.  This would lead to bursty behavior and unpredictable 
application responsiveness.  RC leads to smoother performance between the 
endnode pair and with the use of multiple RC QP, one can differentiate 
traffic for QoS purposes which is something that will benefit applications.

>>2) How would you configure the RNR retry counters. Would they be 
>>configured to infinity ? Doesn't sound
>>     good. Would they be configured to a finite value (should be <7), in 
>> which case, in the case of a slow
>>     receiver you'd end up recreating connections that had end to end 
>> credits problem, which is a real
>>     overhead on the protocol.
>
>RNR would be no different for IP over IB than for any other IB RC instance.
>[dg] Example ULPs such as SDP and SRP use SW level flow control and do not 
>rely on RNR NAKs.

These are also not IP based ULP.

>  What I am trying to say is if you configure your QP for finite retries 
> and a reasonable timeout, then when the receiver is slow, you'd often get 
> the QP into the error state, after RNR retries are exhausted. The 
> overhead of reestablishing a new connection each time the QP gets into 
> the error state is high.  If you use UC, then this is not a problem, 
> because none of this happens.

Given RC uses send credits and therefore should not see a new message 
unless there is an associated buffer available which increments the credit 
count, one should not get a RNR NAK ever.  The reason for RNR NAK was to 
deal with a resource other than a receive buffer missing, e.g. QP context 
or V-to-P translation or whatever not being chip resident and some time 
would be required to refresh without going into the error state.  Given RC 
is still send-receive based, there should not be any reason for a RNR NAK 
and no SEND will occur unless a credit is provided.

>
>>3) What happens with implementations that don't support RNR Nak 
>>generation ? That poses more
>>     difficulties on (2).
>
>A HCA is required to support RNR NAK.  A TCA has the option.  If you don't 
>support RC, then use UD.  Where is the real problem as nothing shown here 
>on either side is more than speculation?
>[dg] Agree, there isn't a real problem here.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib