RE: A Couple of IPoIB Questions

Hi Mike,
 
Please see below.
 
Thanks
Dror

-----Original Message-----
From: Michael Krause [mailto:[email protected]] 
Sent: Monday, November 22, 2004 8:37 PM
To: IPoverIB
Subject: RE: [Ipoverib] A Couple of IPoIB Questions


At 08:49 AM 11/20/2004, Dror Goldenberg wrote:





-----Original Message-----


From: Michael Krause [mailto:[email protected] <mailto:[email protected]> ] 


Sent: Friday, November 19, 2004 3:45 AM


To: Vivek Kashyap


Cc: IPoverIB


Subject: Re: [Ipoverib] A Couple of IPoIB Questions




At 05:14 PM 11/18/2004, Vivek Kashyap wrote:





RC and UC both have benefits. There is almost no difference other than


the connection flag between the two. 


Many host OS implementations do not support UC as RC and UD are all that is
really required within the industry.  The ACK overhead associated with RC is
truly noise and the end-to-end credits are very nice as IB now supports
three signaling rates combined with 4 link widths (though only three are
really being implemented).  Such a permutation in bandwidth capability makes
RC a more tenable / good citizen as we designed it to be so I'd prefer RC. 



[DG] Mike,


 A few reasons I think that the end to end credits / RNR  in an RC
connection is a problem.


It may be worth discussing it:


1) Lack of receive WQEs in the responder implies a slow responder. Getting
the messaged dropped 


    in this case is desirable from protocols that have injection control
such as TCP.  In this case it


    is supposed to back off and restart slowlier. While UC/UD result  in a
similar behavior of messages


    being dropped at the receiver when it's slow, RC does not. Instead,
there is persistence in getting the


    message transmitted and the receiver won't be able to tell the requester
that it's being slow.


TCP on the sending side will regulate due to lack of update window  credits.
Hence, there is no need to restart the large messages that are put forth as
the reason for using *C instead of UD. 
 
[dg] I think it'll be common to find very large TCP windows being
advertised. Therefore, when you work against a very slow receiver, I think
that it makes sense to activate the TCP congestion mechanism rather than to
rely on the TCP window which is not intended to take care of congestion.
Typically, the overall advertised TCP windows (from all connections
together) is much more than actually being posted on the IPoIB QP receive
queue. In a slow receiver, the replenishment pace on receive WQEs is slow,
and you'd want remote senders to slow down when trying to fill its TCP
Windows.



2) How would you configure the RNR retry counters. Would they be configured
to infinity ? Doesn't sound


    good. Would they be configured to a finite value (should be <7), in
which case, in the case of a slow


    receiver you'd end up recreating connections that had end to end credits
problem, which is a real 


    overhead on the protocol.


RNR would be no different for IP over IB than for any other IB RC instance.

[dg] Example ULPs such as SDP and SRP use SW level flow control and do not
rely on RNR NAKs. What I am trying to say is if you configure your QP for
finite retries and a reasonable timeout, then when the receiver is slow,
you'd often get the QP into the error state, after RNR retries are
exhausted. The overhead of reestablishing a new connection each time the QP
gets into the error state is high.  If you use UC, then this is not a
problem, because none of this happens.
 



3) What happens with implementations that don't support RNR Nak generation ?
That poses more


    difficulties on (2).



A HCA is required to support RNR NAK.  A TCA has the option.  If you don't
support RC, then use UD.  Where is the real problem as nothing shown here on
either side is more than speculation?
[dg] Agree, there isn't a real problem here.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib