RE: A Couple of IPoIB Questions

Hi Mike,
My comments below.
-Dror

-----Original Message-----
From: Michael Krause [mailto:[email protected]] 
Sent: Tuesday, November 23, 2004 12:27 AM
To: IPoverIB
Subject: RE: [Ipoverib] A Couple of IPoIB Questions


At 01:36 PM 11/22/2004, Dror Goldenberg wrote:


Hi Mike,
 
Please see below.
 
Thanks
Dror


-----Original Message-----


From: Michael Krause [mailto:[email protected] <mailto:[email protected]> ] 


Sent: Monday, November 22, 2004 8:37 PM


To: IPoverIB


Subject: RE: [Ipoverib] A Couple of IPoIB Questions



At 08:49 AM 11/20/2004, Dror Goldenberg wrote:



-----Original Message----- 

From: Michael Krause [mailto:[email protected] <mailto:[email protected]> ] 

Sent: Friday, November 19, 2004 3:45 AM 

To: Vivek Kashyap 

Cc: IPoverIB 

Subject: Re: [Ipoverib] A Couple of IPoIB Questions




At 05:14 PM 11/18/2004, Vivek Kashyap wrote:





RC and UC both have benefits. There is almost no difference other than 

the connection flag between the two. 

Many host OS implementations do not support UC as RC and UD are all that is
really required within the industry.  The ACK overhead associated with RC is
truly noise and the end-to-end credits are very nice as IB now supports
three signaling rates combined with 4 link widths (though only three are
really being implemented).  Such a permutation in bandwidth capability makes
RC a more tenable / good citizen as we designed it to be so I'd prefer RC. 

[DG] Mike, 

 A few reasons I think that the end to end credits / RNR  in an RC
connection is a problem. 

It may be worth discussing it: 

1) Lack of receive WQEs in the responder implies a slow responder. Getting
the messaged dropped 

    in this case is desirable from protocols that have injection control
such as TCP.  In this case it 

    is supposed to back off and restart slowlier. While UC/UD result  in a
similar behavior of messages 

    being dropped at the receiver when it's slow, RC does not. Instead,
there is persistence in getting the 

    message transmitted and the receiver won't be able to tell the requester
that it's being slow.



TCP on the sending side will regulate due to lack of update window  credits.
Hence, there is no need to restart the large messages that are put forth as
the reason for using *C instead of UD. 



  

[dg] I think it'll be common to find very large TCP windows being
advertised.


A TCP window that is advertised is required to have the associated buffering
available.  While some implementations assume statistical provisioning in
the kernel, they assume that the application buffers are available and the
only problem is being able to move kernel buffers quickly enough to
application buffers which is a transient issue. 
 
[dg] Right, this is exactly the kind of implementations I refer to. These
tend to oversubscribe buffers both at the NIC level (i.e. there is much less
buffers posted to the RX of the NIC than the sum of TCP windows), and at the
TCP level. When such a machine is busy, then packets start dropping.


 Therefore, when you work against a very slow receiver, I think that it
makes sense to activate the TCP congestion mechanism rather than to rely on
the TCP window which is not intended to take care of congestion. Typically,
the overall advertised TCP windows (from all connections together) is much
more than actually being posted on the IPoIB QP receive queue. In a slow
receiver, the replenishment pace on receive WQEs is slow, and you'd want
remote senders to slow down when trying to fill its TCP Windows.



Dropping a buffer is fine but that should be at the TCP/IP level and not a
driver decision.  A driver should have sufficient buffers to avoid having
wasted the network bandwidth.  Hence, the driver should be posting
sufficient buffers to keep up with the workload which may span multiple
connections / datagrams.  Use of UC or RC does not change anything in this
regard.  A drop using UC would simply waste IB network bandwidth, consume
HCA resources flushing the work (the transmitter would continue to transmit
so nothing is saved there), etc. and only impact one connection at a time.
It does nothing for the rest of the connections.  So while one might get a
bit of benefit akin to a RED scheme, if the endnode pairs are operating at a
high workload, all one gets with UC is the ability of one endnode to flood
another with no push back except on random connections.  This would lead to
bursty behavior and unpredictable application responsiveness.  RC leads to
smoother performance between the endnode pair and with the use of multiple
RC QP, one can differentiate traffic for QoS purposes which is something
that will benefit applications. 
 
[dg] If you work with RC, then in the slow receiver case, backpressure will
propagate into the sender (RQ is full, no end to end credits are reflected,
peer SQ becomes full and you're out of SQ WQEs). In this case, what will you
do in the requester side ?
- Tell the upper TCP/IP layers that the NIC TX ring is full - this will
cause OS not to 
   post buffers to ANY of current RC connections. I don't think it's
desirable, it'll slow down / block
   your connections with the other remote peers
- Pretend as if there is still room in the SQ - but when OS posts to the
full SQ, you'll drop
    the packet -> this will be just the same as the UC case, except that you
do it in the sender
    instead of the receiver
- Pretend as if there is still room in the SQ - but when OS posts to the
full SQ, you'll queue
    it in SW. I think it'll risk shared resources.
 
What I am trying to say, is that we need to understand what happens in the
case of the slow receiver. I think that in RC what you'll end up having is
the peer requester dropping the packets. In UC, you'll get the responder
dropping the packets. As of how much you flood the IB fabric, see my comment
on the second question.

 


2) How would you configure the RNR retry counters. Would they be configured
to infinity ? Doesn't sound 

    good. Would they be configured to a finite value (should be <7), in
which case, in the case of a slow 

    receiver you'd end up recreating connections that had end to end credits
problem, which is a real 

    overhead on the protocol.



RNR would be no different for IP over IB than for any other IB RC instance.



[dg] Example ULPs such as SDP and SRP use SW level flow control and do not
rely on RNR NAKs.


These are also not IP based ULP.  



 What I am trying to say is if you configure your QP for finite retries and
a reasonable timeout, then when the receiver is slow, you'd often get the QP
into the error state, after RNR retries are exhausted. The overhead of
reestablishing a new connection each time the QP gets into the error state
is high.  If you use UC, then this is not a problem, because none of this
happens.



Given RC uses send credits and therefore should not see a new message unless
there is an associated buffer available which increments the credit count,
one should not get a RNR NAK ever.  The reason for RNR NAK was to deal with
a resource other than a receive buffer missing, e.g. QP context or V-to-P
translation or whatever not being chip resident and some time would be
required to refresh without going into the error state.  Given RC is still
send-receive based, there should not be any reason for a RNR NAK and no SEND
will occur unless a credit is provided.
 
[dg] yes and no. If you work with regular RC, then when RQ is empty, then
the peer SQ will send probing packets (e.g. send first/send only) to see if
credits became available. In this case you will see RNR Nak, but what you
inject to the fabric before getting it is a single packet. So I agree that
you don't flood the IB fabric in this case.
The issue with e2e credits reflection is when one wants to use SRQ, instead
of posting to each RQ separately and consuming many resources. In this case,
e2e credits are no longer reflected by ACK packets. And you're going to send
messages to the remote side without any flow control, and get RNR Naks when
peer RQ is empty. If most implementations use SRQ, then fabric is going to
be flooded anyways because of slow receivers.
I know that SRQ today is not allowed on UC, but that's a different story...
 



Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib