RE: A Couple of IPoIB Questions

At 11:54 PM 11/22/2004, Dror Goldenberg wrote:
>Hi Mike,
>My comments below.
>-Dror

<snip>

>Dropping a buffer is fine but that should be at the TCP/IP level and not a 
>driver decision.  A driver should have sufficient buffers to avoid having 
>wasted the network bandwidth.  Hence, the driver should be posting 
>sufficient buffers to keep up with the workload which may span multiple 
>connections / datagrams.  Use of UC or RC does not change anything in this 
>regard.  A drop using UC would simply waste IB network bandwidth, consume 
>HCA resources flushing the work (the transmitter would continue to 
>transmit so nothing is saved there), etc. and only impact one connection 
>at a time.  It does nothing for the rest of the connections.  So while one 
>might get a bit of benefit akin to a RED scheme, if the endnode pairs are 
>operating at a high workload, all one gets with UC is the ability of one 
>endnode to flood another with no push back except on random 
>connections.  This would lead to bursty behavior and unpredictable 
>application responsiveness.  RC leads to smoother performance between the 
>endnode pair and with the use of multiple RC QP, one can differentiate 
>traffic for QoS purposes which is something that will benefit applications.
>
>[dg] If you work with RC, then in the slow receiver case, backpressure 
>will propagate into the sender (RQ is full, no end to end credits are 
>reflected, peer SQ becomes full and you're out of SQ WQEs). In this case, 
>what will you do in the requester side ?

The SQ can become full and the send side driver can start to drop datagrams 
just like one does within any driver below IP.  This only impacts the one 
RC QP and not others (one of my reasons for wanting multiple RC between 
endnode pairs if performance is critical).   BTW, the same issue occurs if 
one were using Ethernet pause functionality and forward progress could not 
occur.

>- Tell the upper TCP/IP layers that the NIC TX ring is full - this will 
>cause OS not to
>    post buffers to ANY of current RC connections. I don't think it's 
> desirable, it'll slow down / block
>    your connections with the other remote peers

It is treated no differently than today's solutions.

>- Pretend as if there is still room in the SQ - but when OS posts to the 
>full SQ, you'll drop
>     the packet -> this will be just the same as the UC case, except that 
> you do it in the sender
>     instead of the receiver

What is wrong with this as it is aligns with today's solutions.

>- Pretend as if there is still room in the SQ - but when OS posts to the 
>full SQ, you'll queue
>     it in SW. I think it'll risk shared resources.

This is a local implementation choice and one that has been implemented in 
some OS.  This deals with thin hardware resources on a given device and 
works reasonable well under burst traffic.

>
>What I am trying to say, is that we need to understand what happens in the 
>case of the slow receiver. I think that in RC what you'll end up having is 
>the peer requester dropping the packets. In UC, you'll get the responder 
>dropping the packets. As of how much you flood the IB fabric, see my 
>comment on the second question.

 From what I know, it has always been the ph

>
>>>2) How would you configure the RNR retry counters. Would they be 
>>>configured to infinity ? Doesn't sound
>>>     good. Would they be configured to a finite value (should be <7), in 
>>> which case, in the case of a slow
>>>     receiver you'd end up recreating connections that had end to end 
>>> credits problem, which is a real
>>>     overhead on the protocol.
>>RNR would be no different for IP over IB than for any other IB RC 
>>instance.
>>[dg] Example ULPs such as SDP and SRP use SW level flow control and do 
>>not rely on RNR NAKs.
>These are also not IP based ULP.
>
>>  What I am trying to say is if you configure your QP for finite retries 
>> and a reasonable timeout, then when the receiver is slow, you'd often 
>> get the QP into the error state, after RNR retries are exhausted. The 
>> overhead of reestablishing a new connection each time the QP gets into 
>> the error state is high.  If you use UC, then this is not a problem, 
>> because none of this happens.
>Given RC uses send credits and therefore should not see a new message 
>unless there is an associated buffer available which increments the credit 
>count, one should not get a RNR NAK ever.  The reason for RNR NAK was to 
>deal with a resource other than a receive buffer missing, e.g. QP context 
>or V-to-P translation or whatever not being chip resident and some time 
>would be required to refresh without going into the error state.  Given RC 
>is still send-receive based, there should not be any reason for a RNR NAK 
>and no SEND will occur unless a credit is provided.
>
>[dg] yes and no. If you work with regular RC, then when RQ is empty, then 
>the peer SQ will send probing packets (e.g. send first/send only) to see 
>if credits became available. In this case you will see RNR Nak, but what 
>you inject to the fabric before getting it is a single packet. So I agree 
>that you don't flood the IB fabric in this case.
>The issue with e2e credits reflection is when one wants to use SRQ, 
>instead of posting to each RQ separately and consuming many resources. In 
>this case, e2e credits are no longer reflected by ACK packets. And you're 
>going to send messages to the remote side without any flow control, and 
>get RNR Naks when peer RQ is empty. If most implementations use SRQ, then 
>fabric is going to be flooded anyways because of slow receivers.
>I know that SRQ today is not allowed on UC, but that's a different story...

So you are arguing for SRQ which is not supported by UC.   I don't know of 
SRQ has value or not but if you want to discuss SRQ, then let's discuss 
that and not RC vs. UC as I don't think the counter arguments against RC 
are significant.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib