RE: ipoib-cm multiple connection model

Hi folks,

Have we reached a concensus the connection setup procedure in the
current draft is "good enough", or folks still think it needs to be
improved?

I'd like to see any remaining issue resolved asap so we can start a WG
last call (after Vivke revs the draft based on all the comments so far).

Jerry

>Date: Thu, 27 Oct 2005 22:47:16 -0700 (PDT)
>From: Vivek Kashyap <[email protected]>
>X-X-Sender: [email protected]
>To: Dror Goldenberg <[email protected]>
>Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>MIME-Version: 1.0
>X-Spam-Score: 0.0 (/)
>X-Scan-Signature: f8ee348dcc4be4a59bc395f7cd6343ad
>Cc: "'[email protected]'" <[email protected]>, Vandana Rao <[email protected]>
>X-BeenThere: [email protected]
>X-Mailman-Version: 2.1.5
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>, 
<mailto:[email protected]?subject=unsubscribe>
>List-Post: <mailto:[email protected]>
>List-Help: <mailto:[email protected]?subject=help>
>List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>, 
<mailto:[email protected]?subject=subscribe>
>
>Hi Dror,
>
>On Thu, 27 Oct 2005, Dror Goldenberg wrote:
>
>> Hi Vivek,
>>
>> I agree with the flows that you describe, and with the fact that the
>> original
>> requester has always the "second chance" to withdraw its request by
>> rejecting the REP. However, in my opinion, giving all this freedom to
>> the implementation can lead to interoperability issues and poor
>> implementations. If an implementation tries to open n connections with
>> a certain peer, it may end up opening more or less than n and that
>> might take few iterations and that all depends on the CM timing and
>> some other "unrelated" parameters - how can you even test it ?
>> I therefore feel that we should provide a robust and deterministic
>> specification of how to establish more than one connection in the
>> RFC or live with a single connection.
>
>I'd like to delve into this a bit more to be sure. As per your/Vandana's
>comments, the left peer will cancel the request. Isn't this is a 'conscious' 
>decision by the ULP? The ULP would then remove the its internal object
>that it would associate with this IB connection. If it was successful
>in cancelling the REQ then there is only one connection.
>
>If, however, as you pointed out it is unable to, it might receive a REP
>*if* the remote peer accepts multiple connections. The left peer's ulp
>however will not find any object associated with the private data and 
>the connection request (there will be another one already existing though)
>and can use that to indicate rejection.
>
>Does the above sound right?
>
>Vivek
>
>>
>> BTW, this is my opinion in this case, and if folks feel OK with the
>> current draft, then maybe we should move on ?
>>
>> Thanks
>> Dror
>>
>>> -----Original Message-----
>>> From: Vivek Kashyap [mailto:[email protected]]
>>> Sent: Wednesday, October 26, 2005 10:33 PM
>>> To: Dror Goldenberg
>>> Cc: Vandana Rao; '[email protected]'
>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>>>
>>>
>>> On Wed, 26 Oct 2005, Dror Goldenberg wrote:
>>>
>>>> Vivek,
>>>>
>>>> I agree that the two scenarios you mention here are the interesting
>>>> ones. I don't agree about the solution. Let's look at the two cases:
>>>>
>>>> 1) Left peer accepts
>>>> 	I disagree. You have an assumption that left peer can
>>>> 	revoke its original connection request - BEFORE transmitting
>>>> 	the REP that accepts the peer request. I am not sure that
>>>> 	you can do it with every CM implementation. Even if you can,
>>>> 	then the HCA can reorder those packets if they are on
>>>> 	different QPs, and the fabric can reorder them as well if
>>>> 	they are on different VLs.
>>>
>>> Yes, that has the potential for extra connections..however,
>>> the following will occur if CM checks don't work.
>>>
>>> Left peer accepted. Sent a REP. And of course the REQ was
>>> also sent. The right receiver can receive :
>>>
>>>  	a. REP first
>>>
>>>  	It sets up the connection. Then receives a REQ.
>>>
>>>  	i) If it accepts multiple connections from the same peer:
>>>  		 it will  REP with a new QP (connection).
>>>
>>>  		Now, if the left peer decides that it didn't
>>> need another
>>>  		connection it terminates the connection.
>>>
>>>  		Otherwise, it is legitimate to setup multiple
>>> connections
>>>  		since it is equivalent to a second valid request.
>>>
>>>  	ii) If the right peer does not accept multiple
>>> connections from the
>>>  	same peer then it will reject the request.
>>>
>>>  	Note that these decisions in part depend on the ipoib
>>> component peeking
>>>  	into the local link address and not on CM.
>>>
>>>
>>>  	b. REQ first
>>>
>>>  	Since its own REQ is outstanding, and as per link
>>> address comparison,
>>>  	the left peer accepts then this will be rejected.
>>>
>>> Vivek
>>>>
>>>> 2) Left peer rejects
>>>> 	I think that this scenario works.
>>>>
>>>> Thanks
>>>> Dror
>>>>
>>>>> -----Original Message-----
>>>>> From: Vivek Kashyap [mailto:[email protected]]
>>>>> Sent: Tuesday, October 25, 2005 11:38 PM
>>>>> To: Vandana Rao
>>>>> Cc: Dror Goldenberg; '[email protected]'
>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>>>>>
>>>>>
>>>>> Hi Vandana,
>>>>>
>>>>> I too agree there is no problem here.
>>>>>
>>>>> I think you are right if the left peer accepts the connection.
>>>>> However, if the reverse was the case, that is the left
>>> peer rejected
>>>>> the connection since it is found that the local address is
>>>>> numerically larger. It will do this comparison since its
>>> REQ is also
>>>>> outstanding.  The right peer will not receive any REQ  (it
>>> is delayed
>>>>> or yet to be retransmitted). If it receives the reject it
>>> will cancel
>>>>> its request. At a later time it will receive the
>>>>> delayed/retransmitted request and one connection will be
>>>>> established.
>>>>>
>>>>> Dror, do these two scenarios look right to you?
>>>>>
>>>>> Vivek
>>>>>
>>>>> On Thu, 20 Oct 2005, Vandana Rao wrote:
>>>>>
>>>>>> Hi Dror,
>>>>>> I think you are confusing the 2 IDs that come into play in
>>>>> this setup.
>>>>>> The
>>>>>> communication ID is something that the CM entity on a node
>>>>> understands but
>>>>>> the transaction ID is something that the MAD handling
>>>>> entity uses (for
>>>>>> retransmission, req/resp matchup etc.).
>>>>>> So, in the second slide, there should not be a
>>>>> retransmission of the left
>>>>>> peer REQ MAD since when the left peer accepts the
>>>>> connection, it will cancel
>>>>>> its outstanding REQ request and hence the retransmission
>>>>> should not take
>>>>>> place.
>>>>>> -Vandana
>>>>>>
>>>>>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
>>>>>>> After giving it some more thought, I think that there is still a
>>>>>>> problem. Please look at the attached pdf.
>>>>>>>
>>>>>>> In the first slide, everything happens smoothly, the left
>>>>> peer knows
>>>>>>> that
>>>>>>> it has to accept the connection and the right peer knows
>>>>> that it has to
>>>>>>> decline.
>>>>>>> On the 2nd slide, the left peer REQ transmission is
>>>>> dropped in the fabric.
>>>>>>> It is retransmitted independently by the CM way after a
>>>>> channel has already
>>>>>>> been established. Now the right side can not really tell
>>>>> whether this is a
>>>>>>> new channel request or an old "simultaneous connection
>>>>> channel". This may
>>>>>>> lead to two channels being opened mistakenly.
>>>>>>>
>>>>>>> I don't believe it can be solved by looking at the
>>>>> communication ID.
>>>>>>> This
>>>>>>> number is just being fabricated by the sender of the REQ
>>>>> and is just a
>>>>>>> "random number" with respect to what we're trying to determine.
>>>>>>>
>>>>>>> -Dror
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Dror Goldenberg
>>>>>>> Sent: Wednesday, October 19, 2005 9:37 PM
>>>>>>> To: 'Vandana Rao'; Dror Goldenberg; [email protected]
>>>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>>>>>>>
>>>>>>> Hi Vandana,
>>>>>>>
>>>>>>> Thanks for clarifying this. I agree that if IPoiB-CM
>>>>> implementation
>>>>>>> looks
>>>>>>> at the Communication ID it can make those decisions
>>>>> correctly. The question
>>>>>>> is whether this value is exposed through common CM APIs.
>>>>> For all other
>>>>>>> purposes, the CM doesn't need to expose the Communication
>>>>> ID to the ULP.
>>>>>>> However, seems like it's not a big deal to expose this
>>>>> value to the ULP.
>>>>>>> Anyway, it worth noting this explanation in the RFC so
>>>>> that it's clear that
>>>>>>> a proper implementation of IPoIB-CM should monitor
>>>>> Communication ID as part
>>>>>>> of the simultaneous connection decision.
>>>>>>>
>>>>>>> -Dror
>>>>>>> -----Original Message-----
>>>>>>> From: Vandana Rao [mailto:[email protected]]
>>>>>>> Sent: Wednesday, October 19, 2005 6:40 PM
>>>>>>> To: Dror Goldenberg; [email protected]
>>>>>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
>>>>>>>
>>>>>>> Hi Dror,
>>>>>>> The purpose of the MAD transaction ID is to allow one to
>>>>> do what you
>>>>>>> say
>>>>>>> below, i.e. have multiple connection establishment MADs
>>>>> between 2 peers. It
>>>>>>> is essentially the same as the "channel ID" that you talk
>>>>> about below. Its
>>>>>>> just that it is a generic channel ID for any MAD
>>>>> transactions, not just CM
>>>>>>> MADs.
>>>>>>> I do not believe there is any issue with existing IB CM in
>>>>> this regard.
>>>>>>> -Vandana
>>>>>>>
>>>>>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
>>>>>>>> I think that the current model for multiple connections
>>>>>>>> establishment between
>>>>>>>> two peers works well for 1 connection but is broken for
>>>>> more than one
>>>>>>>> connection.
>>>>>>>> Because of cases where CM packets are dropped or travel
>>>>> on different VLs,
>>>>>>>> they
>>>>>>>> might get reordered on the way and observed in a
>>>>> different order than were
>>>>>>>> originally
>>>>>>>> sent. This breaks the current model which assumes that
>>>>> the REQs are sent
>>>>>>>> and
>>>>>>>> processed in order.
>>>>>>>>
>>>>>>>> I think that we need to think of a model which is robust to CM
>>>>>>>> packets reordering
>>>>>>>> which can be one of:
>>>>>>>> - Adding a "channel ID" in the private data. The field
>>>>> goes 0,1,2 and on.
>>>>>>>> It helps
>>>>>>>>    synchronizing both peers on which is the channel at
>>>>> question now. The
>>>>>>>> rest
>>>>>>>>    of the decision (per channel) of whether to accept or
>>>>> reject can be the
>>>>>>>> same
>>>>>>>>    as defined in the draft.
>>>>>>>> - Not supporting multiple connections between the same peers.
>>>>>>>>
>>>>>>>> -Dror
>>>>>>>> _______________________________________________
>>>>>>>> IPoverIB mailing list
>>>>>>>> [email protected]
>>> https://www1.ietf.org/mailman/listinfo/ipoverib
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>_______________________________________________
>IPoverIB mailing list
>[email protected]
>https://www1.ietf.org/mailman/listinfo/ipoverib