RE: ipoib-cm multiple connection model

I think it's OK. We can move forward from my perspective.

> -----Original Message-----
> From: H.K. Jerry Chu [mailto:[email protected]]=20
> Sent: Monday, November 14, 2005 9:34 PM
> To: [email protected]; [email protected]
> Cc: [email protected]; [email protected]
> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>=20
>=20
> Hi folks,
>=20
> Have we reached a concensus the connection setup procedure in=20
> the current draft is "good enough", or folks still think it=20
> needs to be improved?
>=20
> I'd like to see any remaining issue resolved asap so we can=20
> start a WG last call (after Vivke revs the draft based on all=20
> the comments so far).
>=20
> Jerry
>=20
> >Date: Thu, 27 Oct 2005 22:47:16 -0700 (PDT)
> >From: Vivek Kashyap <[email protected]>
> >X-X-Sender: [email protected]
> >To: Dror Goldenberg <[email protected]>
> >Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >MIME-Version: 1.0
> >X-Spam-Score: 0.0 (/)
> >X-Scan-Signature: f8ee348dcc4be4a59bc395f7cd6343ad
> >Cc: "'[email protected]'" <[email protected]>, Vandana Rao=20
> ><[email protected]>
> >X-BeenThere: [email protected]
> >X-Mailman-Version: 2.1.5
> >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> >List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,=20
> <mailto:[email protected]?subject=3Dunsubscribe>
> >List-Post: <mailto:[email protected]>
> >List-Help: <mailto:[email protected]?subject=3Dhelp>
> >List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
> <mailto:[email protected]?subject=3Dsubscribe>
> >
> >Hi Dror,
> >
> >On Thu, 27 Oct 2005, Dror Goldenberg wrote:
> >
> >> Hi Vivek,
> >>
> >> I agree with the flows that you describe, and with the=20
> fact that the=20
> >> original requester has always the "second chance" to withdraw its=20
> >> request by rejecting the REP. However, in my opinion,=20
> giving all this=20
> >> freedom to the implementation can lead to interoperability=20
> issues and=20
> >> poor implementations. If an implementation tries to open n=20
> >> connections with a certain peer, it may end up opening=20
> more or less=20
> >> than n and that might take few iterations and that all=20
> depends on the=20
> >> CM timing and some other "unrelated" parameters - how can you even=20
> >> test it ? I therefore feel that we should provide a robust and=20
> >> deterministic specification of how to establish more than one=20
> >> connection in the RFC or live with a single connection.
> >
> >I'd like to delve into this a bit more to be sure. As per=20
> >your/Vandana's comments, the left peer will cancel the=20
> request. Isn't=20
> >this is a 'conscious' decision by the ULP? The ULP would then remove=20
> >the its internal object that it would associate with this IB=20
> >connection. If it was successful in cancelling the REQ then there is=20
> >only one connection.
> >
> >If, however, as you pointed out it is unable to, it might=20
> receive a REP
> >*if* the remote peer accepts multiple connections. The left=20
> peer's ulp=20
> >however will not find any object associated with the private=20
> data and=20
> >the connection request (there will be another one already existing=20
> >though) and can use that to indicate rejection.
> >
> >Does the above sound right?
> >
> >Vivek
> >
> >>
> >> BTW, this is my opinion in this case, and if folks feel OK=20
> with the=20
> >> current draft, then maybe we should move on ?
> >>
> >> Thanks
> >> Dror
> >>
> >>> -----Original Message-----
> >>> From: Vivek Kashyap [mailto:[email protected]]
> >>> Sent: Wednesday, October 26, 2005 10:33 PM
> >>> To: Dror Goldenberg
> >>> Cc: Vandana Rao; '[email protected]'
> >>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>
> >>>
> >>> On Wed, 26 Oct 2005, Dror Goldenberg wrote:
> >>>
> >>>> Vivek,
> >>>>
> >>>> I agree that the two scenarios you mention here are the=20
> interesting=20
> >>>> ones. I don't agree about the solution. Let's look at the two=20
> >>>> cases:
> >>>>
> >>>> 1) Left peer accepts
> >>>> 	I disagree. You have an assumption that left peer can
> >>>> 	revoke its original connection request - BEFORE transmitting
> >>>> 	the REP that accepts the peer request. I am not sure that
> >>>> 	you can do it with every CM implementation. Even if you can,
> >>>> 	then the HCA can reorder those packets if they are on
> >>>> 	different QPs, and the fabric can reorder them as well if
> >>>> 	they are on different VLs.
> >>>
> >>> Yes, that has the potential for extra connections..however, the=20
> >>> following will occur if CM checks don't work.
> >>>
> >>> Left peer accepted. Sent a REP. And of course the REQ was=20
> also sent.=20
> >>> The right receiver can receive :
> >>>
> >>>  	a. REP first
> >>>
> >>>  	It sets up the connection. Then receives a REQ.
> >>>
> >>>  	i) If it accepts multiple connections from the same peer:
> >>>  		 it will  REP with a new QP (connection).
> >>>
> >>>  		Now, if the left peer decides that it didn't
> >>> need another
> >>>  		connection it terminates the connection.
> >>>
> >>>  		Otherwise, it is legitimate to setup multiple=20
> connections
> >>>  		since it is equivalent to a second valid request.
> >>>
> >>>  	ii) If the right peer does not accept multiple connections from=20
> >>> the
> >>>  	same peer then it will reject the request.
> >>>
> >>>  	Note that these decisions in part depend on the ipoib component=20
> >>> peeking
> >>>  	into the local link address and not on CM.
> >>>
> >>>
> >>>  	b. REQ first
> >>>
> >>>  	Since its own REQ is outstanding, and as per link
> >>> address comparison,
> >>>  	the left peer accepts then this will be rejected.
> >>>
> >>> Vivek
> >>>>
> >>>> 2) Left peer rejects
> >>>> 	I think that this scenario works.
> >>>>
> >>>> Thanks
> >>>> Dror
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Vivek Kashyap [mailto:[email protected]]
> >>>>> Sent: Tuesday, October 25, 2005 11:38 PM
> >>>>> To: Vandana Rao
> >>>>> Cc: Dror Goldenberg; '[email protected]'
> >>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>
> >>>>>
> >>>>> Hi Vandana,
> >>>>>
> >>>>> I too agree there is no problem here.
> >>>>>
> >>>>> I think you are right if the left peer accepts the connection.=20
> >>>>> However, if the reverse was the case, that is the left
> >>> peer rejected
> >>>>> the connection since it is found that the local address is=20
> >>>>> numerically larger. It will do this comparison since its
> >>> REQ is also
> >>>>> outstanding.  The right peer will not receive any REQ  (it
> >>> is delayed
> >>>>> or yet to be retransmitted). If it receives the reject it
> >>> will cancel
> >>>>> its request. At a later time it will receive the=20
> >>>>> delayed/retransmitted request and one connection will be=20
> >>>>> established.
> >>>>>
> >>>>> Dror, do these two scenarios look right to you?
> >>>>>
> >>>>> Vivek
> >>>>>
> >>>>> On Thu, 20 Oct 2005, Vandana Rao wrote:
> >>>>>
> >>>>>> Hi Dror,
> >>>>>> I think you are confusing the 2 IDs that come into play in
> >>>>> this setup.
> >>>>>> The
> >>>>>> communication ID is something that the CM entity on a node
> >>>>> understands but
> >>>>>> the transaction ID is something that the MAD handling
> >>>>> entity uses (for
> >>>>>> retransmission, req/resp matchup etc.).
> >>>>>> So, in the second slide, there should not be a
> >>>>> retransmission of the left
> >>>>>> peer REQ MAD since when the left peer accepts the
> >>>>> connection, it will cancel
> >>>>>> its outstanding REQ request and hence the retransmission
> >>>>> should not take
> >>>>>> place.
> >>>>>> -Vandana
> >>>>>>
> >>>>>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
> >>>>>>> After giving it some more thought, I think that there=20
> is still a=20
> >>>>>>> problem. Please look at the attached pdf.
> >>>>>>>
> >>>>>>> In the first slide, everything happens smoothly, the left
> >>>>> peer knows
> >>>>>>> that
> >>>>>>> it has to accept the connection and the right peer knows
> >>>>> that it has to
> >>>>>>> decline.
> >>>>>>> On the 2nd slide, the left peer REQ transmission is
> >>>>> dropped in the fabric.
> >>>>>>> It is retransmitted independently by the CM way after a
> >>>>> channel has already
> >>>>>>> been established. Now the right side can not really tell
> >>>>> whether this is a
> >>>>>>> new channel request or an old "simultaneous connection
> >>>>> channel". This may
> >>>>>>> lead to two channels being opened mistakenly.
> >>>>>>>
> >>>>>>> I don't believe it can be solved by looking at the
> >>>>> communication ID.
> >>>>>>> This
> >>>>>>> number is just being fabricated by the sender of the REQ
> >>>>> and is just a
> >>>>>>> "random number" with respect to what we're trying to=20
> determine.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Dror Goldenberg
> >>>>>>> Sent: Wednesday, October 19, 2005 9:37 PM
> >>>>>>> To: 'Vandana Rao'; Dror Goldenberg; [email protected]
> >>>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Vandana,
> >>>>>>>
> >>>>>>> Thanks for clarifying this. I agree that if IPoiB-CM
> >>>>> implementation
> >>>>>>> looks
> >>>>>>> at the Communication ID it can make those decisions
> >>>>> correctly. The question
> >>>>>>> is whether this value is exposed through common CM APIs.
> >>>>> For all other
> >>>>>>> purposes, the CM doesn't need to expose the Communication
> >>>>> ID to the ULP.
> >>>>>>> However, seems like it's not a big deal to expose this
> >>>>> value to the ULP.
> >>>>>>> Anyway, it worth noting this explanation in the RFC so
> >>>>> that it's clear that
> >>>>>>> a proper implementation of IPoIB-CM should monitor
> >>>>> Communication ID as part
> >>>>>>> of the simultaneous connection decision.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>> -----Original Message-----
> >>>>>>> From: Vandana Rao [mailto:[email protected]]
> >>>>>>> Sent: Wednesday, October 19, 2005 6:40 PM
> >>>>>>> To: Dror Goldenberg; [email protected]
> >>>>>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Dror,
> >>>>>>> The purpose of the MAD transaction ID is to allow one to
> >>>>> do what you
> >>>>>>> say
> >>>>>>> below, i.e. have multiple connection establishment MADs
> >>>>> between 2 peers. It
> >>>>>>> is essentially the same as the "channel ID" that you talk
> >>>>> about below. Its
> >>>>>>> just that it is a generic channel ID for any MAD
> >>>>> transactions, not just CM
> >>>>>>> MADs.
> >>>>>>> I do not believe there is any issue with existing IB CM in
> >>>>> this regard.
> >>>>>>> -Vandana
> >>>>>>>
> >>>>>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
> >>>>>>>> I think that the current model for multiple connections=20
> >>>>>>>> establishment between two peers works well for 1=20
> connection but=20
> >>>>>>>> is broken for
> >>>>> more than one
> >>>>>>>> connection.
> >>>>>>>> Because of cases where CM packets are dropped or travel
> >>>>> on different VLs,
> >>>>>>>> they
> >>>>>>>> might get reordered on the way and observed in a
> >>>>> different order than were
> >>>>>>>> originally
> >>>>>>>> sent. This breaks the current model which assumes that
> >>>>> the REQs are sent
> >>>>>>>> and
> >>>>>>>> processed in order.
> >>>>>>>>
> >>>>>>>> I think that we need to think of a model which is=20
> robust to CM=20
> >>>>>>>> packets reordering which can be one of:
> >>>>>>>> - Adding a "channel ID" in the private data. The field
> >>>>> goes 0,1,2 and on.
> >>>>>>>> It helps
> >>>>>>>>    synchronizing both peers on which is the channel at
> >>>>> question now. The
> >>>>>>>> rest
> >>>>>>>>    of the decision (per channel) of whether to accept or
> >>>>> reject can be the
> >>>>>>>> same
> >>>>>>>>    as defined in the draft.
> >>>>>>>> - Not supporting multiple connections between the same peers.
> >>>>>>>>
> >>>>>>>> -Dror
> >>>>>>>> _______________________________________________
> >>>>>>>> IPoverIB mailing list
> >>>>>>>> [email protected]
> >>> https://www1.ietf.org/mailman/listinfo/ipoverib
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email protected] https://www1.ietf.org/mailman/listinfo/ipoverib
>=20