RE: Comments on IPoIB Connected Mode Connection Establ ishment
Dror Goldenberg <[email protected]> Sun, 11 Sep 2005 08:23:41 +0300
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0438489395== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5B690.F8CC94B0" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5B690.F8CC94B0 Content-Type: text/plain Sounds like we agree that active/active is very difficult and will be really hard to define given how IB service ID and communication manager work. Therefore, I think that we should go for the active/passive with the ability to reject at the application (ipoib) level. For the case of more than one connection between two peers, we first need to decide whether we want to allow it (now or in the future). If we disallow it, then it's very simple, for each crossing request we need to decide which side should accept and which side should reject it (e.g. GID compare). If we want to allow more than one connection in the future, then maybe we should think of embedding a "channel index" into the private data. This way, the two peers can add the channel index on which they are transmitting. BTW, it'll take more than that to enable more than one channel, for example, we should try to avoid reordering of IP datagrams that can happen when more than one channel is being used to send them. What about connection termination ? This should probably be added to the RFC draft. I can think of two cases where one would want to terminate a connection. 1) When the ipoib driver goes down; 2) Inactive connections can be taken down to reduce resource consumption. -Dror > -----Original Message----- > From: Vivek Kashyap [mailto:[email protected]] > Sent: Tuesday, September 06, 2005 10:40 AM > To: Dror Goldenberg > Cc: IPoverIB > Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode > Connection Establ ishment > > > On Sun, 21 Aug 2005, Dror Goldenberg wrote: > > > > > > >> From: Vivek Kashyap [mailto:[email protected]] > >> Sent: Thursday, August 18, 2005 9:36 AM > >> > >> On Wed, 17 Aug 2005, Dror Goldenberg wrote: > >> > >>> > >>> > >>>> From: Vivek Kashyap [mailto:[email protected]] > >>>> Sent: Monday, August 15, 2005 9:41 AM > >>>> > >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote: > > > > <snip> > > > >> > >>>>> (2) Active/Active Model Support > >>>>> > > > > > > <snip> > > > >>>> 3. In case of the active-active connection the request from the > >>>> numerically smaller MAC Address (QPN+GID or probably just > >> the GID) is > >>>> terminated. The error code 'state connection' is a > >> possibility here. > >>>> > >>>> The active-active situation is known when it is determined > >> that there > >>>> is an outstanding request to the remote peer (as determined by > >>>> GID+QPN) while a request from the same is received on local > >>>> GID+ServiceID. > >>> > >>> In IB, active-active is only determined by equal Service > IDs. See IB > >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to > >>> Peer-Compare state based solely on the Service ID. > Therefore, if you > >>> want to have active-active, then you must > >> ensure that > >>> both service IDs are identical. In the current draft, the > >> Service IDs are > >>> different, as they include a remote QP number. > >>> > >> > >> Yes, if we are determining it based on the CM. However, I was > >> suggesting the above as what the ipoib component will do > when it is > >> informed by CM. It can return a suitable error to CM when it > >> determines that a particular > >> connection shouldn't be allowed. Is that not possible? > >> > > > > That wouldn't be "active/active" per se. But maybe it's > just as good, > > because it just pushes this model to the ipoib layer. In a case of > > crossing connection > > requests one side is supposed to accept, while the other is > supposed to > > reject. The rules for rejection have to be described in the > RFC. Maybe it > > worth describing the state machine that now needs to be implemented > > in ipoib. > > yes, need to look into it in detail. My initial thought is that the > method of determining if there are connections in flight is > based on receiving a request source GID+QPN while attempting > a connection to the same. > > > > > BTW, what will be the rejection reason ? Seems like the most > > appropriate is 28 - consumer reject. > > seems ok to me. > > > > > Note that it becomes tricky if you want to allow more than one > > connection on the same service ID. So, is there a way you can tell > > whether your peer is trying to establish the 1st or 2nd connection > > while you are trying to set up the 1st one ? > > hmm...good point, unless we take it that any request that > crosses is deemed active-active. Once a connection is > established further requests can be made on the same > Service-ID (or denied depending on the implementation). > > > > I think it'll be best if we can use the standard active-active > > mechanism. > > We need a common Service-ID for it that must be > known/derivable by both the peers. Since masks are not > supported by CMs using both QPNs in the > service-ID is difficult. > > > > However, if we don't manage to find how, then this approach > (rejection > > at the app level) will do the work. However, it will complicate the > > ipoib implementation and will duplicate the active-active > work that is > > already implemented in the cm. > > > > -Dror > > > > <snip> > > > > > > _______________________________________________ > IPoverIB mailing list > [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib > ------_=_NextPart_001_01C5B690.F8CC94B0 Content-Type: text/html <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII"> <META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45"> <TITLE>RE: [Ipoverib] Comments on IPoIB Connected Mode Connection Establ ishment</TITLE> </HEAD> <BODY> <P><FONT SIZE=2>Sounds like we agree that active/active is very difficult and will be really hard</FONT> <BR><FONT SIZE=2>to define given how IB service ID and communication manager work. Therefore, </FONT> <BR><FONT SIZE=2>I think that we should go for the active/passive with the ability to reject at the </FONT> <BR><FONT SIZE=2>application (ipoib) level.</FONT> </P> <P><FONT SIZE=2>For the case of more than one connection between two peers, we first need to</FONT> <BR><FONT SIZE=2>decide whether we want to allow it (now or in the future). If we disallow it, then</FONT> <BR><FONT SIZE=2>it's very simple, for each crossing request we need to decide which side should</FONT> <BR><FONT SIZE=2>accept and which side should reject it (e.g. GID compare). </FONT> </P> <P><FONT SIZE=2>If we want to allow more than one connection in the future, then maybe we </FONT> <BR><FONT SIZE=2>should think of embedding a "channel index" into the private data. This way, </FONT> <BR><FONT SIZE=2>the two peers can add the channel index on which they are transmitting.</FONT> <BR><FONT SIZE=2>BTW, it'll take more than that to enable more than one channel, for example,</FONT> <BR><FONT SIZE=2>we should try to avoid reordering of IP datagrams that can happen when more </FONT> <BR><FONT SIZE=2>than one channel is being used to send them.</FONT> </P> <P><FONT SIZE=2>What about connection termination ? This should probably be added to the RFC</FONT> <BR><FONT SIZE=2>draft. I can think of two cases where one would want to terminate a connection.</FONT> <BR><FONT SIZE=2>1) When the ipoib driver goes down; 2) Inactive connections can be taken down</FONT> <BR><FONT SIZE=2>to reduce resource consumption.</FONT> </P> <P><FONT SIZE=2>-Dror</FONT> </P> <P><FONT SIZE=2>> -----Original Message-----</FONT> <BR><FONT SIZE=2>> From: Vivek Kashyap [<A HREF="mailto:[email protected]">mailto:[email protected]</A>] </FONT> <BR><FONT SIZE=2>> Sent: Tuesday, September 06, 2005 10:40 AM</FONT> <BR><FONT SIZE=2>> To: Dror Goldenberg</FONT> <BR><FONT SIZE=2>> Cc: IPoverIB</FONT> <BR><FONT SIZE=2>> Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode </FONT> <BR><FONT SIZE=2>> Connection Establ ishment</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> On Sun, 21 Aug 2005, Dror Goldenberg wrote:</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> >> From: Vivek Kashyap [<A HREF="mailto:[email protected]">mailto:[email protected]</A>]</FONT> <BR><FONT SIZE=2>> >> Sent: Thursday, August 18, 2005 9:36 AM</FONT> <BR><FONT SIZE=2>> >></FONT> <BR><FONT SIZE=2>> >> On Wed, 17 Aug 2005, Dror Goldenberg wrote:</FONT> <BR><FONT SIZE=2>> >></FONT> <BR><FONT SIZE=2>> >>></FONT> <BR><FONT SIZE=2>> >>></FONT> <BR><FONT SIZE=2>> >>>> From: Vivek Kashyap [<A HREF="mailto:[email protected]">mailto:[email protected]</A>]</FONT> <BR><FONT SIZE=2>> >>>> Sent: Monday, August 15, 2005 9:41 AM</FONT> <BR><FONT SIZE=2>> >>>></FONT> <BR><FONT SIZE=2>> >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote:</FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > <snip></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> >></FONT> <BR><FONT SIZE=2>> >>>>> (2) Active/Active Model Support</FONT> <BR><FONT SIZE=2>> >>>>></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > <snip></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> >>>> 3. In case of the active-active connection the request from the </FONT> <BR><FONT SIZE=2>> >>>> numerically smaller MAC Address (QPN+GID or probably just</FONT> <BR><FONT SIZE=2>> >> the GID) is</FONT> <BR><FONT SIZE=2>> >>>> terminated. The error code 'state connection' is a</FONT> <BR><FONT SIZE=2>> >> possibility here.</FONT> <BR><FONT SIZE=2>> >>>></FONT> <BR><FONT SIZE=2>> >>>> The active-active situation is known when it is determined</FONT> <BR><FONT SIZE=2>> >> that there</FONT> <BR><FONT SIZE=2>> >>>> is an outstanding request to the remote peer (as determined by</FONT> <BR><FONT SIZE=2>> >>>> GID+QPN) while a request from the same is received on local </FONT> <BR><FONT SIZE=2>> >>>> GID+ServiceID.</FONT> <BR><FONT SIZE=2>> >>></FONT> <BR><FONT SIZE=2>> >>> In IB, active-active is only determined by equal Service </FONT> <BR><FONT SIZE=2>> IDs. See IB </FONT> <BR><FONT SIZE=2>> >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to </FONT> <BR><FONT SIZE=2>> >>> Peer-Compare state based solely on the Service ID. </FONT> <BR><FONT SIZE=2>> Therefore, if you </FONT> <BR><FONT SIZE=2>> >>> want to have active-active, then you must</FONT> <BR><FONT SIZE=2>> >> ensure that</FONT> <BR><FONT SIZE=2>> >>> both service IDs are identical. In the current draft, the</FONT> <BR><FONT SIZE=2>> >> Service IDs are</FONT> <BR><FONT SIZE=2>> >>> different, as they include a remote QP number.</FONT> <BR><FONT SIZE=2>> >>></FONT> <BR><FONT SIZE=2>> >></FONT> <BR><FONT SIZE=2>> >> Yes, if we are determining it based on the CM. However, I was </FONT> <BR><FONT SIZE=2>> >> suggesting the above as what the ipoib component will do </FONT> <BR><FONT SIZE=2>> when it is </FONT> <BR><FONT SIZE=2>> >> informed by CM. It can return a suitable error to CM when it </FONT> <BR><FONT SIZE=2>> >> determines that a particular</FONT> <BR><FONT SIZE=2>> >> connection shouldn't be allowed. Is that not possible?</FONT> <BR><FONT SIZE=2>> >></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > That wouldn't be "active/active" per se. But maybe it's </FONT> <BR><FONT SIZE=2>> just as good, </FONT> <BR><FONT SIZE=2>> > because it just pushes this model to the ipoib layer. In a case of </FONT> <BR><FONT SIZE=2>> > crossing connection</FONT> <BR><FONT SIZE=2>> > requests one side is supposed to accept, while the other is </FONT> <BR><FONT SIZE=2>> supposed to</FONT> <BR><FONT SIZE=2>> > reject. The rules for rejection have to be described in the </FONT> <BR><FONT SIZE=2>> RFC. Maybe it</FONT> <BR><FONT SIZE=2>> > worth describing the state machine that now needs to be implemented</FONT> <BR><FONT SIZE=2>> > in ipoib.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> yes, need to look into it in detail. My initial thought is that the </FONT> <BR><FONT SIZE=2>> method of determining if there are connections in flight is </FONT> <BR><FONT SIZE=2>> based on receiving a request source GID+QPN while attempting </FONT> <BR><FONT SIZE=2>> a connection to the same.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > BTW, what will be the rejection reason ? Seems like the most </FONT> <BR><FONT SIZE=2>> > appropriate is 28 - consumer reject.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> seems ok to me.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > Note that it becomes tricky if you want to allow more than one </FONT> <BR><FONT SIZE=2>> > connection on the same service ID. So, is there a way you can tell </FONT> <BR><FONT SIZE=2>> > whether your peer is trying to establish the 1st or 2nd connection </FONT> <BR><FONT SIZE=2>> > while you are trying to set up the 1st one ?</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> hmm...good point, unless we take it that any request that </FONT> <BR><FONT SIZE=2>> crosses is deemed active-active. Once a connection is </FONT> <BR><FONT SIZE=2>> established further requests can be made on the same </FONT> <BR><FONT SIZE=2>> Service-ID (or denied depending on the implementation).</FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > I think it'll be best if we can use the standard active-active </FONT> <BR><FONT SIZE=2>> > mechanism.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> We need a common Service-ID for it that must be </FONT> <BR><FONT SIZE=2>> known/derivable by both the peers. Since masks are not </FONT> <BR><FONT SIZE=2>> supported by CMs using both QPNs in the </FONT> <BR><FONT SIZE=2>> service-ID is difficult.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> > However, if we don't manage to find how, then this approach </FONT> <BR><FONT SIZE=2>> (rejection </FONT> <BR><FONT SIZE=2>> > at the app level) will do the work. However, it will complicate the </FONT> <BR><FONT SIZE=2>> > ipoib implementation and will duplicate the active-active </FONT> <BR><FONT SIZE=2>> work that is </FONT> <BR><FONT SIZE=2>> > already implemented in the cm.</FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > -Dror</FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > <snip></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> _______________________________________________</FONT> <BR><FONT SIZE=2>> IPoverIB mailing list</FONT> <BR><FONT SIZE=2>> [email protected] <A HREF="https://www1.ietf.org/mailman/listinfo/ipoverib" TARGET="_blank">https://www1.ietf.org/mailman/listinfo/ipoverib</A></FONT> <BR><FONT SIZE=2>> </FONT> </P> </BODY> </HTML> ------_=_NextPart_001_01C5B690.F8CC94B0-- --===============0438489395== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============0438489395==--