RE: Comments on IPoIB Connected Mode Connection Establ ishment
Michael Krause <[email protected]> Mon, 12 Sep 2005 08:34:54 -0700
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
--===============0546663266== Content-Type: multipart/alternative; boundary="=====================_183658236==.ALT" --=====================_183658236==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed At 10:23 PM 9/10/2005, Dror Goldenberg wrote: >Sounds like we agree that active/active is very difficult and will be >really hard >to define given how IB service ID and communication manager work. Therefore, >I think that we should go for the active/passive with the ability to >reject at the >application (ipoib) level. The IBTA can develop what is required to fix the SID issues. >For the case of more than one connection between two peers, we first need to >decide whether we want to allow it (now or in the future). If we disallow >it, then >it's very simple, for each crossing request we need to decide which side >should >accept and which side should reject it (e.g. GID compare). A multi-connection model just like Ethernet aggregation solutions deployed today enables aggregate performance to be achieved without burdening the upper layers. Flows are predictable, i.e. a given heuristic generally insures that IP datagrams flow over consistent paths thus eliminating the re-ordering problem. These heuristics are locally controlled with Ethernet today and do not need to be specified. As a general comment, people should review what is already defined and deployed today with aggregation technology and see how this might be leveraged for IB. While Ethernet currently limits aggregation to a single switch attachment, that restriction will be removed in the future thus matching what IB can do with its multi-path support. Given the desire to minimize how much a customer really sees of IB from a stack integration perspective, it would be good to see both technologies sharing as much of the infrastructure as possible. >If we want to allow more than one connection in the future, then maybe we >should think of embedding a "channel index" into the private data. This way, >the two peers can add the channel index on which they are transmitting. >BTW, it'll take more than that to enable more than one channel, for example, >we should try to avoid reordering of IP datagrams that can happen when more >than one channel is being used to send them. > >What about connection termination ? This should probably be added to the RFC >draft. I can think of two cases where one would want to terminate a >connection. >1) When the ipoib driver goes down; 2) Inactive connections can be taken down >to reduce resource consumption. Termination should only occur when an unplumb operation occurs. This is when the network stack is effectively taken down. The cause of the unplumb can be any number of reasons thus it does not matter what the specific reason is only that an admin has decided to unplumb the stack. Note: An error, loss of cable, etc. are not unplumb operations so should not impact the connection until the IB fabric says communication is no longer possible and errors it out. Mike >-Dror > > > -----Original Message----- > > From: Vivek Kashyap > [<mailto:[email protected]>mailto:[email protected]] > > Sent: Tuesday, September 06, 2005 10:40 AM > > To: Dror Goldenberg > > Cc: IPoverIB > > Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode > > Connection Establ ishment > > > > > > On Sun, 21 Aug 2005, Dror Goldenberg wrote: > > > > > > > > > > >> From: Vivek Kashyap > [<mailto:[email protected]>mailto:[email protected]] > > >> Sent: Thursday, August 18, 2005 9:36 AM > > >> > > >> On Wed, 17 Aug 2005, Dror Goldenberg wrote: > > >> > > >>> > > >>> > > >>>> From: Vivek Kashyap > [<mailto:[email protected]>mailto:[email protected]] > > >>>> Sent: Monday, August 15, 2005 9:41 AM > > >>>> > > >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote: > > > > > > <snip> > > > > > >> > > >>>>> (2) Active/Active Model Support > > >>>>> > > > > > > > > > <snip> > > > > > >>>> 3. In case of the active-active connection the request from the > > >>>> numerically smaller MAC Address (QPN+GID or probably just > > >> the GID) is > > >>>> terminated. The error code 'state connection' is a > > >> possibility here. > > >>>> > > >>>> The active-active situation is known when it is determined > > >> that there > > >>>> is an outstanding request to the remote peer (as determined by > > >>>> GID+QPN) while a request from the same is received on local > > >>>> GID+ServiceID. > > >>> > > >>> In IB, active-active is only determined by equal Service > > IDs. See IB > > >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to > > >>> Peer-Compare state based solely on the Service ID. > > Therefore, if you > > >>> want to have active-active, then you must > > >> ensure that > > >>> both service IDs are identical. In the current draft, the > > >> Service IDs are > > >>> different, as they include a remote QP number. > > >>> > > >> > > >> Yes, if we are determining it based on the CM. However, I was > > >> suggesting the above as what the ipoib component will do > > when it is > > >> informed by CM. It can return a suitable error to CM when it > > >> determines that a particular > > >> connection shouldn't be allowed. Is that not possible? > > >> > > > > > > That wouldn't be "active/active" per se. But maybe it's > > just as good, > > > because it just pushes this model to the ipoib layer. In a case of > > > crossing connection > > > requests one side is supposed to accept, while the other is > > supposed to > > > reject. The rules for rejection have to be described in the > > RFC. Maybe it > > > worth describing the state machine that now needs to be implemented > > > in ipoib. > > > > yes, need to look into it in detail. My initial thought is that the > > method of determining if there are connections in flight is > > based on receiving a request source GID+QPN while attempting > > a connection to the same. > > > > > > > > BTW, what will be the rejection reason ? Seems like the most > > > appropriate is 28 - consumer reject. > > > > seems ok to me. > > > > > > > > Note that it becomes tricky if you want to allow more than one > > > connection on the same service ID. So, is there a way you can tell > > > whether your peer is trying to establish the 1st or 2nd connection > > > while you are trying to set up the 1st one ? > > > > hmm...good point, unless we take it that any request that > > crosses is deemed active-active. Once a connection is > > established further requests can be made on the same > > Service-ID (or denied depending on the implementation). > > > > > > I think it'll be best if we can use the standard active-active > > > mechanism. > > > > We need a common Service-ID for it that must be > > known/derivable by both the peers. Since masks are not > > supported by CMs using both QPNs in the > > service-ID is difficult. > > > > > > > However, if we don't manage to find how, then this approach > > (rejection > > > at the app level) will do the work. However, it will complicate the > > > ipoib implementation and will duplicate the active-active > > work that is > > > already implemented in the cm. > > > > > > -Dror > > > > > > <snip> > > > > > > > > > > _______________________________________________ > > IPoverIB mailing list > > [email protected] > <https://www1.ietf.org/mailman/listinfo/ipoverib>https://www1.ietf.org/mailman/listinfo/ipoverib > > > >_______________________________________________ >IPoverIB mailing list >[email protected] >https://www1.ietf.org/mailman/listinfo/ipoverib --=====================_183658236==.ALT Content-Type: text/html; charset="us-ascii" <html> <body> <font size=3>At 10:23 PM 9/10/2005, Dror Goldenberg wrote:<br><br> </font><blockquote type=cite class=cite cite=""><font size=2>Sounds like we agree that active/active is very difficult and will be really hard</font><font size=3> <br> </font><font size=2>to define given how IB service ID and communication manager work. Therefore, </font><font size=3><br> </font><font size=2>I think that we should go for the active/passive with the ability to reject at the </font><font size=3><br> </font><font size=2>application (ipoib) level.</font><font size=3> </font></blockquote><br> The IBTA can develop what is required to fix the SID issues.<br><br> <br> <blockquote type=cite class=cite cite=""><font size=2>For the case of more than one connection between two peers, we first need to</font><font size=3> <br> </font><font size=2>decide whether we want to allow it (now or in the future). If we disallow it, then</font><font size=3> <br> </font><font size=2>it's very simple, for each crossing request we need to decide which side should</font><font size=3> <br> </font><font size=2>accept and which side should reject it (e.g. GID compare). </font></blockquote><br> A multi-connection model just like Ethernet aggregation solutions deployed today enables aggregate performance to be achieved without burdening the upper layers. Flows are predictable, i.e. a given heuristic generally insures that IP datagrams flow over consistent paths thus eliminating the re-ordering problem. These heuristics are locally controlled with Ethernet today and do not need to be specified. As a general comment, people should review what is already defined and deployed today with aggregation technology and see how this might be leveraged for IB. While Ethernet currently limits aggregation to a single switch attachment, that restriction will be removed in the future thus matching what IB can do with its multi-path support. Given the desire to minimize how much a customer really sees of IB from a stack integration perspective, it would be good to see both technologies sharing as much of the infrastructure as possible.<br><br> <br> <blockquote type=cite class=cite cite=""><font size=2>If we want to allow more than one connection in the future, then maybe we </font><font size=3><br> </font><font size=2>should think of embedding a "channel index" into the private data. This way, </font><font size=3><br> </font><font size=2>the two peers can add the channel index on which they are transmitting.</font><font size=3> <br> </font><font size=2>BTW, it'll take more than that to enable more than one channel, for example,</font><font size=3> <br> </font><font size=2>we should try to avoid reordering of IP datagrams that can happen when more </font><font size=3><br> </font><font size=2>than one channel is being used to send them.</font><font size=3> <br><br> </font><font size=2>What about connection termination ? This should probably be added to the RFC</font><font size=3> <br> </font><font size=2>draft. I can think of two cases where one would want to terminate a connection.</font><font size=3> <br> </font><font size=2>1) When the ipoib driver goes down; 2) Inactive connections can be taken down</font><font size=3> <br> </font><font size=2>to reduce resource consumption.</font><font size=3> </font></blockquote><br> Termination should only occur when an unplumb operation occurs. This is when the network stack is effectively taken down. The cause of the unplumb can be any number of reasons thus it does not matter what the specific reason is only that an admin has decided to unplumb the stack. Note: An error, loss of cable, etc. are not unplumb operations so should not impact the connection until the IB fabric says communication is no longer possible and errors it out.<br><br> Mike<br><br> <br> <blockquote type=cite class=cite cite=""><font size=2>-Dror</font> <font size=3> <br><br> </font><font size=2>> -----Original Message-----</font><font size=3> <br> </font><font size=2>> From: Vivek Kashyap [<a href="mailto:[email protected]">mailto:[email protected]</a>] </font><font size=3><br> </font><font size=2>> Sent: Tuesday, September 06, 2005 10:40 AM</font><font size=3> <br> </font><font size=2>> To: Dror Goldenberg</font><font size=3> <br> </font><font size=2>> Cc: IPoverIB</font><font size=3> <br> </font><font size=2>> Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode </font><font size=3><br> </font><font size=2>> Connection Establ ishment</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> On Sun, 21 Aug 2005, Dror Goldenberg wrote:</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> >> From: Vivek Kashyap [<a href="mailto:[email protected]">mailto:[email protected]</a> ]</font><font size=3> <br> </font><font size=2>> >> Sent: Thursday, August 18, 2005 9:36 AM</font><font size=3> <br> </font><font size=2>> >></font><font size=3> <br> </font><font size=2>> >> On Wed, 17 Aug 2005, Dror Goldenberg wrote:</font><font size=3> <br> </font><font size=2>> >></font><font size=3> <br> </font><font size=2>> >>></font><font size=3> <br> </font><font size=2>> >>></font><font size=3> <br> </font><font size=2>> >>>> From: Vivek Kashyap [<a href="mailto:[email protected]">mailto:[email protected]</a> ]</font><font size=3> <br> </font><font size=2>> >>>> Sent: Monday, August 15, 2005 9:41 AM</font><font size=3> <br> </font><font size=2>> >>>></font><font size=3> <br> </font><font size=2>> >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote:</font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > <snip></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> >></font><font size=3> <br> </font><font size=2>> >>>>> (2) Active/Active Model Support</font><font size=3> <br> </font><font size=2>> >>>>></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > <snip></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> >>>> 3. In case of the active-active connection the request from the </font><font size=3><br> </font><font size=2>> >>>> numerically smaller MAC Address (QPN+GID or probably just</font><font size=3> <br> </font><font size=2>> >> the GID) is</font><font size=3> <br> </font><font size=2>> >>>> terminated. The error code 'state connection' is a</font><font size=3> <br> </font><font size=2>> >> possibility here.</font><font size=3> <br> </font><font size=2>> >>>></font><font size=3> <br> </font><font size=2>> >>>> The active-active situation is known when it is determined</font><font size=3> <br> </font><font size=2>> >> that there</font><font size=3> <br> </font><font size=2>> >>>> is an outstanding request to the remote peer (as determined by</font><font size=3> <br> </font><font size=2>> >>>> GID+QPN) while a request from the same is received on local </font><font size=3><br> </font><font size=2>> >>>> GID+ServiceID.</font><font size=3> <br> </font><font size=2>> >>></font><font size=3> <br> </font><font size=2>> >>> In IB, active-active is only determined by equal Service </font><font size=3><br> </font><font size=2>> IDs. See IB </font><font size=3><br> </font><font size=2>> >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to </font><font size=3><br> </font><font size=2>> >>> Peer-Compare state based solely on the Service ID. </font><font size=3><br> </font><font size=2>> Therefore, if you </font><font size=3><br> </font><font size=2>> >>> want to have active-active, then you must</font><font size=3> <br> </font><font size=2>> >> ensure that</font><font size=3> <br> </font><font size=2>> >>> both service IDs are identical. In the current draft, the</font><font size=3> <br> </font><font size=2>> >> Service IDs are</font><font size=3> <br> </font><font size=2>> >>> different, as they include a remote QP number.</font><font size=3> <br> </font><font size=2>> >>></font><font size=3> <br> </font><font size=2>> >></font><font size=3> <br> </font><font size=2>> >> Yes, if we are determining it based on the CM. However, I was </font><font size=3><br> </font><font size=2>> >> suggesting the above as what the ipoib component will do </font><font size=3><br> </font><font size=2>> when it is </font><font size=3><br> </font><font size=2>> >> informed by CM. It can return a suitable error to CM when it </font><font size=3><br> </font><font size=2>> >> determines that a particular</font><font size=3> <br> </font><font size=2>> >> connection shouldn't be allowed. Is that not possible?</font><font size=3> <br> </font><font size=2>> >></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > That wouldn't be "active/active" per se. But maybe it's </font><font size=3><br> </font><font size=2>> just as good, </font><font size=3><br> </font><font size=2>> > because it just pushes this model to the ipoib layer. In a case of </font><font size=3><br> </font><font size=2>> > crossing connection</font><font size=3> <br> </font><font size=2>> > requests one side is supposed to accept, while the other is </font><font size=3><br> </font><font size=2>> supposed to</font><font size=3> <br> </font><font size=2>> > reject. The rules for rejection have to be described in the </font><font size=3><br> </font><font size=2>> RFC. Maybe it</font><font size=3> <br> </font><font size=2>> > worth describing the state machine that now needs to be implemented</font><font size=3> <br> </font><font size=2>> > in ipoib.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> yes, need to look into it in detail. My initial thought is that the </font><font size=3><br> </font><font size=2>> method of determining if there are connections in flight is </font><font size=3><br> </font><font size=2>> based on receiving a request source GID+QPN while attempting </font><font size=3><br> </font><font size=2>> a connection to the same.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > BTW, what will be the rejection reason ? Seems like the most </font><font size=3><br> </font><font size=2>> > appropriate is 28 - consumer reject.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> seems ok to me.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > Note that it becomes tricky if you want to allow more than one </font><font size=3><br> </font><font size=2>> > connection on the same service ID. So, is there a way you can tell </font><font size=3><br> </font><font size=2>> > whether your peer is trying to establish the 1st or 2nd connection </font><font size=3><br> </font><font size=2>> > while you are trying to set up the 1st one ?</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> hmm...good point, unless we take it that any request that </font><font size=3><br> </font><font size=2>> crosses is deemed active-active. Once a connection is </font><font size=3><br> </font><font size=2>> established further requests can be made on the same </font><font size=3><br> </font><font size=2>> Service-ID (or denied depending on the implementation).</font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > I think it'll be best if we can use the standard active-active </font><font size=3><br> </font><font size=2>> > mechanism.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> We need a common Service-ID for it that must be </font><font size=3><br> </font><font size=2>> known/derivable by both the peers. Since masks are not </font><font size=3><br> </font><font size=2>> supported by CMs using both QPNs in the </font><font size=3><br> </font><font size=2>> service-ID is difficult.</font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> > However, if we don't manage to find how, then this approach </font><font size=3><br> </font><font size=2>> (rejection </font><font size=3><br> </font><font size=2>> > at the app level) will do the work. However, it will complicate the </font><font size=3><br> </font><font size=2>> > ipoib implementation and will duplicate the active-active </font><font size=3><br> </font><font size=2>> work that is </font><font size=3><br> </font><font size=2>> > already implemented in the cm.</font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > -Dror</font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> > <snip></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> ></font><font size=3> <br> </font><font size=2>> </font><font size=3><br> </font><font size=2>> _______________________________________________</font><font size=3> <br> </font><font size=2>> IPoverIB mailing list</font><font size=3> <br> </font><font size=2>> [email protected] <a href="https://www1.ietf.org/mailman/listinfo/ipoverib"> https://www1.ietf.org/mailman/listinfo/ipoverib</a></font><font size=3> <br> </font><font size=2>> <br> </font><font size=3>_______________________________________________<br> IPoverIB mailing list<br> [email protected]<br> <a href="https://www1.ietf.org/mailman/listinfo/ipoverib" eudora="autourl"> https://www1.ietf.org/mailman/listinfo/ipoverib</a></font></blockquote> </body> </html> --=====================_183658236==.ALT-- --===============0546663266== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============0546663266==--