RE: ipoib-cm multiple connection model
Dror Goldenberg <[email protected]> Wed, 26 Oct 2005 09:18:00 +0200
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0540686602== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D9FD.CD85A6EE" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D9FD.CD85A6EE Content-Type: text/plain Vivek, I agree that the two scenarios you mention here are the interesting ones. I don't agree about the solution. Let's look at the two cases: 1) Left peer accepts I disagree. You have an assumption that left peer can revoke its original connection request - BEFORE transmitting the REP that accepts the peer request. I am not sure that you can do it with every CM implementation. Even if you can, then the HCA can reorder those packets if they are on different QPs, and the fabric can reorder them as well if they are on different VLs. 2) Left peer rejects I think that this scenario works. Thanks Dror > -----Original Message----- > From: Vivek Kashyap [mailto:[email protected]] > Sent: Tuesday, October 25, 2005 11:38 PM > To: Vandana Rao > Cc: Dror Goldenberg; '[email protected]' > Subject: RE: [Ipoverib] ipoib-cm multiple connection model > > > Hi Vandana, > > I too agree there is no problem here. > > I think you are right if the left peer accepts the > connection. However, if the reverse was the case, that is the > left peer rejected the connection since it is found that the > local address is numerically larger. It will do this > comparison since its REQ is also outstanding. The right peer > will not receive any REQ (it is delayed or yet to be > retransmitted). If it receives the reject it will cancel its > request. At a later time it will receive the > delayed/retransmitted request and one connection will be > established. > > Dror, do these two scenarios look right to you? > > Vivek > > On Thu, 20 Oct 2005, Vandana Rao wrote: > > > Hi Dror, > > I think you are confusing the 2 IDs that come into play in > this setup. > > The > > communication ID is something that the CM entity on a node > understands but > > the transaction ID is something that the MAD handling > entity uses (for > > retransmission, req/resp matchup etc.). > > So, in the second slide, there should not be a > retransmission of the left > > peer REQ MAD since when the left peer accepts the > connection, it will cancel > > its outstanding REQ request and hence the retransmission > should not take > > place. > > -Vandana > > > > At 12:39 AM 10/20/2005, Dror Goldenberg wrote: > >> After giving it some more thought, I think that there is still a > >> problem. > >> Please look at the attached pdf. > >> > >> In the first slide, everything happens smoothly, the left > peer knows > >> that > >> it has to accept the connection and the right peer knows > that it has to > >> decline. > >> On the 2nd slide, the left peer REQ transmission is > dropped in the fabric. > >> It is retransmitted independently by the CM way after a > channel has already > >> been established. Now the right side can not really tell > whether this is a > >> new channel request or an old "simultaneous connection > channel". This may > >> lead to two channels being opened mistakenly. > >> > >> I don't believe it can be solved by looking at the > communication ID. > >> This > >> number is just being fabricated by the sender of the REQ > and is just a > >> "random number" with respect to what we're trying to determine. > >> > >> -Dror > >> > >> -----Original Message----- > >> From: Dror Goldenberg > >> Sent: Wednesday, October 19, 2005 9:37 PM > >> To: 'Vandana Rao'; Dror Goldenberg; [email protected] > >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >> > >> Hi Vandana, > >> > >> Thanks for clarifying this. I agree that if IPoiB-CM > implementation > >> looks > >> at the Communication ID it can make those decisions > correctly. The question > >> is whether this value is exposed through common CM APIs. > For all other > >> purposes, the CM doesn't need to expose the Communication > ID to the ULP. > >> However, seems like it's not a big deal to expose this > value to the ULP. > >> Anyway, it worth noting this explanation in the RFC so > that it's clear that > >> a proper implementation of IPoIB-CM should monitor > Communication ID as part > >> of the simultaneous connection decision. > >> > >> -Dror > >> -----Original Message----- > >> From: Vandana Rao [mailto:[email protected]] > >> Sent: Wednesday, October 19, 2005 6:40 PM > >> To: Dror Goldenberg; [email protected] > >> Subject: Re: [Ipoverib] ipoib-cm multiple connection model > >> > >> Hi Dror, > >> The purpose of the MAD transaction ID is to allow one to > do what you > >> say > >> below, i.e. have multiple connection establishment MADs > between 2 peers. It > >> is essentially the same as the "channel ID" that you talk > about below. Its > >> just that it is a generic channel ID for any MAD > transactions, not just CM > >> MADs. > >> I do not believe there is any issue with existing IB CM in > this regard. > >> -Vandana > >> > >> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: > >>> I think that the current model for multiple connections > >>> establishment > >>> between > >>> two peers works well for 1 connection but is broken for > more than one > >>> connection. > >>> Because of cases where CM packets are dropped or travel > on different VLs, > >>> they > >>> might get reordered on the way and observed in a > different order than were > >>> originally > >>> sent. This breaks the current model which assumes that > the REQs are sent > >>> and > >>> processed in order. > >>> > >>> I think that we need to think of a model which is robust to CM > >>> packets > >>> reordering > >>> which can be one of: > >>> - Adding a "channel ID" in the private data. The field > goes 0,1,2 and on. > >>> It helps > >>> synchronizing both peers on which is the channel at > question now. The > >>> rest > >>> of the decision (per channel) of whether to accept or > reject can be the > >>> same > >>> as defined in the draft. > >>> - Not supporting multiple connections between the same peers. > >>> > >>> -Dror > >>> _______________________________________________ > >>> IPoverIB mailing list > >>> [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib > >> > > > ------_=_NextPart_001_01C5D9FD.CD85A6EE Content-Type: text/html <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45"> <TITLE>RE: [Ipoverib] ipoib-cm multiple connection model</TITLE> </HEAD> <BODY> <P><FONT SIZE=2>Vivek,</FONT> </P> <P><FONT SIZE=2>I agree that the two scenarios you mention here are the interesting ones. </FONT> <BR><FONT SIZE=2>I don't agree about the solution. Let's look at the two cases:</FONT> </P> <P><FONT SIZE=2>1) Left peer accepts</FONT> <BR> <FONT SIZE=2>I disagree. You have an assumption that left peer can</FONT> <BR><FONT SIZE=2> revoke its original connection request - BEFORE transmitting</FONT> <BR> <FONT SIZE=2>the REP that accepts the peer request. I am not sure that </FONT> <BR> <FONT SIZE=2>you can do it with every CM implementation. Even if you can,</FONT> <BR> <FONT SIZE=2>then the HCA can reorder those packets if they are on</FONT> <BR> <FONT SIZE=2>different QPs, and the fabric can reorder them as well if</FONT> <BR> <FONT SIZE=2>they are on different VLs.</FONT> </P> <P><FONT SIZE=2>2) Left peer rejects</FONT> <BR> <FONT SIZE=2>I think that this scenario works.</FONT> </P> <P><FONT SIZE=2>Thanks</FONT> <BR><FONT SIZE=2>Dror</FONT> </P> <P><FONT SIZE=2>> -----Original Message-----</FONT> <BR><FONT SIZE=2>> From: Vivek Kashyap [<A HREF="mailto:[email protected]">mailto:[email protected]</A>] </FONT> <BR><FONT SIZE=2>> Sent: Tuesday, October 25, 2005 11:38 PM</FONT> <BR><FONT SIZE=2>> To: Vandana Rao</FONT> <BR><FONT SIZE=2>> Cc: Dror Goldenberg; '[email protected]'</FONT> <BR><FONT SIZE=2>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> Hi Vandana,</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> I too agree there is no problem here.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> I think you are right if the left peer accepts the </FONT> <BR><FONT SIZE=2>> connection. However, if the reverse was the case, that is the </FONT> <BR><FONT SIZE=2>> left peer rejected the connection since it is found that the </FONT> <BR><FONT SIZE=2>> local address is numerically larger. It will do this </FONT> <BR><FONT SIZE=2>> comparison since its REQ is also outstanding. The right peer </FONT> <BR><FONT SIZE=2>> will not receive any REQ (it is delayed or yet to be </FONT> <BR><FONT SIZE=2>> retransmitted). If it receives the reject it will cancel its </FONT> <BR><FONT SIZE=2>> request. At a later time it will receive the </FONT> <BR><FONT SIZE=2>> delayed/retransmitted request and one connection will be </FONT> <BR><FONT SIZE=2>> established.</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> Dror, do these two scenarios look right to you?</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> Vivek</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> On Thu, 20 Oct 2005, Vandana Rao wrote:</FONT> <BR><FONT SIZE=2>> </FONT> <BR><FONT SIZE=2>> > Hi Dror,</FONT> <BR><FONT SIZE=2>> > I think you are confusing the 2 IDs that come into play in </FONT> <BR><FONT SIZE=2>> this setup. </FONT> <BR><FONT SIZE=2>> > The</FONT> <BR><FONT SIZE=2>> > communication ID is something that the CM entity on a node </FONT> <BR><FONT SIZE=2>> understands but </FONT> <BR><FONT SIZE=2>> > the transaction ID is something that the MAD handling </FONT> <BR><FONT SIZE=2>> entity uses (for </FONT> <BR><FONT SIZE=2>> > retransmission, req/resp matchup etc.).</FONT> <BR><FONT SIZE=2>> > So, in the second slide, there should not be a </FONT> <BR><FONT SIZE=2>> retransmission of the left </FONT> <BR><FONT SIZE=2>> > peer REQ MAD since when the left peer accepts the </FONT> <BR><FONT SIZE=2>> connection, it will cancel </FONT> <BR><FONT SIZE=2>> > its outstanding REQ request and hence the retransmission </FONT> <BR><FONT SIZE=2>> should not take </FONT> <BR><FONT SIZE=2>> > place.</FONT> <BR><FONT SIZE=2>> > -Vandana</FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> > At 12:39 AM 10/20/2005, Dror Goldenberg wrote:</FONT> <BR><FONT SIZE=2>> >> After giving it some more thought, I think that there is still a </FONT> <BR><FONT SIZE=2>> >> problem.</FONT> <BR><FONT SIZE=2>> >> Please look at the attached pdf.</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> In the first slide, everything happens smoothly, the left </FONT> <BR><FONT SIZE=2>> peer knows </FONT> <BR><FONT SIZE=2>> >> that</FONT> <BR><FONT SIZE=2>> >> it has to accept the connection and the right peer knows </FONT> <BR><FONT SIZE=2>> that it has to </FONT> <BR><FONT SIZE=2>> >> decline.</FONT> <BR><FONT SIZE=2>> >> On the 2nd slide, the left peer REQ transmission is </FONT> <BR><FONT SIZE=2>> dropped in the fabric. </FONT> <BR><FONT SIZE=2>> >> It is retransmitted independently by the CM way after a </FONT> <BR><FONT SIZE=2>> channel has already </FONT> <BR><FONT SIZE=2>> >> been established. Now the right side can not really tell </FONT> <BR><FONT SIZE=2>> whether this is a </FONT> <BR><FONT SIZE=2>> >> new channel request or an old "simultaneous connection </FONT> <BR><FONT SIZE=2>> channel". This may </FONT> <BR><FONT SIZE=2>> >> lead to two channels being opened mistakenly.</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> I don't believe it can be solved by looking at the </FONT> <BR><FONT SIZE=2>> communication ID. </FONT> <BR><FONT SIZE=2>> >> This</FONT> <BR><FONT SIZE=2>> >> number is just being fabricated by the sender of the REQ </FONT> <BR><FONT SIZE=2>> and is just a </FONT> <BR><FONT SIZE=2>> >> "random number" with respect to what we're trying to determine.</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> -Dror</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> -----Original Message-----</FONT> <BR><FONT SIZE=2>> >> From: Dror Goldenberg</FONT> <BR><FONT SIZE=2>> >> Sent: Wednesday, October 19, 2005 9:37 PM</FONT> <BR><FONT SIZE=2>> >> To: 'Vandana Rao'; Dror Goldenberg; [email protected]</FONT> <BR><FONT SIZE=2>> >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> Hi Vandana,</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> Thanks for clarifying this. I agree that if IPoiB-CM </FONT> <BR><FONT SIZE=2>> implementation </FONT> <BR><FONT SIZE=2>> >> looks</FONT> <BR><FONT SIZE=2>> >> at the Communication ID it can make those decisions </FONT> <BR><FONT SIZE=2>> correctly. The question </FONT> <BR><FONT SIZE=2>> >> is whether this value is exposed through common CM APIs. </FONT> <BR><FONT SIZE=2>> For all other </FONT> <BR><FONT SIZE=2>> >> purposes, the CM doesn't need to expose the Communication </FONT> <BR><FONT SIZE=2>> ID to the ULP. </FONT> <BR><FONT SIZE=2>> >> However, seems like it's not a big deal to expose this </FONT> <BR><FONT SIZE=2>> value to the ULP.</FONT> <BR><FONT SIZE=2>> >> Anyway, it worth noting this explanation in the RFC so </FONT> <BR><FONT SIZE=2>> that it's clear that </FONT> <BR><FONT SIZE=2>> >> a proper implementation of IPoIB-CM should monitor </FONT> <BR><FONT SIZE=2>> Communication ID as part </FONT> <BR><FONT SIZE=2>> >> of the simultaneous connection decision.</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> -Dror</FONT> <BR><FONT SIZE=2>> >> -----Original Message-----</FONT> <BR><FONT SIZE=2>> >> From: Vandana Rao [<A HREF="mailto:[email protected]">mailto:[email protected]</A>]</FONT> <BR><FONT SIZE=2>> >> Sent: Wednesday, October 19, 2005 6:40 PM</FONT> <BR><FONT SIZE=2>> >> To: Dror Goldenberg; [email protected]</FONT> <BR><FONT SIZE=2>> >> Subject: Re: [Ipoverib] ipoib-cm multiple connection model</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> Hi Dror,</FONT> <BR><FONT SIZE=2>> >> The purpose of the MAD transaction ID is to allow one to </FONT> <BR><FONT SIZE=2>> do what you </FONT> <BR><FONT SIZE=2>> >> say</FONT> <BR><FONT SIZE=2>> >> below, i.e. have multiple connection establishment MADs </FONT> <BR><FONT SIZE=2>> between 2 peers. It </FONT> <BR><FONT SIZE=2>> >> is essentially the same as the "channel ID" that you talk </FONT> <BR><FONT SIZE=2>> about below. Its </FONT> <BR><FONT SIZE=2>> >> just that it is a generic channel ID for any MAD </FONT> <BR><FONT SIZE=2>> transactions, not just CM </FONT> <BR><FONT SIZE=2>> >> MADs.</FONT> <BR><FONT SIZE=2>> >> I do not believe there is any issue with existing IB CM in </FONT> <BR><FONT SIZE=2>> this regard.</FONT> <BR><FONT SIZE=2>> >> -Vandana</FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> >> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:</FONT> <BR><FONT SIZE=2>> >>> I think that the current model for multiple connections </FONT> <BR><FONT SIZE=2>> >>> establishment</FONT> <BR><FONT SIZE=2>> >>> between</FONT> <BR><FONT SIZE=2>> >>> two peers works well for 1 connection but is broken for </FONT> <BR><FONT SIZE=2>> more than one </FONT> <BR><FONT SIZE=2>> >>> connection.</FONT> <BR><FONT SIZE=2>> >>> Because of cases where CM packets are dropped or travel </FONT> <BR><FONT SIZE=2>> on different VLs, </FONT> <BR><FONT SIZE=2>> >>> they</FONT> <BR><FONT SIZE=2>> >>> might get reordered on the way and observed in a </FONT> <BR><FONT SIZE=2>> different order than were </FONT> <BR><FONT SIZE=2>> >>> originally</FONT> <BR><FONT SIZE=2>> >>> sent. This breaks the current model which assumes that </FONT> <BR><FONT SIZE=2>> the REQs are sent </FONT> <BR><FONT SIZE=2>> >>> and</FONT> <BR><FONT SIZE=2>> >>> processed in order.</FONT> <BR><FONT SIZE=2>> >>> </FONT> <BR><FONT SIZE=2>> >>> I think that we need to think of a model which is robust to CM </FONT> <BR><FONT SIZE=2>> >>> packets</FONT> <BR><FONT SIZE=2>> >>> reordering</FONT> <BR><FONT SIZE=2>> >>> which can be one of:</FONT> <BR><FONT SIZE=2>> >>> - Adding a "channel ID" in the private data. The field </FONT> <BR><FONT SIZE=2>> goes 0,1,2 and on. </FONT> <BR><FONT SIZE=2>> >>> It helps</FONT> <BR><FONT SIZE=2>> >>> synchronizing both peers on which is the channel at </FONT> <BR><FONT SIZE=2>> question now. The </FONT> <BR><FONT SIZE=2>> >>> rest</FONT> <BR><FONT SIZE=2>> >>> of the decision (per channel) of whether to accept or </FONT> <BR><FONT SIZE=2>> reject can be the </FONT> <BR><FONT SIZE=2>> >>> same</FONT> <BR><FONT SIZE=2>> >>> as defined in the draft.</FONT> <BR><FONT SIZE=2>> >>> - Not supporting multiple connections between the same peers.</FONT> <BR><FONT SIZE=2>> >>> </FONT> <BR><FONT SIZE=2>> >>> -Dror</FONT> <BR><FONT SIZE=2>> >>> _______________________________________________</FONT> <BR><FONT SIZE=2>> >>> IPoverIB mailing list</FONT> <BR><FONT SIZE=2>> >>> [email protected] <A HREF="https://www1.ietf.org/mailman/listinfo/ipoverib" TARGET="_blank">https://www1.ietf.org/mailman/listinfo/ipoverib</A></FONT> <BR><FONT SIZE=2>> >> </FONT> <BR><FONT SIZE=2>> ></FONT> <BR><FONT SIZE=2>> </FONT> </P> </BODY> </HTML> ------_=_NextPart_001_01C5D9FD.CD85A6EE-- --===============0540686602== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============0540686602==--