RE: ipoib-cm multiple connection model
Dror Goldenberg <[email protected]> Thu, 27 Oct 2005 09:08:37 +0200
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0258544351== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5DAC5.40B46E8A" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5DAC5.40B46E8A Content-Type: text/plain -----Original Message----- From: Vandana Rao [mailto:[email protected]] Sent: Wednesday, October 26, 2005 8:35 PM To: Dror Goldenberg; '[email protected]' Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Dror, You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP. But the point is that those MADs in question belong to different transactions, one is REP for transaction 1 and other is REQ for transaction 2. Therefore the CM need not guarantee any ordering between them. If not, there is potential for the state machines to get messed up. -Vandana At 12:20 AM 10/26/2005, Dror Goldenberg wrote: Hi Vandana, The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases: 1) The CM uses more than one QP, in which case the HCA can reorder those packets 2) The CM uses different SL/VL, in which case the fabric can reorder the packets 3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU. My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required. -Dror -----Original Message----- From: Vandana Rao [ mailto:[email protected] <mailto:[email protected]> ] Sent: Thursday, October 20, 2005 7:25 PM To: Dror Goldenberg; Dror Goldenberg; '[email protected]' Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Dror, I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.). So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place. -Vandana At 12:39 AM 10/20/2005, Dror Goldenberg wrote: After giving it some more thought, I think that there is still a problem. Please look at the attached pdf. In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline. On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly. I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine. -Dror -----Original Message----- From: Dror Goldenberg Sent: Wednesday, October 19, 2005 9:37 PM To: 'Vandana Rao'; Dror Goldenberg; [email protected] Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Vandana, Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision. -Dror -----Original Message----- From: Vandana Rao [ mailto:[email protected] <mailto:[email protected]> ] Sent: Wednesday, October 19, 2005 6:40 PM To: Dror Goldenberg; [email protected] Subject: Re: [Ipoverib] ipoib-cm multiple connection model Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib <https://www1.ietf.org/mailman/listinfo/ipoverib> _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib <https://www1.ietf.org/mailman/listinfo/ipoverib> ------_=_NextPart_001_01C5DAC5.40B46E8A Content-Type: text/html <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <TITLE>Message</TITLE> <META content="MSHTML 6.00.2900.2769" name=GENERATOR></HEAD> <BODY> <DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV> <BLOCKQUOTE dir=ltr style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px"> <DIV></DIV> <DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT face=Tahoma size=2>-----Original Message-----<BR><B>From:</B> Vandana Rao [mailto:[email protected]] <BR><B>Sent:</B> Wednesday, October 26, 2005 8:35 PM<BR><B>To:</B> Dror Goldenberg; '[email protected]'<BR><B>Subject:</B> RE: [Ipoverib] ipoib-cm multiple connection model<BR><BR></FONT></DIV> <DIV>Hi Dror,<BR>You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP. <SPAN class=880275706-27102005><FONT face=Arial color=#0000ff size=2> </FONT></SPAN></DIV></BLOCKQUOTE> <DIV dir=ltr><SPAN class=880275706-27102005> </SPAN><BR><SPAN class=880275706-27102005><FONT face=Arial color=#0000ff size=2>But the point is that those MADs in question belong to different transactions, one is REP for transaction 1 and other is REQ for transaction 2. Therefore the CM need not guarantee any ordering between them.</FONT></SPAN></DIV> <BLOCKQUOTE dir=ltr style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px"> <DIV><SPAN class=880275706-27102005> </SPAN></DIV> <DIV><SPAN class=880275706-27102005> </SPAN>If not, there is potential for the state machines to get messed up.<BR>-Vandana<BR><BR>At 12:20 AM 10/26/2005, Dror Goldenberg wrote:<BR></DIV> <BLOCKQUOTE class=cite cite="" type="cite"><FONT color=#0000ff size=2>Hi Vandana,<BR></FONT> <BR><FONT color=#0000ff size=2>The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases:<BR>1) The CM uses more than one QP, in which case the HCA can reorder those packets<BR>2) The CM uses different SL/VL, in which case the fabric can reorder the packets<BR>3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU.<BR></FONT> <BR><FONT color=#0000ff size=2>My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required.<BR></FONT> <BR><FONT color=#0000ff size=2>-Dror<BR></FONT> <DL> <DD><FONT face=Tahoma size=2>-----Original Message-----<BR> <DD>From:</B> Vandana Rao [<A href="mailto:[email protected]" eudora="autourl"> mailto:[email protected]</A>] <BR> <DD>Sent:</B> Thursday, October 20, 2005 7:25 PM<BR> <DD>To:</B> Dror Goldenberg; Dror Goldenberg; '[email protected]'<BR> <DD>Subject:</B> RE: [Ipoverib] ipoib-cm multiple connection model<BR><BR></FONT> <DD>Hi Dror,<BR> <DD>I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.).<BR> <DD> So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place.<BR> <DD>-Vandana<BR><BR> <DD>At 12:39 AM 10/20/2005, Dror Goldenberg wrote:<BR> <BLOCKQUOTE class=cite cite="" type="cite"> <DD><FONT color=#0000ff size=2>After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.<BR></FONT> <DD><BR> <DD><FONT color=#0000ff size=2>In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.<BR> <DD>On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.<BR></FONT> <DD><BR> <DD><FONT color=#0000ff size=2>I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.<BR></FONT> <DD><BR> <DD><FONT color=#0000ff size=2>-Dror<BR></FONT> <DD> <DL> <DD><FONT face=Tahoma size=2>-----Original Message----- <DD>From: Dror Goldenberg <DD>Sent: Wednesday, October 19, 2005 9:37 PM <DD>To: 'Vandana Rao'; Dror Goldenberg; [email protected] <DD>Subject: RE: [Ipoverib] ipoib-cm multiple connection model<BR><BR></FONT> <DD><FONT color=#0000ff size=2>Hi Vandana,</FONT> <DD> <DD><FONT color=#0000ff size=2>Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. <DD>Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.</FONT> <DD> <DD><FONT color=#0000ff size=2>-Dror</FONT> <DL> <DD><FONT face=Tahoma size=2>-----Original Message----- <DD>From: Vandana Rao [ <A href="mailto:[email protected]" eudora="autourl">mailto:[email protected]</A>] <DD>Sent: Wednesday, October 19, 2005 6:40 PM <DD>To: Dror Goldenberg; [email protected] <DD>Subject: Re: [Ipoverib] ipoib-cm multiple connection model<BR><BR></FONT> <DD>Hi Dror, <DD>The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. <DD>I do not believe there is any issue with existing IB CM in this regard. <DD>-Vandana<BR> <DD>At 03:11 AM 10/19/2005, Dror Goldenberg wrote: <BLOCKQUOTE class=cite cite="" type="cite"> <DD><FONT size=2>I think that the current model for multiple connections establishment between <DD>two peers works well for 1 connection but is broken for more than one connection. <DD>Because of cases where CM packets are dropped or travel on different VLs, they <DD>might get reordered on the way and observed in a different order than were originally <DD>sent. This breaks the current model which assumes that the REQs are sent and <DD>processed in order. </FONT> <DD> <DD><FONT size=2>I think that we need to think of a model which is robust to CM packets reordering <DD>which can be one of: <DD>- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps <DD> synchronizing both peers on which is the channel at question now. The rest <DD> of the decision (per channel) of whether to accept or reject can be the same <DD> as defined in the draft. <DD>- Not supporting multiple connections between the same peers.<BR> <DD> <DD>-Dror</FONT> <DD>_______________________________________________ <DD>IPoverIB mailing list <DD>[email protected] <DD><A href="https://www1.ietf.org/mailman/listinfo/ipoverib" eudora="autourl">https://www1.ietf.org/mailman/listinfo/ipoverib</A></DD></BLOCKQUOTE></DD></DL></DD></DL></DD></BLOCKQUOTE></DD></DL> <DL></DL> <DL></DL>_______________________________________________<BR>IPoverIB mailing list<BR>[email protected]<BR><A href="https://www1.ietf.org/mailman/listinfo/ipoverib" eudora="autourl">https://www1.ietf.org/mailman/listinfo/ipoverib</A></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML> ------_=_NextPart_001_01C5DAC5.40B46E8A-- --===============0258544351== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============0258544351==--