RE: small change in the connected mode draft
"Dror Goldenberg" <[email protected]> Fri, 24 Mar 2006 00:51:52 +0200
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============1810027139== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C64ECC.6093202C" This is a multi-part message in MIME format. ------_=_NextPart_001_01C64ECC.6093202C Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable =20 =09 From: Michael Krause [mailto:[email protected]]=20 Sent: Thursday, March 23, 2006 10:58 PM =09 =09 At 04:04 PM 3/21/2006, H.K. Jerry Chu wrote: =09 Hi folks, =09 Allison Mankin, the transport area AD who brought up some concern during IESG review of the connected draft regarding simultaneous retransmissions at different layers, has suggested and Vivek agreed to the following change to section 7.1 "A Cautionary Note on IPoIB-RC". =09 The revised section reads like this: =09 The RC mode of InfiniBand guarantees in-order delivery of packets. Every message transmitted over the RC connection is broken into physical MTU sized packets by the RC connection. If any packet is lost, it is retransmitted until the complete message is exchanged. Therefore, there is a possibility of an upper transport layer experiencing a timeout, while the RC layer is still in the process of transferring the complete message. TCP will view the timeout as an indicator of congestion and enter slow-start thereby affecting throughput drastically [RFC2581]. Other upper layer protocols might insert retransmissions into the fabric adding to the already existing congestion. =09 The applicability of Infiniband reliability is on a fabric with short latencies (not wide area). Therefore, the RC timer values should be short compared with the starting minimum time values used by the upper end-to-end transports. In addition, because the RC mode does not have measurement based reliable transmission, its use over fabrics with long latency or very dynamic latency may be a concern for congestion- aware traffic traversing those fabrics. =09 If you have any comment/issue on the proposed change please post them to the list before COB this Friday (3/24/06). I'd like to move the draft past IESG review so we can wrap up the WG asap. =09 BTW, please be informed that my email address will change after this Friday. My new address is [email protected]. =09 What about RNR which can go into an infinite timer state? It should also be noted that IB does not mandate timer ranges that necessarily correspond to TCP timers. In fact, in the face of real IB congestion or a port / VL arbitration policy that places such traffic in a best effort QoS slot, then it is quite possible to see delays that would be interpreted by any ULP such as TCP as congestion. Is this really an issue as the applications continue to operate albeit at a slower rate? =09 Mike =20 =20 Mike,=20 =20 I am not sure what is the problem in question now. =20 If people are afraid of retransmission of TCP while a retransmission of IB RC is taking place, then: 1) packet drop is relatively rare in IB and 2) usually retransmission timeouts are fast in IB due to being low latency fabric. Given the above, I'd say that this event is rare anyway and if and when IB retransmission happens, there is a good chance to recover it at the IB level a while before TCP notices it. =20 The case that you mentioned, RNR driven retransmission, is another case of retransmission. It has nothing to do with congestion or packet drop in the fabric. It is just being impacted by the ability of the receiver to post receive buffers on the RQ/SRQ. Here the timeouts are application based and as you wrote can be configured to infinity. I agree that infinit number of retries would be a bad choice of RNR retry count. So, maybe we should recommend on selecting also the RNR timeout and retry count to be low too. I am also wondering what happens when there is a slow receiver, e.g. posting too slow on the RQ/SRQ, in which case the RNR Nak will happen very frequently and might cause the QP to get into error state because of RNR timeout exahusted. One approach would be to downgrade such peers to IPoIB-UD. Other option would be to indirectly detect those cases and provide hints to the TCP stack to slow down. By indirectly detecting it, I am thinking along the lines of when the send queue becomes full, we can start dropping packets at the transmitter (RED algorithm or something similar). This packet drop can cause TCP to back off and activate the congestion flow and reduce the transmission rate. This has the right impact in my opinion. =20 And, the last thing that you mentioned has to do with TCP retransmission triggering when the IB is congested. IB packet delivery slows down because of congestion in the IB fabric (slowing down can be because of real congestion which backpressures the transmitters or because of IB congestion management). Eventhough there is no packet drop at the IB level and no retransmission at the IB level, still TCP times out. Luckily, TCP is doing RTT evaluation, so hopefully this will be rare too. If it happens, then the desired effect would be to slow down the TCP requester (at the TCP level) which is OK. The side effect will be retransmission. And the big question is whether this is so bad to have this retransmission. BTW, I believe that also in this case, dropping packets at the transmitter will ease the problem. =20 Dror =20 =20 =20 ------_=_NextPart_001_01C64ECC.6093202C Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD><TITLE>Message</TITLE> <META http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <META content=3D"MSHTML 6.00.3790.2491" name=3DGENERATOR></HEAD> <BODY> <DIV><FONT face=3DArial color=3D#0000ff size=3D2></FONT> </DIV> <BLOCKQUOTE style=3D"MARGIN-RIGHT: 0px"> <DIV></DIV> <DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr = align=3Dleft><FONT=20 face=3DTahoma size=3D2><BR><B>From:</B> Michael Krause = [mailto:[email protected]]=20 <BR><B>Sent:</B> Thursday, March 23, 2006 10:58 = PM<BR><BR></FONT></DIV><FONT=20 size=3D3>At 04:04 PM 3/21/2006, H.K. Jerry Chu wrote:<BR> <BLOCKQUOTE class=3Dcite cite=3D"" type=3D"cite">Hi = folks,<BR><BR>Allison Mankin,=20 the transport area AD who brought up some<BR>concern during IESG = review of=20 the connected draft regarding<BR>simultaneous retransmissions at = different=20 layers, has suggested<BR>and Vivek agreed to the following change to = section=20 7.1<BR>"A Cautionary Note on IPoIB-RC".<BR><BR>The revised section = reads=20 like=20 = this:<BR><BR><X-TAB> </X-T= AB>The=20 RC mode of InfiniBand guarantees in-order delivery=20 = of<BR><X-TAB> </X-TAB>pack= ets.=20 Every message transmitted over the RC connection=20 = is<BR><X-TAB> </X-TAB>brok= en=20 into physical MTU sized packets by the RC connection.=20 = If<BR><X-TAB> </X-TAB>any = packet is lost, it is retransmitted until the=20 = complete<BR><X-TAB> </X-TA= B>message=20 is exchanged. Therefore, there is a possibility of=20 = an<BR><X-TAB> </X-TAB>uppe= r=20 transport layer experiencing a timeout, while the RC=20 = layer<BR><X-TAB> </X-TAB>i= s=20 still in the process of transferring the complete=20 = message.<BR><X-TAB> </X-TA= B>TCP=20 will view the timeout as an indicator of congestion=20 = and<BR><X-TAB> </X-TAB>ent= er=20 slow-start thereby affecting throughput=20 = drastically<BR><X-TAB> </X= -TAB>[RFC2581].=20 Other upper layer protocols might=20 = insert<BR><X-TAB> </X-TAB>= =20 retransmissions into the fabric adding to the already=20 = existing<BR><X-TAB> </X-TA= B>=20 = congestion.<BR><BR><X-TAB>  = ;</X-TAB>The=20 applicability of Infiniband reliability is on a=20 = fabric<BR><X-TAB> </X-TAB>= with=20 short latencies (not wide area). Therefore, the RC=20 = timer<BR><X-TAB> </X-TAB>v= alues=20 should be short compared with the starting=20 = minimum<BR><X-TAB> </X-TAB= >time=20 values used by the upper end-to-end transports. =20 = In<BR><X-TAB> </X-TAB>addi= tion,=20 because the RC mode does not have=20 = measurement<BR><X-TAB> </X= -TAB>based=20 reliable transmission, its use over fabrics with=20 = long<BR><X-TAB> </X-TAB>la= tency=20 or very dynamic latency may be a concern for=20 = congestion-<BR><X-TAB> </X= -TAB>aware=20 traffic traversing those fabrics.<BR><BR>If you have any = comment/issue on=20 the proposed change please post them<BR>to the list before COB this = Friday=20 (3/24/06). I'd like to move the<BR>draft past IESG review so we can = wrap up=20 the WG asap.<BR><BR>BTW, please be informed that my email address = will=20 change after this<BR>Friday. My new address is=20 [email protected].</BLOCKQUOTE> <DIV><FONT face=3DArial color=3D#0000ff size=3D2></FONT><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT><FONT face=3DArial color=3D#0000ff=20 size=3D2></FONT><BR>What about RNR which can go into an infinite timer = state? It should also be noted that IB does not mandate timer = ranges=20 that necessarily correspond to TCP timers. In fact, in the face = of real=20 IB congestion or a port / VL arbitration policy that places such = traffic in a=20 best effort QoS slot, then it is quite possible to see delays that = would be=20 interpreted by any ULP such as TCP as congestion. Is this really = an=20 issue as the applications continue to operate albeit at a slower=20 rate?<BR><BR>Mike</FONT> <SPAN class=3D416134121-23032006><FONT = face=3DArial=20 color=3D#0000ff size=3D2> </FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial = color=3D#0000ff=20 size=3D2></FONT></SPAN> </DIV></BLOCKQUOTE> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>Mike,=20 </FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>I am=20 not sure what is the problem in question now.</FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>If=20 people are afraid of retransmission of TCP while a retransmission of IB = RC is=20 taking place, then: 1) packet drop is relatively rare in IB and 2) = usually=20 retransmission timeouts are fast in IB due to being low latency fabric. = Given=20 the above, I'd say that this event is rare anyway and if and when = IB=20 retransmission happens, there is a good chance to recover it at the IB = level a=20 while before TCP notices it.</FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>The=20 case that you mentioned, RNR driven retransmission, is another case of=20 retransmission. It has nothing to do with congestion or packet drop in = the=20 fabric. It is just being impacted by the ability of the receiver to post = receive=20 buffers on the RQ/SRQ. Here the timeouts are application based and as = you wrote=20 can be configured to infinity. I agree that infinit number of = retries would=20 be a bad choice of </FONT> <FONT face=3DArial color=3D#0000ff = size=3D2>RNR=20 retry count. So, maybe we should recommend on selecting also the RNR = timeout and=20 retry count to be low too. I am also wondering what happens when there = is a slow=20 receiver, e.g. posting too slow on the RQ/SRQ, in which case the RNR Nak = will=20 happen very frequently and might cause the QP to get into error state = because of=20 RNR timeout exahusted.</FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>One=20 approach would be to downgrade such peers to IPoIB-UD. Other option = would be to=20 indirectly detect those cases and provide hints to the TCP stack to slow = down.=20 By indirectly detecting it, I am thinking along the lines of when the = send queue=20 becomes full, we can start dropping packets at the transmitter (RED = algorithm or=20 something similar). This packet drop can cause TCP to back off and = activate the=20 congestion flow and reduce the transmission rate. This has the right = impact in=20 my opinion.</FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>And,=20 the last thing that you mentioned has to do with TCP retransmission = triggering=20 when the IB is congested. IB packet delivery slows down because of = congestion in=20 the IB fabric (slowing down can be because of real congestion which=20 backpressures the transmitters or because of IB congestion management).=20 Eventhough there is no packet drop at the IB level and no retransmission = at the=20 IB level, still TCP times out. Luckily, TCP is doing RTT evaluation, so=20 hopefully this will be rare too. If it happens, then the desired effect = would be=20 to slow down the TCP requester (at the TCP level) which is OK. The side = effect=20 will be retransmission. And the big question is whether this is so bad = to have=20 this retransmission. BTW, I believe that also in this case, dropping = packets at=20 the transmitter will ease the problem.</FONT></SPAN></DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial color=3D#0000ff = size=3D2>Dror</FONT></SPAN></DIV> <BLOCKQUOTE style=3D"MARGIN-RIGHT: 0px"> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial = color=3D#0000ff=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D416134121-23032006><FONT face=3DArial = color=3D#0000ff=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN=20 class=3D416134121-23032006> </SPAN></DIV></BLOCKQUOTE></BODY></HTML>= =00 ------_=_NextPart_001_01C64ECC.6093202C-- --===============1810027139== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============1810027139==--