Re: small change in the connected mode draft
"H.K. Jerry Chu" <[email protected]> Mon, 10 Apr 2006 12:09:50 -0700
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
--===============0053311172== Content-Type: multipart/alternative; boundary="----=_Part_4287_24005520.1144696190871" ------=_Part_4287_24005520.1144696190871 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline [Note my new email addr is [email protected]. The old [email protected] is no longer valid.] On 4/6/06, Michael Krause <[email protected]> wrote: > > At 08:41 AM 4/5/2006, Harald Alvestrand wrote: > > Michael Krause wrote: > > At 02:51 PM 3/23/2006, Dror Goldenberg wrote: > > > The case that you mentioned, RNR driven retransmission, is another case o= f > retransmission. It has nothing to do with congestion or packet drop in th= e > fabric. It is just being impacted by the ability of the receiver to post > receive buffers on the RQ/SRQ. Here the timeouts are application based an= d > as you wrote can be configured to infinity. I agree that infinit number o= f > retries would be a bad choice of RNR retry count. So, maybe we should > recommend on selecting also the RNR timeout and retry count to be low too= . I > am also wondering what happens when there is a slow receiver, e.g. postin= g > too slow on the RQ/SRQ, in which case the RNR Nak will happen very > frequently and might cause the QP to get into error state because of RNR > timeout exahusted. > > > The question raised was what impact will retransmission have on a ULP suc= h > as TCP which would treat timeouts as congestion events. I raised two cas= es > which are not easy to control at the IB protocol level since they are adm= in > managed: > - RNR which can lead to very long delays in the connection > transmission completion > - Arbitration which can lead to very slow forward progress on a > given QoS channel > > Both of these should be comprehended minimally as informative text to > guide developers and management solutions to do the right thing. They al= so > need to be aware that in the case of RNR, there is no single right answer > and its usage may entail long delays while the OS does whatever it needs > that triggered the RNR operation. > > My naive understanding of how IB stacks operate would be to give STRONG > advice to TCP/IP/IB-CM implementors that the IB interface should be fed > enough buffers to fill all the open TCP connection windows. If that is do= ne, > the only time RNR could occur is when the TCP window is closed - and in t= hat > situation, the application is in trouble anyway. > > > I am not aware of any OS or network stack that tracks the total available > window space of all TCP connections. Most implementations use statistica= l > provisioning and hope for the best that few packets will be dropped as a > result of a lack of buffer availability. In practice depending upon the > service rate and inbound packet delivery, this has not always been possib= le > to do and customers can see, especially with high-speed links, packet los= s > at the device level. > > One difference here for IB / RC is that there is a connection per endnode > pair so the problem can be mitigated. This does not mean that RNR will n= ot > occur or its occurrence will be rare. > In my mind the receive buffer allocation in the regular IPoIB/RC where all ULP/TCP/UDP sharing the same RC is much like IPoIB/UD in that the receive side link layer software will allocate and replenish the receive queue with anonymous memory as fast as possible. In that case the occurence of RNR should be like receive buffer underflow for UD; it should be very rare. When that happens the system is running out of any useable memory that all TCP connections should be advertising zero-window and ULP struggles to find memory too. If one wants to tie the receive buffer mgmt in RC to a particular set of ULP/TCP connections, possibly for better performance (e.g., avoiding copy), it's a different problem all together. First one must solve the problem of how to multiplex traffic into multiple RCs. Then presumably the transport should go into zero-window before running out of receive buffers to post. > Of course, this approach has its own set of cross-layer issues (and > doesn't work for UDP-based applications). But they are all about buffer > management within a single host, which makes layering violations slightly > less egregrious. > > > IB / RC is for all IP so it does not differentiate between UDP, TCP, > whatever is at IP and above. This makes it a bit more difficult to creat= e > even a reasonable view of the amount of receive buffers that need to be > posted per RC connection. Further, given the IP / above layers can now p= ost > very large buffers, all endpoints will see increased HOL blocking and thu= s > QoS issues as a result. This is where local management will need to be > involved to determine what exactly should flow over a RC connection or > perhaps one of several RC connections or just over the existing IP over I= B > UD QP in order to meet QoS requirements. > IPoIB/CM still requires a "MTU" size to be negotiated. Presumably there is = a tradeoff on the MTU size. A large size can help bulk data but may create HO= L blocking and waste a lot of memory for small msgs. Jerry Mike > > > > Harald > > _______________________________________________ > IPoverIB mailing list > [email protected] > https://www1.ietf.org/mailman/listinfo/ipoverib > > > _______________________________________________ > IPoverIB mailing list > [email protected] > https://www1.ietf.org/mailman/listinfo/ipoverib > > > ------=_Part_4287_24005520.1144696190871 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline [Note my new email addr is <a href=3D"mailto:[email protected]">hkjerry= [email protected]</a>. The old<br> <a href=3D"mailto:[email protected]">[email protected]</a> is no lo= nger valid.]<br><br><div><span class=3D"gmail_quote">On 4/6/06, <b class=3D= "gmail_sendername">Michael Krause</b> <<a href=3D"mailto:[email protected]= om"> [email protected]</a>> wrote:</span><blockquote class=3D"gmail_quote" st= yle=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex= ; padding-left: 1ex;"><div style=3D"direction: ltr;"><span class=3D"q"> <font size=3D"3">At 08:41 AM 4/5/2006, Harald Alvestrand wrote:<br> </font><blockquote type=3D"cite" cite=3D"http://"><font size=3D"3">Michael = Krause wrote:<br> </font><blockquote type=3D"cite" cite=3D"http://"><font size=3D"3">At 02:51= PM 3/23/2006, Dror Goldenberg wrote:<br> </font><blockquote type=3D"cite" cite=3D"http://"><font size=3D"3"><br> The case that you mentioned, RNR driven retransmission, is another case of retransmission. It has nothing to do with congestion or packet drop in the fabric. It is just being impacted by the ability of the receiver to post receive buffers on the RQ/SRQ. Here the timeouts are application based and as you wrote can be configured to infinity. I agree that infinit number of retries would be a bad choice of RNR retry count. So, maybe we should recommend on selecting also the RNR timeout and retry count to be low too. I am also wondering what happens when there is a slow receiver, e.g. posting too slow on the RQ/SRQ, in which case the RNR Nak will happen very frequently and might cause the QP to get into error state because of RNR timeout exahusted.</font></blockquote><font size=3D"3"= ><br> The question raised was what impact will retransmission have on a ULP such as TCP which would treat timeouts as congestion events. I raised two cases which are not easy to control at the IB protocol level since they are admin managed:<br> - RNR which can lead to very long delays in the connection transmission completion<br> - Arbitration which can lead to very slow forward progress on a given QoS channel<br><br> Both of these should be comprehended minimally as informative text to guide developers and management solutions to do the right thing. They also need to be aware that in the case of RNR, there is no single right answer and its usage may entail long delays while the OS does whatever it needs that triggered the RNR operation.</font></blockquote><fon= t size=3D"3">My naive understanding of how IB stacks operate would be to give STRONG advice to TCP/IP/IB-CM implementors that the IB interface should be fed enough buffers to fill all the open TCP connection windows. If that is done, the only time RNR could occur is when the TCP window is closed - and in that situation, the application is in trouble anyway.</font></blockquote><br></s= pan></div><div style=3D"direction: ltr;"> I am not aware of any OS or network stack that tracks the total available window space of all TCP connections. Most implementations use statistical provisioning and hope for the best that few packets will be dropped as a result of a lack of buffer availability. In practice depending upon the service rate and inbound packet delivery, this has not always been possible to do and customers can see, especially with high-speed links, packet loss at the device level. <br><br> One difference here for IB / RC is that there is a connection per endnode pair so the problem can be mitigated. This does not mean that RNR will not occur or its occurrence will be rare.</div></blockquote><div><br> In my mind the receive buffer allocation in the regular IPoIB/RC where all<= br> ULP/TCP/UDP sharing the same RC is much like IPoIB/UD in that the receive s= ide<br> link layer software will allocate and replenish the receive queue with anonymous memory as fast as possible. In that case the occurence of RNR should be like<br> receive buffer underflow for UD; it should be very rare. When that happens = the<br> system is running out of any useable memory that all TCP connections should= <br> be advertising zero-window and ULP struggles to find memory too.<br> <br> If one wants to tie the receive buffer mgmt in RC to a particular set of UL= P/TCP<br> connections, possibly for better performance (e.g., avoiding copy), it's a = different<br> problem all together. First one must solve the problem of how to multiplex = traffic<br> into multiple RCs. Then presumably the transport should go into zero-window= <br> before running out of receive buffers to post.<br> </div><blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb= (204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style= =3D"direction: ltr;"><span class=3D"q"><blockquote type=3D"cite" cite=3D"ht= tp://"><font size=3D"3"> Of course, this approach has its own set of cross-layer issues (and doesn't work for UDP-based applications). But they are all about buffer management within a single host, which makes layering violations slightly less egregrious.</font></blockquote><br></span></div> <div> IB / RC is for all IP so it does not differentiate between UDP, TCP, whatever is at IP and above. This makes it a bit more difficult to create even a reasonable view of the amount of receive buffers that need to be posted per RC connection. Further, given the IP / above layers can now post very large buffers, all endpoints will see increased HOL blocking and thus QoS issues as a result. This is where local management will need to be involved to determine what exactly should flow over a RC connection or perhaps one of several RC connections or just over the existing IP over IB UD QP in order to meet QoS requirements.</div> </blockquote><div><br> </div> <div>IPoIB/CM still requires a "MTU" size to be negotiated. Presu= mably there is a<br> tradeoff on the MTU size. A large size can help bulk data but may create HO= L<br> blocking and waste a lot of memory for small msgs.<br> <br> Jerry<br> </div> <br><blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(2= 04, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=3D= "direction: ltr;"> Mike</div><div style=3D"direction: ltr;"><span class=3D"e" id=3D"q_10a705cb= 3c8ce1a5_4"><br><br> <br> <blockquote type=3D"cite" cite=3D"http://"><font size=3D"3"> &nb= sp; = Harald<br><br> _______________________________________________<br> IPoverIB mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank" onclick=3D"return to= p.js.OpenExtLink(window,event,this)">[email protected]</a><br> <a href=3D"https://www1.ietf.org/mailman/listinfo/ipoverib" target=3D"_blan= k" onclick=3D"return top.js.OpenExtLink(window,event,this)"> https://www1.ietf.org/mailman/listinfo/ipoverib</a></font></blockquote> </span></div><br>_______________________________________________<br>IPoverI= B mailing list<br><a onclick=3D"return top.js.OpenExtLink(window,event,this= )" href=3D"mailto:[email protected]">[email protected]</a><br><a onclick=3D= "return top.js.OpenExtLink(window,event,this)" href=3D"https://www1.ietf.or= g/mailman/listinfo/ipoverib" target=3D"_blank"> https://www1.ietf.org/mailman/listinfo/ipoverib</a><br><br><br></blockquote= ></div><br> ------=_Part_4287_24005520.1144696190871-- --===============0053311172== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============0053311172==--