Re: small change in the connected mode draft
Michael Krause <[email protected]> Thu, 06 Apr 2006 10:54:28 -0700
| Newsgroups | gmane.ietf.ipoib |
|---|---|
| Message-ID | <[email protected]> |
--===============1016994745== Content-Type: multipart/alternative; boundary="=====================_335034554==.ALT" --=====================_335034554==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed At 08:41 AM 4/5/2006, Harald Alvestrand wrote: >Michael Krause wrote: >>At 02:51 PM 3/23/2006, Dror Goldenberg wrote: >>> >>>The case that you mentioned, RNR driven retransmission, is another case >>>of retransmission. It has nothing to do with congestion or packet drop >>>in the fabric. It is just being impacted by the ability of the receiver >>>to post receive buffers on the RQ/SRQ. Here the timeouts are application >>>based and as you wrote can be configured to infinity. I agree that >>>infinit number of retries would be a bad choice of RNR retry count. So, >>>maybe we should recommend on selecting also the RNR timeout and retry >>>count to be low too. I am also wondering what happens when there is a >>>slow receiver, e.g. posting too slow on the RQ/SRQ, in which case the >>>RNR Nak will happen very frequently and might cause the QP to get into >>>error state because of RNR timeout exahusted. >> >>The question raised was what impact will retransmission have on a ULP >>such as TCP which would treat timeouts as congestion events. I raised >>two cases which are not easy to control at the IB protocol level since >>they are admin managed: >> - RNR which can lead to very long delays in the connection >> transmission completion >> - Arbitration which can lead to very slow forward progress on a >> given QoS channel >> >>Both of these should be comprehended minimally as informative text to >>guide developers and management solutions to do the right thing. They >>also need to be aware that in the case of RNR, there is no single right >>answer and its usage may entail long delays while the OS does whatever it >>needs that triggered the RNR operation. >My naive understanding of how IB stacks operate would be to give STRONG >advice to TCP/IP/IB-CM implementors that the IB interface should be fed >enough buffers to fill all the open TCP connection windows. If that is >done, the only time RNR could occur is when the TCP window is closed - and >in that situation, the application is in trouble anyway. I am not aware of any OS or network stack that tracks the total available window space of all TCP connections. Most implementations use statistical provisioning and hope for the best that few packets will be dropped as a result of a lack of buffer availability. In practice depending upon the service rate and inbound packet delivery, this has not always been possible to do and customers can see, especially with high-speed links, packet loss at the device level. One difference here for IB / RC is that there is a connection per endnode pair so the problem can be mitigated. This does not mean that RNR will not occur or its occurrence will be rare. >Of course, this approach has its own set of cross-layer issues (and >doesn't work for UDP-based applications). But they are all about buffer >management within a single host, which makes layering violations slightly >less egregrious. IB / RC is for all IP so it does not differentiate between UDP, TCP, whatever is at IP and above. This makes it a bit more difficult to create even a reasonable view of the amount of receive buffers that need to be posted per RC connection. Further, given the IP / above layers can now post very large buffers, all endpoints will see increased HOL blocking and thus QoS issues as a result. This is where local management will need to be involved to determine what exactly should flow over a RC connection or perhaps one of several RC connections or just over the existing IP over IB UD QP in order to meet QoS requirements. Mike > Harald > >_______________________________________________ >IPoverIB mailing list >[email protected] >https://www1.ietf.org/mailman/listinfo/ipoverib --=====================_335034554==.ALT Content-Type: text/html; charset="us-ascii" <html> <body> <font size=3>At 08:41 AM 4/5/2006, Harald Alvestrand wrote:<br> <blockquote type=cite class=cite cite="">Michael Krause wrote:<br> <blockquote type=cite class=cite cite="">At 02:51 PM 3/23/2006, Dror Goldenberg wrote:<br> <blockquote type=cite class=cite cite=""><br> The case that you mentioned, RNR driven retransmission, is another case of retransmission. It has nothing to do with congestion or packet drop in the fabric. It is just being impacted by the ability of the receiver to post receive buffers on the RQ/SRQ. Here the timeouts are application based and as you wrote can be configured to infinity. I agree that infinit number of retries would be a bad choice of RNR retry count. So, maybe we should recommend on selecting also the RNR timeout and retry count to be low too. I am also wondering what happens when there is a slow receiver, e.g. posting too slow on the RQ/SRQ, in which case the RNR Nak will happen very frequently and might cause the QP to get into error state because of RNR timeout exahusted.</blockquote><br> The question raised was what impact will retransmission have on a ULP such as TCP which would treat timeouts as congestion events. I raised two cases which are not easy to control at the IB protocol level since they are admin managed:<br> - RNR which can lead to very long delays in the connection transmission completion<br> - Arbitration which can lead to very slow forward progress on a given QoS channel<br><br> Both of these should be comprehended minimally as informative text to guide developers and management solutions to do the right thing. They also need to be aware that in the case of RNR, there is no single right answer and its usage may entail long delays while the OS does whatever it needs that triggered the RNR operation.</blockquote>My naive understanding of how IB stacks operate would be to give STRONG advice to TCP/IP/IB-CM implementors that the IB interface should be fed enough buffers to fill all the open TCP connection windows. If that is done, the only time RNR could occur is when the TCP window is closed - and in that situation, the application is in trouble anyway.</font></blockquote><br> I am not aware of any OS or network stack that tracks the total available window space of all TCP connections. Most implementations use statistical provisioning and hope for the best that few packets will be dropped as a result of a lack of buffer availability. In practice depending upon the service rate and inbound packet delivery, this has not always been possible to do and customers can see, especially with high-speed links, packet loss at the device level. <br><br> One difference here for IB / RC is that there is a connection per endnode pair so the problem can be mitigated. This does not mean that RNR will not occur or its occurrence will be rare. <br><br> <br> <blockquote type=cite class=cite cite=""><font size=3>Of course, this approach has its own set of cross-layer issues (and doesn't work for UDP-based applications). But they are all about buffer management within a single host, which makes layering violations slightly less egregrious.</font></blockquote><br> IB / RC is for all IP so it does not differentiate between UDP, TCP, whatever is at IP and above. This makes it a bit more difficult to create even a reasonable view of the amount of receive buffers that need to be posted per RC connection. Further, given the IP / above layers can now post very large buffers, all endpoints will see increased HOL blocking and thus QoS issues as a result. This is where local management will need to be involved to determine what exactly should flow over a RC connection or perhaps one of several RC connections or just over the existing IP over IB UD QP in order to meet QoS requirements.<br><br> Mike<br><br> <br> <blockquote type=cite class=cite cite=""><font size=3> Harald<br><br> _______________________________________________<br> IPoverIB mailing list<br> [email protected]<br> <a href="https://www1.ietf.org/mailman/listinfo/ipoverib" eudora="autourl"> https://www1.ietf.org/mailman/listinfo/ipoverib</a></font></blockquote> </body> </html> --=====================_335034554==.ALT-- --===============1016994745== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib --===============1016994745==--