Re: [tsvwg] [CCWG] New Internet Draft: Congestion Signaling (CSIG)
"Rodney W. Grimes" <[email protected]>
| Newsgroups | gmane.ietf.irtf.iccrg,gmane.ietf.tsvwg |
|---|---|
| Message-ID | <[email protected]> |
Hesham, While I found a read of some of the IEEE 802.1Qau papers interesting, I have NOT been able to find anyone who has real world deployment data. In a datacenter or otherwise. Do you know of any or does anyone here have data to point at? Regards, Rod Grimes > Hi Abhiram, > IEEE 802.1 had a data center bridging TG which developed sometime ago the > spec for quantized congestion control (QCN). The spec is IEEE 802.1Qau > which is now part of 802.1Q. > > QCN provide a means for a bridge to notify a source of congestion causing > the source to reduce the flow rate. > > QCN is independent of upper layer protocol. It coexists with TCP (there is > a nested control loops of QCN and TCP flow control). > > QCN only works for unicast traffic, and operates over a domain where all > bridges and end stations support it. > > QCN does not require per flow state or per flow queuing in bridge. > > Over the years there have been proposals to enhance it, but the basic idea > stayed the same. > > How you compare your congestion signaling proposal with QCN. One obvious > difference is that yours is end-2end while QCN is not. But is end-end > congestion control the best way to go? > > Comments? > > Hesham > PS. I removed IPPM but kept other WG who may have an opinion on this issue. > > > On Wed, Sep 13, 2023, 5:09 PM Abhiram Ravi <abhiramr= > [email protected]> wrote: > > > Thank you for your comments. Responses inline. > > > > > Tom: The signaling being described in the draft is network layer > > information, and hence IMO should be conveyed in network layer headers. > > That's is L3 which conveniently is the average of L2+L4 :-) > > > Rachel: Implementing in L2 may be difficult to be used in e2e transport. > > > Hang: I agree L2 may not be the best choice to carry the congestion > > signaling end-to-end and more bits are needed. > > > > The choice of using L2+L4 is an intentional and important design choice, > > one that is driven by need, for practicality and ease of deployment in data > > center networks. We acknowledge that previous approaches have attempted to > > extend L3 or L4, but these approaches do not meet the requirements for > > deployability in a heterogeneous network, including interoperability and > > backward compatibility. CSIG targets specific yet important use cases while > > meeting these requirements. > > > > Section 7.1 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-choice-of-layer-2> > > of the draft explicitly addresses why such a design was chosen. > > > > The key reasons are: > > > > - > > > > Simplicity of device implementations of CSIG (both at NICs and > > switches), given the location of the CSIG-tag in the packet header is not > > affected by any header at layer 3 and above. CSIG-tag being the last tag > > before the ethertype also makes it simple for ASICs to implement it inside > > the L2 header. > > - > > > > Support for delivering congestion signals even in the presence of > > in-network tunneling and encryption, which are commonly used in data > > centers, especially for Cloud traffic and in Traffic engineered domains. > > CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN, > > Geneve, at a fixed offset in the packet header. This avoids the need to > > copy and relocate CSIG tags across inner / outer headers during > > encapsulation and decapsulation of packets, which is a complex hardware > > operation and would be necessary if implemented instead at layers 3 or > > higher. > > > > > Tom: The signaling being described in the draft is network layer > > information, and hence IMO should be conveyed in network layer headers. > > > > The information carried in L2 CSIG-tags are device signals or port signals > > from the bottleneck, and should not be viewed as network-layer signals. > > Although the bottleneck signal is also a property of the packet path in > > aggregate, the signal carried on the tag has a scope of one hop i.e., the > > measurement on one specific link or device on the path. The tag gets > > conditionally swapped on each device in a compare-and-replace manner as > > described in Section 4.3 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-operation-life-of-a-pa> > > > > Reflection is a property of the transport as described in Section 4.2, and > > hence CSIG reflection is implemented at layer 4 and is specific to the > > transport. This allows a clean separation of the mechanism used to collect > > signals from the network from the mechanism used for leveraging those > > signals at the transport. > > > > > Tom: Also, L2 is not really an end-to-end protocol (would legacy > > switches in the path also forward the header) > > > > Legacy devices can not only forward the packet, but also participate in > > the CSIG protocol via a software-assisted approach by treating CSIG-tags as > > VLAN-tags and leveraging VLAN resources to perform CSIG updates. This L2 > > approach is the only solution we?ve found which allows immediate deployment > > for certain signals with legacy devices in data centers, while native > > support for CSIG is easily achievable in future devices. CSIG stripping can > > also be exercised for interop when needed - see Section 6 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-incremental-deployment-of-c> > > for interoperability. > > > > > Tom: As the draft describes, this would entail having to support the > > protocol in multiple L2 and multiple L4 protocols > > > > Section 4.1 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-tag-header-format> > > lists how CSIG-tag operates with each relevant L2 encapsulation and Section > > 4.2 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-reflection-header-form> > > addresses implementation of reflection in relevant transports. > > > > > Tom: IMO, the proper carrier of the signal data is Hop-by-Hop Options. > > IOAM seems pretty extensible, so maybe it could be adapted to carry the > > signals of this draft? > > > Rachel: Of course it can work well in limited domain, like DC or HPC > > clusters. However, I also look for some solutions that could be able to go > > through internet > > > > CSIG is targeted at specific use cases in limited domains, primarily data > > centers, for congestion control, traffic management and network > > debuggability. The requirements for such use cases vary significantly from > > those of internet deployments, which is not in scope for CSIG. > > IOAM and hop-by-hop options in general are tuned for a broader set of use > > cases and do not meet all the requirements for the specific use cases that > > CSIG addresses, including interop with legacy devices and in-network > > tunneling, due to the placement of the header. Section 2 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-design-principles> > > speaks to these requirements. > > > > > Rachel: For L2 solution, it should be developed together with IEEE 802.1. > > > > Acknowledging that IEEE will be involved as well. > > > > > Sebastian: What happens if bottlenecks hand out immediate capacity/rate > > estimates and all greedy flows start increasing their cwin really fast? > > > > CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each > > flow can increase its rate in proportion to the % available bandwidth, so > > that in aggregate the flows respect the available bottleneck bandwidth with > > reduced risk of overshoot. See Section 8.1.2 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-using-maximum-link-utilizat> > > . > > > > > Ingemar: CSIG sounds to me like something that belongs more in ICCRG > > > > CSIG is applicable to use cases beyond Congestion control (which is an > > important one), such as traffic management and network debugging. Section > > 8 > > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-use-cases-defined-by-bottle> > > describes the use cases in more detail. > > > > > Tom: A related proposal might be FAST draft-herbert-fast. These might be > > complementary and options for both may be in the same packet > > > Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn > > > Ingemar: L4S is recently standardised and it is definitely gaining > > traction also in 3GPP > > > > Thank you for the pointers. We agree that these proposals are all > > complementary to CSIG. > > > > > > On Wed, Sep 13, 2023 at 5:20?AM <[email protected]> wrote: > > > >> Hi Ingemar, hi Sebastian, > >> > >> Skimming over the draft only, isn't CSIG about Ethernet-domains, while > >> L4S is E2E? > >> > >> Regards, > >> > >> Ruediger > >> > >> -----Urspr?ngliche Nachricht----- > >> Von: tsvwg <[email protected]> Im Auftrag von Sebastian Moeller > >> Gesendet: Mittwoch, 13. September 2023 13:42 > >> An: Ingemar Johansson S <ingemar.s.johansson= > >> [email protected]> > >> Cc: Vidhi Goel <[email protected]>; Shihang(Vincent) > >> <[email protected]>; tsvwg <[email protected]>; Jai > >> Kumar <[email protected]>; IETF IPPM WG <[email protected]>; Tom > >> Herbert <[email protected]>; [email protected]; Abhiram > >> Ravi <[email protected]>; Nandita Dukkipati < > >> [email protected]>; [email protected]; Huangyihong (Rachel) <rachel.huang= > >> [email protected]>; Naoshad Mehta <[email protected]> > >> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling > >> (CSIG) > >> > >> Hi Ingemar, > >> > >> > >> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson= > >> [email protected]> wrote: > >> > > >> > Hi > >> > > >> > I agree with Vihdi > >> > > >> > L4S is recently standardised > >> > >> [SM] In experimental track, the goal is currently to test whether > >> it can/should be deployed at scale.... > >> > >> > and it is definitely gaining traction also in 3GPP. We have an echo > >> system that is looking forward to having L4S widely deployed. > >> > Still, the congestion control aspects are not fully explored yet. One > >> interesting topic is if L4S allows to more safely deviate from additive > >> increase to make congestion control algorithms more quickly converge to > >> higher link capacity. There are a number of study topics around L4S > >> congestion control that are listed in e.g the TCP Prague draft. > >> > > >> > I cannot dictate what others should do with their time and money but > >> personally I'd prefer that the IETF explores L4S and its possibilities and > >> downsides before jumping on the next idea. > >> > >> [SM] L4S can be described as taking the ideas behind DCTCP and > >> making them fit for use over the internet*. Yet the signaling discussed > >> here is to be used in e.g. data center contexts where DCTCP is already used > >> and found lacking compared to newer methods operating on richer congestion > >> information (HPCC, Swift, Poseidon, ...). > >> Given that L4S essentially uses a multi-packet signal(**) to > >> report the "queue filling state" that is then stochastically distributed > >> over all concurrent flows, it seems obvious to me that reconstructing a > >> reliable estimate of on-path queueing will take some time and averaging for > >> each individual flow, I would guess that in some environments this delay > >> simply is too costly. > >> So L4S and CSIG seem complementary and in no way mutually > >> exclusive. > >> > >> > >> Regards > >> Sebastian > >> > >> > >> > >> *) I will not further discuss whether that is achieved or not as it seems > >> irrelevant here. > >> **) In essence transmission of congestion state via a 1-bit serial > >> channel, clocked at the (variable***) packet rate at the bottleneck. > >> ***) as packets are not of uniform size > >> > >> > > >> > CSIG sounds to me like something that belongs more in ICCRG or ? > >> > > >> > /Ingemar > >> > > >> > From: tsvwg <[email protected]> On Behalf Of Vidhi Goel > >> > Sent: Wednesday, 13 September 2023 00:59 > >> > To: Shihang(Vincent) <[email protected]> > >> > Cc: Huangyihong (Rachel) <[email protected]>; > >> Tom Herbert <[email protected]>; Abhiram Ravi > >> <[email protected]>; IETF IPPM WG <[email protected]>; > >> tsvwg <[email protected]>; [email protected]; [email protected]; Nandita Dukkipati > >> <[email protected]>; Naoshad Mehta <[email protected]>; Jai Kumar < > >> [email protected]> > >> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling > >> (CSIG) > >> > > >> > Not sure why we are coming up with so many new techniques when ECN just > >> works fine. > >> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to indicate > >> extent of congestion by marking it per packet. Adding more complexity to > >> any layer whether it is L2 or L3 doesn?t work well in deployments. Our goal > >> should be to simplify things and only add new headers if absolutely > >> necessary. > >> > > >> > Vidhi > >> > > >> > > >> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9= > >> [email protected]> wrote: > >> > > >> > Hi, > >> > I agree L2 may not be the best choice to carry the congestion signaling > >> end-to-end and more bits are needed. We have submitted a draft to carry the > >> multi-bits congestion signaling in L3. We call it Advanced ECN. See > >> https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/. > >> > > >> > Thanks, > >> > Hang > >> > > >> > From: CCWG <[email protected]> On Behalf Of Huangyihong (Rachel) > >> > Sent: Tuesday, September 12, 2023 5:41 PM > >> > To: Tom Herbert <[email protected]>; Abhiram Ravi > >> <[email protected]> > >> > Cc: IETF IPPM WG <[email protected]>; tsvwg <[email protected]>; [email protected]; > >> [email protected]; Nandita Dukkipati <[email protected]>; Naoshad Mehta < > >> [email protected]>; Jai Kumar <[email protected]> > >> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion > >> Signaling (CSIG) > >> > > >> > Hi, > >> > > >> > I also have the same feeling. Implementing in L2 may be difficult to be > >> used in e2e transport. Of course it can work well in limited domain, like > >> DC or HPC clusters. However, I also look for some solutions that could be > >> able to go through internet. We have submitted a draft to describe the > >> transport challenges. See > >> https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges > >> . > >> > > >> > I share the same opinion that the congestion signal is useful and > >> current 1-bit ECN solution is not fully sufficient. But I also feel like > >> the more straight way is to extend L3, or l4, like update IOAM, to carry > >> the information. For L2 solution, it should be developed together with IEEE > >> 802.1. > >> > > >> > BR? > >> > Rachel > >> > > >> > ???: iccrg <[email protected]> ?? Tom Herbert > >> > ????: 2023?9?10? 0:10 > >> > ???: Abhiram Ravi <[email protected]> > >> > ??: IETF IPPM WG <[email protected]>; tsvwg <[email protected]>; [email protected]; > >> [email protected]; Nandita Dukkipati <[email protected]>; Naoshad Mehta < > >> [email protected]>; Jai Kumar <[email protected]> > >> > ??: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG) > >> > > >> > Hi, thanks for draft! > >> > > >> > The first thing that stands out to me is the carrier of the new packet > >> headers. In the forward path it would be in L2 and in reflection it would > >> be L4. As the draft describes, this would entail having to support the > >> protocol in multiple L2 and multiple L4 protocols-- that's going to be a > >> pretty big lift! Also, L2 is not really an end-to-end protocol (would > >> legacy switches in the path also forward the header)l?). > >> > > >> > The signaling being described in the draft is network layer > >> information, and hence IMO should be conveyed in network layer headers. > >> That's is L3 which conveniently is the average of L2+L4 :-) > >> > > >> > IMO, the proper carrier of the signal data is Hop-by-Hop Options. This > >> is end-to-end and allows modification of data in-flight. The typical > >> concern with Hop-by-Hop Options is high drop rates on the Internet, however > >> in this case the protocol is explicitly confined to a limited domain so I > >> don't see that as a blocking issue for this use case. > >> > > >> > The information being carried seems very similar to that of IOAM (IOAM > >> uses Hop-by-Hop Options and supports reflection). I suppose the differences > >> are that this protocol is meant to be consumed by the transport Layer and > >> the data is a condensed summary of path characteristics. IOAM seems pretty > >> extensible, so maybe it could be adapted to carry the signals of this draft? > >> > > >> > A related proposal might be FAST draft-herbert-fast. Where the CSIG is > >> network to host signaling, FAST is host to network signaling for the > >> purposes of requesting network services. These might be complementary and > >> options for both may be in the same packet. FAST also uses reflection, so > >> we might be able to leverage some common implementation at a destination. > >> > > >> > Tom > >> > > >> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr= > >> [email protected]> wrote: > >> > Hi IPPM folks, > >> > > >> > I am pleased to announce the publication of a new internet draft, > >> Congestion Signaling (CSIG): > >> https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/ > >> > > >> > CSIG is a new end-to-end packet header mechanism for in-band signaling > >> that is simple, efficient, deployable, and grounded in concrete use cases > >> of congestion control, traffic management, and network debuggability. We > >> believe that CSIG is an important new protocol that builds on top of > >> existing in-band network telemetry protocols. > >> > > >> > We encourage you to read the CSIG draft and provide your feedback and > >> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we > >> believe that this work may be of interest to their members as well. > >> > > >> > Thank you for your time and consideration. > >> > > >> > Sincerely, > >> > Abhiram Ravi > >> > On behalf of the CSIG authors > >> > >> > > > > -- > > Abhiram Ravi > > -- > > CCWG mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/ccwg > > -- Rod Grimes [email protected]