IETF 126 Network Report
"Joe Clarke \(jclarke\)" <[email protected]>
| Newsgroups | gmane.ietf.general |
|---|---|
| Message-ID | <CH2PR11MB8867A78BC173861C8059BB35B8DD2@CH2PR11MB8867.namprd11.prod.outlook.com> |
As the new NOC lead, I want to get into a habit of providing a brief report of the previous meeting’s network operations on behalf of the NOC team. As you may know, the NOC consists of a team of volunteers, network contractors (Linespeed), and remote/onsite participation providers (Meetecho). A list of all our names can be found on slide 7 of Jay’s LLC plenary deck: https://datatracker.ietf.org/meeting/126/materials/slides-126-ietf-sessa-ietf-126-ietf-llc-briefing-00. A big thanks to each and every one of the team. If you’d rather have something a bit more graphical, I have a presentation format of the report at https://noc.ietf.org/meeting-reports/ietf126/IETF_126_Vienna_Network_Report.pdf. Let me know if you have any questions. Going forward, we’ll be adding more statistics to these reports. The meeting network in Vienna ran well and served the community reliably all week. It carried 32.02 TB of total Internet traffic, made up of 13.71 TB of IPv6 and 10.70 TB of IPv4. Note that we can only break IPv4 and IPv6 traffic on the Juniper SRX. The router responsible for the hotel network does not report IP protocol counters, so the IPv4 and IPv6 numbers cover the meeting network only, whereas the total traffic number includes both meeting and hotel traffic. Downstream traffic peaked at 1 Gbps several times, which saturated our single 1G external circuit at those moments, yet no one reported a problem. Wireless use peaked at more than 1,000 concurrent clients on the main "ietf" SSID, with eduroam and the dual-stack SSID carrying the rest. We continue to run IPv6-Mostly per RFC 8925 on the "ietf" SSID. The legacy SSID saw almost no use, but we keep it as an absolute basic catch-all. Its name changes each meeting and it is not broadcast. The dual-stack SSID remained available throughout and gave any client with trouble a dependable place to connect. None of this happens without our sponsors. We thank Cisco and Juniper for their equipment donations, and next layer for providing our connectivity. Their support is what makes the meeting network possible. Setup was not without difficulty. The venue had recently handed its IT management to a third party, and that vendor had quietly removed the hotel wireless configuration we had staged before we arrived. The onsite crew put it back quickly. A few coverage gaps on certain floors also had to be worked out during the first two days. On the external side, we were limited to a single 1G circuit because the sponsoring provider could not get the local loop into the hotel equipment room in time. We staged a 5G backup path as a contingency, but it was not needed. During shipping, US Customs drilled inspection holes into the flight cases, which left them no longer air or watertight. This is a rare event, and the freight forwarder is working to make it right. Onsite incidents were few and were handled well. During the hackathon, a spilled drink into a power strip tripped an entire electrical sub-panel, which briefly took down network, projection, and Meetecho across three rooms until a building engineer was found and power was restored (please watch those drinks on the crowded hackathon tables!). There was also a short outage of roughly 15 to 20 minutes on the hotel WiFi when the venue applied client isolation at our request to combat rogue DHCP servers. Both were resolved with no measurable disruption to the community and no tickets. The team also moved several projects forward. We began the WiFi 7 transition on the main SSID, upgraded the wireless controller, improved our overall network automation, added an AI agent to assist with deployment and teardown, stood up new monitoring and dashboards, and modernized our Git and CI/CD infrastructure. We found a compatibility gap between WiFi 7 and certain Linux wireless clients. We documented<https://www.ietf.org/meeting/meeting-network-information/> it on the meeting network info page and are pursuing fixes with Cisco and the open-source community. On the systems side, one trend is shaping how we operate our infrastructure. The pace of AI-driven security research is surfacing vulnerabilities much faster than before, which means our systems now need more frequent updates and reboots than our traditional practice of freezing everything once the meeting begins. We had to apply one such update during the meeting week, though we were able to do it after hours with no impact on the community. We will need a clear policy on how to handle these updates safely during a live meeting. We also added a VM in AWS to give us one more place to host our post-meeting backup archives. The meeting network met the needs of the community, and the team is already planning for IETF 127. Joe on behalf of the Network Operations Team