Re: Google's GSLB gone wrong: Southern Europe goes Middle East (UAE / lcmcta)
Hank Nussbacher via NANOG <[email protected]>
| Newsgroups | gmane.org.operators.nanog |
|---|---|
| Message-ID | <[email protected]> |
On 12/03/2026 22:12, Lukas Tribus via NANOG wrote: I do not see a site in UAE: https://gcping.com/ Doha and Dammam but no UAE. Perhaps gcping hasn't been updated with the UAE site. Don't know. -Hank > Hello, > > here's an interesting case of a global load balancing SNAFU: > > Google DNS-load-balances traffic based on the customers /24 between > the available PoPs for the specific geolocation area. *SOME* those > /24s at *ALL* ISPs in Southern Europe are going to the Middle East, > more specifically the UAE (PTR prefix is "lcmcta"). So the the > customers in Southern Europe hit Google Frontends on another > continent, obviously with a latency well above 100ms. > > It appears all of Southern Europe is impacted, I have found this to be > the case for at least: > > - Italy > - Croatia > - Hungary > - Serbia > - Romania > - Bulgaria > - Albania > - Greece > - Malta > > > This affects websites like google.com, mail.google.com and youtube.com > (although most likely not the video streaming itself though). The > impact due to the latency and traffic is probably not extreme; AFAIK > it does not affect Youtube stream itself (just the main website). > > It has been going on for probably a week or more, Google NOC Ticket > 2588682096 is open since March 10th, 2176711238 since March 11th, but > no luck so far in those tickets. > > > The balancing occurs via the authoritative DNS servers at Google, > looking either at the EDNS client subnet (also used with 8.8.8.8 but > not 1.1.1.1), or without ENDS at the source IP of the DNS request. > > > Thanks to EDNS client subnet this can be easily simulated; in the > following example we see that for clients in 2.112.4.0/24 (Telecom > Italia), the Google Frontend IP is 142.250.186.238 which is a unicast > Google node in the United Arab Emirates (PTR prefix is "lcmcta"): > > $ dig +short +subnet=2.112.4.0/24 -tA @ns1.google.com. google.com. > 142.250.186.238 > $ dig +short -x 142.250.186.238 > lcmcta-af-in-f14.1e100.net. > $ > > > More data from 6 large networks in Italy is available here, but again > it affects every single ISP in the multiple countries: > > https://github.com/lukastribus/ggc-italy-middle-east-routing > > > > The most likely reason for this I believe is a misconfigured GGC > cluster or Edge PoP in the United Arab Emirates - the error being that > it is erroneously set to serve content to clients in Southern Europe > as well a Middle East. If I were to speculate even further, this may > be have been an emergency response related to the ongoing attacks in > the region. > > If someone would happen to have a contact that can escalate the > aforementioned Google Tickets 2588682096 and 2176711238 to the correct > teams, that would be very much appreciated. > > > I hope this was an interesting read :) > > > cheers, > lukas > _______________________________________________ > NANOG mailing list > https://lists.nanog.org/archives/list/[email protected]/message/3EJUOZJKKLBCZTMXASLJKUQBFHOJHVFJ/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/DEM3674BGOEWT3ALNO5HY4O3KUD24ECI/