Google's GSLB gone wrong: Southern Europe goes Middle East (UAE / lcmcta)

Hello,

here's an interesting case of a global load balancing SNAFU:

Google DNS-load-balances traffic based on the customers /24 between
the available PoPs for the specific geolocation area. *SOME* those
/24s at *ALL* ISPs in Southern Europe are going to the Middle East,
more specifically the UAE (PTR prefix is "lcmcta"). So the the
customers in Southern Europe hit Google Frontends on another
continent, obviously with a latency well above 100ms.

It appears all of Southern Europe is impacted, I have found this to be
the case for at least:

- Italy
- Croatia
- Hungary
- Serbia
- Romania
- Bulgaria
- Albania
- Greece
- Malta


This affects websites like google.com, mail.google.com and youtube.com
(although most likely not the video streaming itself though). The
impact due to the latency and traffic is probably not extreme; AFAIK
it does not affect Youtube stream itself (just the main website).

It has been going on for probably a week or more, Google NOC Ticket
2588682096 is open since March 10th, 2176711238 since March 11th, but
no luck so far in those tickets.


The balancing occurs via the authoritative DNS servers at Google,
looking either at the EDNS client subnet (also used with 8.8.8.8 but
not 1.1.1.1), or without ENDS at the source IP of the DNS request.


Thanks to EDNS client subnet this can be easily simulated; in the
following example we see that for clients in 2.112.4.0/24 (Telecom
Italia), the Google Frontend IP is 142.250.186.238 which is a unicast
Google node in the United Arab Emirates (PTR prefix is "lcmcta"):

$ dig +short +subnet=2.112.4.0/24 -tA @ns1.google.com. google.com.
142.250.186.238
$ dig +short -x 142.250.186.238
lcmcta-af-in-f14.1e100.net.
$


More data from 6 large networks in Italy is available here, but again
it affects every single ISP in the multiple countries:

https://github.com/lukastribus/ggc-italy-middle-east-routing



The most likely reason for this I believe is a misconfigured GGC
cluster or Edge PoP in the United Arab Emirates - the error being that
it is erroneously set to serve content to clients in Southern Europe
as well a Middle East. If I were to speculate even further, this may
be have been an emergency response related to the ongoing attacks in
the region.

If someone would happen to have a contact that can escalate the
aforementioned Google Tickets 2588682096 and 2176711238 to the correct
teams, that would be very much appreciated.


I hope this was an interesting read :)


cheers,
lukas
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/[email protected]/message/3EJUOZJKKLBCZTMXASLJKUQBFHOJHVFJ/