Re: Strange NTP packet loss

Hello Stefan

On 27.04.2019 11:36, Stefan Brand wrote:

> After quite some debugging we realized that the packet loss only occurs for NTP packets via one of our transit providers, namely Liberty Global (AS6830).

I had do dig in my e-mail archive as I already ran into this way back in 
March 2017. I did discuss this with the Pool team (ticket #5636). I my 
case it is the IP address (62.2.85.186) on a UPC Business Cable Line 
("Fiber Power"). The NTP Pool is aware, that when traffic goes from 
their monitoring system through NTT, that then occasionally (or more) 
packet loss does happen. From what we discovered back then, it seems 
that NTT may have some DDoS protection in place. From the Pool team I 
even got that information (unfortunately the PDF is not available any more):
"After reading
http://www.nttcomsecurity.com/uploads/files/US_2015_GTIR-ddos-observations_Public_Approved_V1.pdf
I won't be surprised if NTT has some automatic rate limiting on NTP 
traffic in place."

> We queried their support, our theory being that they're trying to do some sort of DDoS protection for NTP reflection attacks.
> However they aren't aware of anything like this and also couldn't figure out why this is happening.

As the problem persisted I also did open a ticket with UPC in October 
2017 (ticket B2BCH0000641230 and B2BCH0000643146). But they did not 
really understand the problem and did only update/downgrade the firmware 
on my modem (I knew that this will not help), but they did not see any 
problems. :-(

As this IP address already had bad scoring for a while, it got dropped 
out of the pool a few weeks later. In the past I had tried to add it 
back a few times, but it still fails even for the initial check the pool 
is doing. I have just tried a few times again without any luck.

But luckily I still have some other servers in the Pool: 
https://www.pool.ntp.org/user/home4u.ch

> So I was wondering, has anyone else encountered this issue or something similar?
> We worked around the issues by routing the traffic around AS6830 but this still bothers me somehow.

Back then the problem was worse, as the routing was asymmetrical, and 
the path was only from the ntp pool monitoring through NTT, but not back 
from UPC. As I see it now, the routing is symmetrical through NTT.

The pool as a tracroute tool available at (change the IP to your own):
https://trace.ntppool.org/traceroute/62.2.85.186

As the pool provides some nice things with e.g. the data of the last 50 
checks (change to your IP, mine does not have any data):
https://www.pool.ntp.org/scores/62.2.85.186/log?limit=50
You see that it does a check around every 20 minutes (the timestamp is 
in seconds since epoch). I did run a tcpdump at my end, when a packet 
arrived, then also the pool monitoring system got my answer back and 
data got recorded. So it was clear to me, that packets got dropped on 
the way from the Pool monitoring to me.

I did monitor on my end with:
tcpdump -tt -i em1 host 207.171.3.17 and port 123
But it may be that the IP has changed now, so you have to ask the Pool 
Team for current information. If you already got e-mails from Ask with 
subject "NTP Pool: Problems with your NTP server ...", just write back 
to it. :)

Eventually it is also helpful to subscribe to the Pool mailing list (or 
search in their archive): http://lists.ntp.org/listinfo/pool

Hope this information helps and you are in a better position to tell UPC 
that their upstream NTT is doing something not very nice.


Best regards,
Fabian


_______________________________________________
swinog mailing list
[email protected]
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog