Restarting the chrony service brakes chrony

I have a strange problem: When chrony is started after boot all works fine.

chronyc --version
chronyc (chrony) version 4.1 (-READLINE +SECHASH +IPV6 -DEBUG)

After 5 min:
chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- excalibur.prolixium.com       2   6   377    13  +5513us[+4097us] +/-   98ms
^+ gowest.hojmark.net            2   6   371    13  -4181us[-5596us] +/-   59ms
^* ntp0.bgwlan.nl                1   6   327    12   -138us[-1553us] +/-   26ms
^? cloud.discloud.biz            2   6     3    13  +2145us[ +729us] +/-   26ms

Now I restart chrony
sudo service chrony restart
and I get after 1 min:
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ntp18.kashra-server.com       0   6     0     -     +0ns[   +0ns] +/-    0ns
^? dns02.wsrs.net                0   6     0     -     +0ns[   +0ns] +/-    0ns
^? time.cloudflare.com           0   6     0     -     +0ns[   +0ns] +/-    0ns
^? ntp1.trans-ix.nl              0   6     0     -     +0ns[   +0ns] +/-    0ns

And after 2min
[bmterra@armxl ~]$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? nts1.time.nl                  0   6     0     -     +0ns[   +0ns] +/-    0ns
^? ntp18.kashra-server.com       0   6     0     -     +0ns[   +0ns] +/-    0ns
^? dns02.wsrs.net                0   6     0     -     +0ns[   +0ns] +/-    0ns
^? time.cloudflare.com           0   6     0     -     +0ns[   +0ns] +/-    0ns
^? docker01.rondie.nl            0   6     0     -     +0ns[   +0ns] +/-    0ns
^? ntp1.trans-ix.nl              0   6     0     -     +0ns[   +0ns] +/-    0ns
^? aardbei.vanderzwet.net        0   6     0     -     +0ns[   +0ns] +/-    0ns

I mean, I can ping the time servers:
ping nts1.time.nl
PING nts1.time.nl(nts1.time.nl (2a00:d78:0:712:94:198:159:11)) 56 data bytes
64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=1 ttl=56 time=53.5 ms
64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=2 ttl=56 time=36.6 ms
64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=3 ttl=56 time=34.8 ms

--- nts1.time.nl ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 5ms
rtt min/avg/max/mdev = 34.818/41.657/53.519/8.423 ms

ping ntp18.kashra-server.com
PING ntp18.kashra-server.com (188.165.11.86) 56(84) bytes of data.
64 bytes from ntp18.kashra-server.com (188.165.11.86): icmp_seq=1 ttl=51 time=91.1 ms
64 bytes from ntp18.kashra-server.com (188.165.11.86): icmp_seq=2 ttl=51 time=76.7 ms

--- ntp18.kashra-server.com ping statistics ---
3 packets transmitted, 2 received, 33.3333% packet loss, time 5ms
rtt min/avg/max/mdev = 76.662/83.883/91.104/7.221 ms

Any idea why chrony stops to work after a restart of the service?

Or how to debug that further?

Uwe