Chrony is making NTP Server offline on the system boot

Hello

We are having issue with chrony and it's making ntp server offline only on system boot.
Need hand and feet support in this regard.


Issue:  Chrony Marking NTP servers as offline Upon reboot of the nodes.
We do not see anything in the message log or puppet logs.

Customers' Multiple Data centers/Sites got affected by this issue.
Due to the marking of NTP offline, the clock went out of sync, resulting in a loss of traffic and causing a nationwide outage.
Output from some of the affected nodes.
This is not limited to CentOS 7, CentOS 8 or Alma Linux and happened in multiple environments.

The last reported issue was in CentOS 7
Currently, we captured the data on CentOS; they are the affected nodes in the customer's setup.

Note: In our implementation, we are not managing network interfaces with Network Manager and Looked into the following.
And it did not make any difference.

Another Note: Some nodes are affected in Multiple Data Centers, and the rest are fine.

In affected nodes, it's reproducible upon reboot and where we have not seen this issue, even in the same env on different nodes.
it's not happening over there.

We tried the following link from red hat, but it was also not applicable.

https://access.redhat.com/solutions/3968471

We made changes to our puppet to bring up interfaces only after network interfaces are up but it didn't make any difference.

Here is the output 

[qns@pcc01 ~]$ uname -a
Linux pcc01 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[qns@pcc01 ~]$ cat /etc/*-release
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL=https://www.centos.org/
BUG_REPORT_URL=https://bugs.centos.org/
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)

[qns@pcc01 ~]$ rpm -qa | sort -i | grep -i '(chrony|ntp)'
chrony-3.2-2.el7.x86_64
fontpackages-filesystem-1.44-8.el7.noarch
ntp-4.2.6p5-28.el7.centos.x86_64
ntpdate-4.2.6p5-28.el7.centos.x86_64

[qns@pcc01 ~]$ ps auxwwwf | grep -i '(chrony|ntp)'
ntp 30958 0.0 0.0 25720 1896 ? Ss 2021 0:50 /usr/sbin/ntpd -u ntp:ntp -g
qns 3827 0.0 0.0 112708 1012 pts/0 S+ 19:30 0:00 _ grep --color=auto -i (chrony|ntp)
[qns@pcc01 ~]$ date
Thu Sep 1 19:30:04 UTC 2022

[qns@pps02 ~]$ chronyc sources
210 Number of sources = 2
MS Name/IP address Stratum Poll Reach LastRx Last sample
^? lb01 2 10 0 44d +3581ms[+3581ms] +/- 16.0s
^? lb02 3 10 0 47d +46us[ +46us] +/- 15.9s

[qns@pps02 ~]$ chronyc tracking
Reference ID : AC1AF20C (lb02)
Stratum : 4
Ref time (UTC) : Thu Jun 09 10:20:05 2022
System time : 0.000000153 seconds slow of NTP time
Last offset : -0.000002489 seconds
RMS offset : 0.187102452 seconds
Frequency : 17.639 ppm slow
Residual freq : -0.000 ppm
Skew : 0.002 ppm
Root delay : 0.038741432 seconds
Root dispersion : 7.168707848 seconds
Update interval : 1031.8 seconds
Leap status : Normal

[qns@pps02 ~]$ chronyc sourcestats
210 Number of sources = 2
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
lb01 18 11 92m +6.918 0.022 +30.4s 42us
lb02 64 33 18h +0.000 0.000 +885us 18us
[qns@lb01 log]$ chronyc sources
210 Number of sources = 2
MS Name/IP address Stratum Poll Reach LastRx Last sample
^* NTP-server1.> 1 6 0 50d -25us[ -161us] +/- 17ms
^- NTP-server2.> 2 6 0 50d +685us[ +685us] +/- 34ms

[qns@lb01 
log]$ chronyc tracking
Reference ID : 9BAED650 (NTP-server1.com)
Stratum : 2
Ref time (UTC) : Wed Jun 08 06:58:03 2022
System time : 0.000000028 seconds slow of NTP time
Last offset : -0.000135603 seconds
RMS offset : 0.000135603 seconds
Frequency : 15.377 ppm slow
Residual freq : -31.511 ppm
Skew : 0.026 ppm
Root delay : 0.030828953 seconds
Root dispersion : 142.074813843 seconds
Update interval : 2.1 seconds
Leap status : Normal

[qns@lb01 log]$ chronyc sourcestats
210 Number of sources = 2
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
NTP-server1.> 4 4 6 -31.511 1011.791 -137.6s 149us
NTP-server2.> 4 4 6 -179.900 1376.575 -785.5s 152us

Thanks
Iqbal Singh 



-- 
To unsubscribe email chrony-users-request-kWFZVVI9zxvPqho9SqqRMmD2FQJk+8+b@public.gmane.org 
with "unsubscribe" in the subject.
For help email chrony-users-request-kWFZVVI9zxvPqho9SqqRMmD2FQJk+8+b@public.gmane.org 
with "help" in the subject.
Trouble?  Email [email protected]