Time synchronization is very sensitive task that need to be taken into consideration very seriously specially for HPC and BIG data area. Here I will explain how to use Chrony in Centos 7.2 for time synchronization.
The protocol that I will used and probably is the only protocol for time synchronization is NTP (Network time Protocol). NTP is a protocol to keep servers time synchronized.
Two main packages are used in RHEL 7 to set up the time synchronization:
- ntp: this is the classic package, already existing in RHEL 6, RHEL 5, etc.
- chrony: this is a new solution better suited for portable PC or servers with network connection problems (time synchronization is quicker).
Chrony is an NTP client and server designed to work well without permanent network connection. It uses different algorithms which has several advantages over NTP package.The master server is the only server that is synchronizing time against external clock sources which are internet NTP servers.
Steps need to be taken:
1. First make sure the Time Zone that the Master node and compute nodes has are matching. It reduce the confusion in the future in order to be sure that all systems are sync. For this we can use date command:
[root@hadoop-master]# date
Tue Aug 16 17:47:10 CEST 2016
2. Usually chrony is installed by default. If not we need to install it in Master node as well as all compute nodes.
[root@hadoop-master ~]# yum -y install chrony
3. There is a file called chrony.conf in both master node and all compute nodes (/etc/chrony.conf) that need to be configured.
a. in Master node
By default the servers that our Master node is synching is Internet servers as follow:
server 0.centos.pool.ntp.org iburst
However if we have a internal NTP server, we can put the IP add here like:
server 192.168.1.10
Another main point is to let internal network (compute nodes) connect to this server (Master node) for time synchronization.
# Allow NTP client access from local network.
allow 10.0.0.0/24
b. in Compute nodes
There is a very exact file as master node here too located same in /etc/chrony.conf. We need to add only One line to it:
server 10.0.0.254
As 10.0.0.254 is the Master node IP address facing the compute nodes. It is better to restart the chronyd service several times until it sync completely with Master node.
3. Usefull commands for testing:
[root@hadoop-master ~]# chronyc tracking
Reference ID : 85.220.190.246 (clint.blazing.de)
Stratum : 3
Ref time (UTC) : Tue Aug 16 16:08:59 2016
System time : 0.000142668 seconds slow of NTP time
Last offset : -0.000049265 seconds
RMS offset : 0.000051135 seconds
Frequency : 39.128 ppm fast
Residual freq : -0.002 ppm
Skew : 0.035 ppm
Root delay : 0.008658 seconds
Root dispersion : 0.016905 seconds
Update interval : 260.3 seconds
Leap status : Normal
[root@hadoop-master ~]# ssh node-01
Last login: Tue Aug 16 19:34:37 2016 from hadoop-master.hpc.cluster
[root@node-01 ~]# chronyc tracking
Reference ID : 10.0.0.254 (hadoop-master.hpc.cluster)
Stratum : 4
Ref time (UTC) : Tue Aug 16 16:09:56 2016
System time : 0.000000103 seconds slow of NTP time
Last offset : -0.000003852 seconds
RMS offset : 0.000030250 seconds
Frequency : 40.448 ppm slow
Residual freq : +0.000 ppm
Skew : 0.084 ppm
Root delay : 0.008897 seconds
Root dispersion : 0.016933 seconds
Update interval : 128.1 seconds
Leap status : Normal
I did it from both Master node and one of the compute nodes and as can be seen the Leap status is Normal.
Important points regarding above commands:
a. Reference ID: It shows with which server we are synchronizing. If the IP add is 127.127.1.1 means our system is not synching with any external source but local mode is enabled.
4. To get the information about the current time sources that chronyd is accessing
[root@node-01 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* hadoop-master.hpc.cluster 3 8 377 113 +113us[ +116us] +/- 22ms
5. Other usefull info can be also taken with following command:
[root@hadoop-master ~]# chronyc sourcestats
210 Number of sources = 4
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
x.fu110.de 21 10 42m +0.062 0.039 -1451us 28us
pyxis.my-rz.de 21 12 41m +0.036 0.085 -840us 64us
gromit.nocabal.de 19 12 41m -0.011 0.030 +726us 25us
clint.blazing.de 12 7 37m -0.253 0.161 +839us 82us
HPC, Big data & information security 