Time Synchronization

Time synchronization is very sensitive task that need to be taken into consideration very seriously specially for HPC and BIG data area. Here I will explain how to use Chrony in Centos 7.2 for time synchronization.

The protocol that I will used and probably is the only protocol for time synchronization is NTP (Network time Protocol). NTP is a protocol to keep servers time synchronized.

Two main packages are used in RHEL 7 to set up the time synchronization:

  • ntp: this is the classic package, already existing in RHEL 6, RHEL 5, etc.
  • chrony: this is a new solution better suited for portable PC or servers with network connection problems (time synchronization is quicker).

Chrony is an NTP client and server designed to work well without permanent network connection. It uses different algorithms which has several advantages over NTP package.The master server is the only server that is synchronizing time against external clock sources which are internet NTP servers.

Steps need to be taken:

1. First make sure the Time Zone that the Master node and compute nodes has are matching. It reduce the confusion in the future in order to be sure that all systems are sync. For this we can use date command:

[root@hadoop-master]# date
Tue Aug 16 17:47:10 CEST 2016

2. Usually chrony is installed by default. If not we need to install it in Master node as well as all compute nodes.

[root@hadoop-master ~]# yum -y install chrony

3. There is a file called chrony.conf in both master node and all compute nodes (/etc/chrony.conf) that need to be configured.

a. in Master node

By default the servers that our Master node is synching is Internet servers as follow:

server 0.centos.pool.ntp.org iburst

However if we have a internal NTP server, we can put the IP add here like:


Another main point is to let internal network (compute nodes) connect to this server (Master node) for time synchronization.

# Allow NTP client access from local network.

b. in Compute nodes

There is a very exact file as master node here too located same in /etc/chrony.conf. We need to add only One line to it:


As is the Master node IP address facing the compute nodes. It is better to restart the chronyd service several times until it sync completely with Master node.

3. Usefull commands for testing:

[root@hadoop-master ~]# chronyc tracking
Reference ID    : (clint.blazing.de)
Stratum         : 3
Ref time (UTC)  : Tue Aug 16 16:08:59 2016
System time     : 0.000142668 seconds slow of NTP time
Last offset     : -0.000049265 seconds
RMS offset      : 0.000051135 seconds
Frequency       : 39.128 ppm fast
Residual freq   : -0.002 ppm
Skew            : 0.035 ppm
Root delay      : 0.008658 seconds
Root dispersion : 0.016905 seconds
Update interval : 260.3 seconds
Leap status     : Normal

[root@hadoop-master ~]# ssh node-01
Last login: Tue Aug 16 19:34:37 2016 from hadoop-master.hpc.cluster
[root@node-01 ~]# chronyc tracking
Reference ID    : (hadoop-master.hpc.cluster)
Stratum         : 4
Ref time (UTC)  : Tue Aug 16 16:09:56 2016
System time     : 0.000000103 seconds slow of NTP time
Last offset     : -0.000003852 seconds
RMS offset      : 0.000030250 seconds
Frequency       : 40.448 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.084 ppm
Root delay      : 0.008897 seconds
Root dispersion : 0.016933 seconds
Update interval : 128.1 seconds
Leap status     : Normal

I did it from both Master node and one of the compute nodes and as can be seen the Leap status is Normal.

Important points regarding above commands:

a. Reference ID: It shows with which server we are synchronizing. If the IP add is means our system is not synching with any external source but local mode is enabled.

4. To get the information about the current time sources that chronyd is accessing

[root@node-01 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
^* hadoop-master.hpc.cluster     3   8   377   113   +113us[ +116us] +/-   22ms

5. Other usefull info can be also taken with following command:

[root@hadoop-master ~]# chronyc sourcestats
210 Number of sources = 4
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
x.fu110.de                 21  10   42m     +0.062      0.039  -1451us    28us
pyxis.my-rz.de             21  12   41m     +0.036      0.085   -840us    64us
gromit.nocabal.de          19  12   41m     -0.011      0.030   +726us    25us
clint.blazing.de           12   7   37m     -0.253      0.161   +839us    82us

%d bloggers like this: