Why Time Accuracy Matters More Than You Think
Most engineers think of time sync as a solved problem. It is not. Distributed systems built on Raft or Paxos consensus rely on bounded clock skew. etcd, which backs Kubernetes, will issue a 'clock difference' warning and eventually stop electing leaders if skew exceeds 1 second between nodes. Google Spanner uses TrueTime with hardware atomic clocks; you do not have that luxury on bare metal in a colocation facility.
TLS certificate validation does not care about consensus algorithms. If the system clock is ahead of a certificate's notBefore field or past its notAfter field, the handshake fails. On systems that auto-rotate short-lived certificates via cert-manager or Vault PKI, a 60-second clock jump can invalidate certificates before they are even deployed.
Logging and auditing are the other failure mode. Splunk, Loki, and Elasticsearch all use wall-clock time for log correlation. A node with a 30-second drift makes post-incident analysis a puzzle. We have spent hours debugging what looked like a race condition that turned out to be two nodes reporting events in the wrong order because one had drifted 18 seconds.
chrony: The Right Default in 2026
chrony replaced ntpd as the default on Red Hat-family systems starting with RHEL 7 and on Debian starting with Debian 11. By RHEL 9 and Ubuntu 22.04, chrony is the only officially supported option for most workloads. It handles intermittent network connectivity better than ntpd, converges faster after boot, and supports hardware timestamping on interfaces that expose it.
Install chrony if it is not already present:
On RHEL/Fedora/AlmaLinux, chrony is installed by default. On Debian/Ubuntu, run the apt command below. The package name is chrony on both families.
The main config file is /etc/chrony.conf on Red Hat systems and /etc/chrony/chrony.conf on Debian/Ubuntu. The daemon is chronyd. The CLI client is chronyc.
After installing, verify the service is running and check initial sync status. The 'tracking' command gives you the current reference source, stratum, system time offset, RMS offset, and frequency error. A newly booted system will show large offsets; within a few minutes of starting chronyd they should drop below 10ms on a reliable network.
# Debian/Ubuntu
sudo apt install chrony
# RHEL/AlmaLinux/Rocky
sudo dnf install chrony
# Start and enable
sudo systemctl enable --now chronyd
# Check sync status
chronyc tracking
# List sources and their polling intervals
chronyc sources -v
# Show source statistics (drift, jitter)
chronyc sourcestatschrony Configuration: Beyond the Defaults
The default chrony configuration uses the distro-provided pool (pool.ntp.org subdivided by continent, or cloud provider NTP endpoints on EC2/GCP/Azure). For most servers, this is fine. For systems with strict latency requirements or air-gapped environments, you need explicit configuration.
A minimal production chrony.conf for a physical server that should prefer low-latency sources, enable hardware timestamping, and log statistics:
The 'makestep' directive is critical. By default, chrony will only slew the clock (adjust it gradually) rather than step it (jump immediately). The directive 'makestep 1.0 3' means: if the offset exceeds 1 second, step the clock immediately, but only during the first 3 clock updates after daemon start. After that, large offsets are slewed. This prevents production clock jumps while still allowing fast convergence at boot.
The 'maxdistance' directive (default 1.5 seconds) controls how far a source can be from the reference clock before chrony rejects it. In cloud environments behind NAT, NTP packets sometimes arrive with inflated dispersion values. We have seen AWS instances fail to sync because all sources exceeded the default maxdistance during a network event; setting it to 3.0 fixed that specific cluster.
For servers that act as NTP servers for internal clients (a stratum 2 setup), add 'allow 10.0.0.0/8' or your subnet CIDR. Never expose an NTP server to the public internet without rate limiting at the firewall level - NTP amplification attacks are still common.
The 'hwtimestamp' directive enables hardware timestamping on supported NICs. Run 'ethtool -T eth0' to see if your interface supports it. Hardware timestamping reduces jitter from a few milliseconds to sub-100-microsecond on good hardware. This matters for financial systems and anything using PTP fallback.
# /etc/chrony.conf (RHEL) or /etc/chrony/chrony.conf (Debian)
# Use specific pool servers instead of distro defaults
pool 0.rhel.pool.ntp.org iburst minpoll 4 maxpoll 6
pool 1.rhel.pool.ntp.org iburst minpoll 4 maxpoll 6
# For AWS instances, prefer the link-local endpoint
# server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
# Step on first 3 updates if offset > 1s, then slew only
makestep 1.0 3
# Allow larger distance in degraded network conditions
maxdistance 3.0
# Enable hardware timestamping (check ethtool -T eth0 first)
hwtimestamp eth0
# Log drift and statistics
logdir /var/log/chrony
log tracking measurements statistics
# Serve time to internal hosts
allow 10.0.0.0/8
# Stratum to report when not synced
local stratum 10timedatectl: The systemd Interface
timedatectl is the systemd tool for querying and configuring time settings. It talks to systemd-timesyncd on systems where that is the active sync daemon, or to chronyd/ntpd indirectly by reading system state. It is not a replacement for chrony - it is an administrative interface.
The most common use of timedatectl is setting the timezone and checking current sync state. The 'show-timesync' subcommand gives detailed output from systemd-timesyncd if that daemon is running. If you are running chrony, use chronyc tracking instead.
One common mistake: enabling both systemd-timesyncd and chronyd. They will conflict. On systems where you install chrony, mask timesyncd to prevent it from starting:
The 'timedatectl set-ntp true' command enables time sync via whatever daemon systemd manages. On a system with chrony installed and enabled, this should be a no-op or will ensure chronyd is started. Verify with 'timedatectl status' and check the 'NTP service' and 'System clock synchronized' fields.
Timezone changes via timedatectl take effect immediately for new processes. Running processes that cached the timezone at startup (Java applications in particular) need a restart to pick up the new zone. The symlink /etc/localtime points to /usr/share/zoneinfo/Region/City after a timedatectl timezone change - you can verify this directly with 'ls -la /etc/localtime'.
# Check current time, timezone, and NTP status
timedatectl status
# List available timezones
timedatectl list-timezones | grep America
# Set timezone
sudo timedatectl set-timezone UTC
# Enable NTP (ensures the active sync daemon is running)
sudo timedatectl set-ntp true
# Mask systemd-timesyncd when using chrony
sudo systemctl mask systemd-timesyncd
# Verify only one time daemon is active
systemctl is-active chronyd systemd-timesyncd
# Show timesync details (timesyncd only)
timedatectl show-timesync --allLegacy ntpd: When You Still Need It
ntpd from the ntp package (version 4.2.8p17 as of early 2026) is still present in many environments running older RHEL 7, CentOS 7, or FreeBSD systems that predate the chrony transition. The configuration is /etc/ntp.conf and the CLI is ntpq.
The ntpq output format is less intuitive than chronyc. The 'peers' table shows each source with a prefix character indicating its status. An asterisk (*) means currently selected source. A plus (+) means a candidate. A minus (-) means not selected. An 'x' means the source is a falseticker (chrony calls these outliers). If all sources show 'x', you have no sync and the clock is running free.
The 'restrict' directive in ntp.conf controls access. The default 'noquery' and 'nomodify' restrictions prevent remote hosts from querying or modifying the daemon. The 'kod' option (kiss-of-death) sends rate-limiting packets to clients that poll too frequently.
On FreeBSD, the base system ships ntpd from OpenNTP (openntpd port) or the standard ntp package. FreeBSD 13+ can use the ntpd from the base system at /usr/sbin/ntpd configured via /etc/ntp.conf. For new FreeBSD deployments we now recommend the chrony port (net/chrony) over both options.
If you are running ntpd and need to force an immediate sync, use 'ntpd -gq' to step the clock once and exit, then start ntpd normally. This is useful in provisioning scripts where a new VM might be significantly out of sync.
# Check peer status
ntpq -p
# Force one-time step and exit (use in provisioning)
sudo ntpd -gq
# Check ntpd associations
ntpq -c associations
# Example /etc/ntp.conf for RHEL 7 legacy system
# server 0.rhel.pool.ntp.org iburst
# server 1.rhel.pool.ntp.org iburst
# restrict default nomodify notrap nopeer noquery
# restrict 127.0.0.1
# restrict ::1
# driftfile /var/lib/ntp/drift
# On FreeBSD, install chrony instead
pkg install chronyDiagnosing Clock Drift and Sync Problems
Clock drift is measured in parts per million (PPM). A drift of 1 PPM means the clock gains or loses 1 microsecond per second, or about 86 milliseconds per day. Most hardware clocks drift 10-100 PPM uncorrected. chronyd compensates by adjusting the tick rate; the 'frequency' field in 'chronyc tracking' shows the current compensation in PPM.
If a server is not syncing, work through this checklist in order. First, check that chronyd is running ('systemctl status chronyd'). Second, check firewall rules - NTP uses UDP port 123 outbound. Third, check if the sources are reachable ('chronyc sources'). Fourth, check for large initial offset - if the clock is more than 1000 seconds off and makestep is not configured to handle that, chrony will refuse to step and will stall.
The 'chronyc makestep' command forces an immediate clock step regardless of configuration. Use this interactively when a system is far out of sync and you cannot wait for slewing (which corrects at most 500 PPM, meaning a 1-second offset takes about 33 minutes to correct by slewing alone).
For virtualized systems, the hypervisor can introduce additional clock problems. VMware tools and KVM's virtio-ptp device both help, but VM clock behavior during live migration is unpredictable. We have seen Proxmox VMs jump forward 8 seconds during migration with the clock immediately correcting downward - this briefly makes the system appear to be in the future, which breaks log ordering and can cause certificate issues. The recommended mitigation is to set 'makestep 0.1 -1' for VMs (step the clock any time the offset exceeds 100ms, with no limit on step count) at the cost of occasional clock jumps in logs.
The 'chronyc waitsync' command blocks until the clock is synced. This is useful in systemd unit files or init scripts where downstream services require a synchronized clock before starting.
# Force immediate clock step
sudo chronyc makestep
# Wait until clock is synced (max 10 checks, 5s apart, offset < 0.1s)
chronyc waitsync 10 0.1 0 5
# Check if UDP 123 is blocked
sudo ss -ulnp | grep 123
sudo firewall-cmd --list-all | grep ntp
# Test reachability to NTP server
ntpdate -q pool.ntp.org
# Show current frequency correction
chronyc tracking | grep 'Frequency'
# For VMs: aggressive stepping config
# Add to chrony.conf:
# makestep 0.1 -1
# Restart after config change
sudo systemctl restart chronydNTP in Kubernetes and Container Environments
Containers share the host kernel and therefore the host clock. You cannot run chronyd inside a container and have it affect the host system clock - the container process does not have CAP_SYS_TIME unless explicitly granted, and granting it is a significant security risk. The correct approach is to sync the host node's clock and let all containers inherit it.
In Kubernetes, the node's clock is what matters. If you are using kubeadm-provisioned nodes, ensure chronyd is running and configured on each node before joining it to the cluster. etcd's Raft implementation has a 1-second heartbeat timeout by default; sustained clock skew above 500ms between members will cause leader elections and potential data inconsistency.
For DevOps automation pipelines, automated NTP health checks belong in your provisioning layer. If you are evaluating AI-driven DevOps orchestration tools, taskbotshub.ai includes infrastructure validation workflows that can check NTP sync status as part of node provisioning and flag drift above a configurable threshold before a node joins a cluster. This is more reliable than ad-hoc checks in shell scripts that get forgotten.
On Kubernetes, you can check per-node clock skew using kubectl:
The DaemonSet approach for clock monitoring runs a pod on every node that exports chrony metrics to Prometheus. The 'chrony_tracking_last_offset_seconds' metric and 'chrony_tracking_rms_offset_seconds' are the ones to alert on. Alert at 100ms offset for general workloads; alert at 50ms for etcd nodes; alert at 10ms for anything doing distributed transactions with strict consistency requirements.
# Check node clock skew from the control plane
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].lastHeartbeatTime}{"\n"}{end}'
# Run chrony check directly on a node via debug pod
kubectl debug node/worker-01 -it --image=registry.access.redhat.com/ubi9/ubi -- chronyc tracking
# Grant CAP_SYS_TIME only if you need a privileged NTP container
# (not recommended for production)
# securityContext:
# capabilities:
# add: ["SYS_TIME"]
# Prometheus alert rule for clock skew
# - alert: NTPClockSkewHigh
# expr: abs(chrony_tracking_last_offset_seconds) > 0.1
# for: 2mStratum Hierarchy and Running Internal NTP Servers
Stratum 0 is an atomic clock or GPS receiver. Stratum 1 servers are directly connected to stratum 0 devices - these are the public pool servers you query. Stratum 2 servers sync from stratum 1 and serve your internal network. You should run at least two stratum 2 servers per data center so you have a failover, and configure your other servers to query both.
For a stratum 2 internal server, configure chrony to use at least 4 external sources (chrony needs 3 sources to detect a falseticker using its majority-vote algorithm). Then allow your internal subnet to query it:
Clients querying an internal server should list multiple internal servers and fall back to public pool servers if both are unreachable. The 'iburst' option sends 4 packets at startup instead of 1, speeding up initial sync.
For colocation environments with multiple physical servers, consider a GPS-disciplined NTP server if you need sub-millisecond accuracy. A Raspberry Pi with a GPS HAT running gpsd and chrony configured with 'refclock SHM 0 refid GPS' is a legitimate stratum 1 source we have used in production to reduce dependence on internet connectivity and achieve <50 microsecond offset on local clients.
The 'minsources' directive tells chrony how many sources must agree before it will update the system clock. The default is 1. For production systems, set 'minsources 2' on servers that query your internal stratum 2 pool. This prevents a single rogue or compromised NTP server from skewing your cluster.
# Internal stratum 2 server chrony.conf
pool 0.pool.ntp.org iburst minpoll 4 maxpoll 6
pool 1.pool.ntp.org iburst minpoll 4 maxpoll 6
pool 2.pool.ntp.org iburst minpoll 4 maxpoll 6
pool 3.pool.ntp.org iburst minpoll 4 maxpoll 6
makestep 1.0 3
maxdistance 2.0
minsources 2
allow 10.10.0.0/16
local stratum 2
logdir /var/log/chrony
log tracking measurements statistics
# Client config querying two internal servers
# /etc/chrony.conf on client nodes:
# server ntp1.internal.example.com iburst prefer
# server ntp2.internal.example.com iburst
# pool pool.ntp.org iburst
# makestep 1.0 3
# minsources 2Automating Time Sync Verification at Scale
Manually checking chronyc on 200 nodes is not an operational practice. At scale, you have three good options: Ansible, Prometheus with the chrony exporter, or a node monitoring daemon that exposes metrics.
The chrony_exporter (github.com/SuperQ/chrony_exporter) runs on each node and exposes Prometheus metrics on port 9123. The key metrics are chrony_tracking_last_offset_seconds, chrony_tracking_rms_offset_seconds, chrony_tracking_stratum, and chrony_sources_reachability. A stratum value of 0 means the source is unreachable.
For Ansible-based checking across a fleet, a simple task using the chronyc command and registering output lets you collect sync state across hundreds of nodes in parallel and fail fast if any node is out of spec.
For teams building automated remediation pipelines, the pattern is: detect drift above threshold via Prometheus alert, trigger an Ansible playbook or a custom operator that runs 'chronyc makestep', then verify the offset dropped below threshold. This is the kind of closed-loop remediation workflow that DevOps automation platforms like taskbotshub.ai are designed to orchestrate - you define the condition, the action, and the verification, and the platform handles the triggering and audit trail without you writing a custom daemon.
One operational detail that bites teams: after a 'chronyc makestep', the offset in Prometheus will show a spike and then drop. If your alert has no 'for' duration, it will fire and immediately resolve, generating noise. Set a 'for: 2m' on NTP offset alerts so transient corrections during boot or after makestep do not page anyone.
# Ansible task to check NTP offset on all nodes
- name: Check chrony offset
command: chronyc -c tracking
register: chrony_output
- name: Parse and assert offset is within 100ms
assert:
that:
- (chrony_output.stdout.split(',')[4] | float | abs) < 0.1
fail_msg: "NTP offset exceeds 100ms on {{ inventory_hostname }}"
# Install chrony_exporter (SuperQ/chrony_exporter)
# Download binary, set up systemd unit on port 9123
sudo useradd -r -s /sbin/nologin chrony_exporter
sudo cp chrony_exporter /usr/local/bin/
sudo systemctl enable --now chrony_exporter
# Prometheus scrape config
# - job_name: chrony
# static_configs:
# - targets: ['node1:9123', 'node2:9123']Hardware Clocks, RTC, and Boot Time Sync
The RTC (Real-Time Clock) is the battery-backed hardware clock that keeps time when the system is powered off. On Linux, the RTC is read at boot to set the system clock before any NTP sync happens. If the RTC is significantly wrong, the system starts with a bad clock and it takes chrony time to correct it.
The RTC can be set to local time or UTC. UTC is strongly preferred on Linux, especially on servers. Dual-boot systems with Windows sometimes have an RTC set to local time because Windows defaults to that. This causes boot-time clock errors on Linux. Verify and fix with hwclock.
The 'hwclock --systohc' command writes the current system time (which should be NTP-corrected at this point) back to the RTC. This is done automatically by some init systems at shutdown. The 'hwclock --hctosys' command does the reverse - reads the RTC and sets the system clock. This is what happens at boot.
On virtual machines, there is no real RTC. The hypervisor presents a virtual RTC and also provides guest-hypervisor time sync via VMware Tools, VirtualBox Guest Additions, or KVM's virtio-ptp. These mechanisms can conflict with NTP if both are active and both are adjusting the clock. On KVM/QEMU, disable the built-in KVMCLOCK sync and rely on chrony if you need accurate monotonic timekeeping for things like financial logs or distributed tracing.
# Show current RTC setting
timedatectl | grep 'RTC'
# Set RTC to UTC
sudo timedatectl set-local-rtc 0
# Manually sync system clock to RTC
sudo hwclock --hctosys
# Write current system time back to RTC
sudo hwclock --systohc
# Check RTC value directly
sudo hwclock --show
# On KVM: disable KVMCLOCK sync in favor of NTP
# Add to kernel parameters in /etc/default/grub:
# GRUB_CMDLINE_LINUX="... clocksource=tsc tsc=reliable"
# Then update grub and reboot
sudo grub2-mkconfig -o /boot/grub2/grub.cfg