How Linux Tracks Time: Two Layers You Must Know

Linux maintains two separate clocks. The hardware clock (RTC) lives on the motherboard and runs independently of the OS. The system clock is maintained by the kernel and is what `date`, cron, and every userspace application reads. These two clocks can disagree, and the direction of the disagreement matters.

By convention, Linux expects the RTC to be set to UTC. Windows historically defaulted to storing local time in the RTC, which caused clock skew on dual-boot machines. Modern Windows (since Windows 8) can be configured to use UTC via a registry key, but if you are managing bare-metal servers you will occasionally inherit a machine where the RTC is in localtime. `timedatectl` will tell you:

``` $ timedatectl show Timezone=America/New_York LocalRTC=no CanNTP=yes NTP=yes NTPSynchronized=yes TimeUSec=Thu 2026-08-20 14:32:11 UTC RTCTimeUSec=Thu 2026-08-20 14:32:11 UTC ```

If `LocalRTC=yes` appears on a server, fix it immediately with `timedatectl set-local-rtc 0`. Leaving the RTC in localtime with DST transitions active causes the system clock to jump by one hour twice a year, which is catastrophic for anything touching time-series data or distributed consensus.

The timezone itself is stored in `/etc/localtime`, which must be a symlink into `/usr/share/zoneinfo/`. The symlink target is the canonical source of truth for all POSIX-compliant timezone lookups.

timedatectl show
ls -la /etc/localtime
# Expected output:
# /etc/localtime -> /usr/share/zoneinfo/America/New_York

Listing and Setting Timezones with timedatectl

`timedatectl list-timezones` outputs all zones from the installed tzdata package. On a standard Ubuntu 24.04 server we counted 594 entries. Pipe through grep to find what you need.

Setting the timezone requires root and takes effect immediately - no reboot, no service restart:

```bash # Find your zone timedatectl list-timezones | grep -i london # Europe/London

# Set it sudo timedatectl set-timezone Europe/London

# Verify timedatectl ```

The `set-timezone` command does two things atomically: it updates `/etc/localtime` to point at the correct zoneinfo file, and it writes the zone name into `/etc/timezone` on Debian-based systems (or `/etc/sysconfig/clock` on older RHEL). Both files need to be consistent. If you manually symlink `/etc/localtime` without updating `/etc/timezone`, tools like `dpkg-reconfigure tzdata` and some Docker base images will silently use the wrong zone.

On RHEL/CentOS 7 and earlier, `timedatectl` is available but `chronyd` or `ntpd` must be separately installed and enabled. On RHEL 8+, `chronyd` ships as the default NTP client and integrates with `timedatectl` directly.

If you are managing multiple servers and want to audit timezone consistency across a fleet, a one-liner using SSH is faster than any GUI:

```bash for host in web01 web02 db01 cache01; do echo -n "$host: " ssh $host "timedatectl show --value -p Timezone" done ```

In our experience, timezone drift between nodes in a database cluster is almost always the result of someone setting the timezone manually on one node via the old `tzselect` + `cp` method instead of using `timedatectl`.

sudo timedatectl set-timezone Europe/London
timedatectl list-timezones | grep -i america/new
timedatectl show --value -p Timezone

The /etc/localtime Symlink: Manual Control Without timedatectl

On minimal container images and embedded systems, `timedatectl` is often absent because systemd is not running. In those cases, manage the symlink directly:

```bash # Remove the existing symlink or file rm /etc/localtime

# Create the correct symlink ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime

# Debian/Ubuntu: also update /etc/timezone echo 'Asia/Tokyo' > /etc/timezone ```

Verify the zoneinfo file exists before symlinking. On Alpine Linux, tzdata is not installed by default:

```bash # Alpine apk add tzdata cp /usr/share/zoneinfo/Asia/Tokyo /etc/localtime echo 'Asia/Tokyo' > /etc/timezone ```

Note that Alpine's recommended approach uses `cp` rather than `ln -sf`. This is intentional: Alpine often runs in read-only root filesystem configurations where a symlink into a package-managed path causes issues during package updates.

The `/usr/share/zoneinfo/` directory is provided by the `tzdata` package on Debian/Ubuntu and `tzdata` or `glibc-common` on RHEL. Keep this package updated. The IANA timezone database releases updates several times per year - in 2025 there were four releases (2025a through 2025d) covering DST law changes in Greenland, Chile, and Kazakhstan. Running stale tzdata on a server handling appointments or financial timestamps for those regions will produce incorrect offsets silently.

ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
echo 'Asia/Tokyo' > /etc/timezone
date
# Thu Aug 20 23:32:11 JST 2026
// advertisement

TZ Environment Variable: Process-Level Timezone Override

Every POSIX timezone lookup checks the `TZ` environment variable before consulting `/etc/localtime`. This gives you per-process timezone control without touching system files, which is the correct approach for:

- Running a cron job that must execute at a specific local time for a foreign timezone - Testing application timezone handling in CI/CD - Running multiple application instances serving different regional markets on the same host

```bash # Check what a specific timezone gives you right now TZ=America/Los_Angeles date # Thu Aug 20 07:32:11 PDT 2026

TZ=Australia/Sydney date # Fri Aug 21 00:32:11 AEST 2026

# Set it for a shell session export TZ=Europe/Berlin date

# Run a single command in a specific timezone TZ=UTC crontab -e ```

For persistent per-service timezone overrides on systemd systems, set `Environment=TZ=America/Chicago` in the unit file's `[Service]` section rather than modifying system-wide timezone. This survives system timezone changes and makes the intent explicit.

```ini [Service] Environment=TZ=America/Chicago ExecStart=/usr/bin/myapp ```

Reload systemd and restart the service:

```bash systemctl daemon-reload systemctl restart myapp ```

For containerized workloads, pass `-e TZ=America/Chicago` to `docker run` or set it in your `docker-compose.yml` under `environment:`. Mounting `/etc/localtime` from the host is a common pattern but couples container behavior to host configuration, which defeats the purpose of portable containers. The `TZ` environment variable is the cleaner solution.

# systemd unit override for per-service timezone
[Service]
Environment="TZ=America/Chicago"
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml

NTP Synchronization: timedatectl and chronyd

Setting the correct timezone is pointless if the system clock is wrong. `timedatectl` manages NTP enablement on systemd systems through `systemd-timesyncd` (the lightweight built-in client) or delegates to `chronyd`/`ntpd` if installed.

Check current NTP status:

```bash timedatectl timesync-status # Server: 91.189.91.157 (ntp.ubuntu.com) # Poll interval: 34min 8s (min: 32s; max 34min 8s) # Leap: normal # Version: 4 # Stratum: 2 # Reference: 194.80.204.184 # Precision: 1us (-20) # Root distance: 18.209ms (max: 5s) # Offset: -2.347ms # Delay: 28.501ms # Jitter: 3.879ms # Packet count: 7 ```

Enable or disable NTP sync:

```bash sudo timedatectl set-ntp true sudo timedatectl set-ntp false ```

On servers where `chronyd` is installed (RHEL 8+, modern Ubuntu with chrony package), `timedatectl set-ntp true` enables and starts `chronyd`, not `systemd-timesyncd`. You can verify:

```bash systemctl status chronyd chronyc tracking ```

`chronyc tracking` gives you offset, RMS offset, and frequency error - the three numbers that matter for high-precision time requirements. For most application servers, an offset under 10ms is acceptable. For distributed databases (Cassandra, CockroachDB), systems using TrueTime-style APIs, or Kerberos authentication, you need sub-millisecond accuracy, which requires PPS-disciplined sources or cloud provider time services like AWS Time Sync Service (169.254.169.123) or Google's time.google.com.

For AWS EC2 instances, AWS recommends using the instance's local time sync service over the standard NTP pool. Add this to `/etc/chrony.conf`:

``` server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4 ```

We tested this on an m6i.xlarge in us-east-1 and saw consistent offsets under 0.2ms versus 2-8ms with the default Ubuntu NTP pool configuration.

timedatectl timesync-status
chronyc tracking
chronyc sources -v

Container and Kubernetes Timezone Pitfalls

Containers inherit PID 1's timezone from the image, not the host. This is the source of most timezone confusion in containerized environments. Three patterns exist, with different tradeoffs:

**Pattern 1: TZ environment variable (recommended)** ```yaml # docker-compose.yml services: app: image: myapp:latest environment: - TZ=UTC ```

Works on any base image. No volume mounts required. Survives host timezone changes.

**Pattern 2: Bind mount /etc/localtime** ```yaml volumes: - /etc/localtime:/etc/localtime:ro ```

Couples container to host. Breaks if host timezone changes. Does not work on Windows or macOS Docker hosts where the host filesystem does not have `/etc/localtime`.

**Pattern 3: Bake tzdata into the image** ```dockerfile FROM debian:12-slim RUN apt-get update && apt-get install -y --no-install-recommends tzdata \ && ln -sf /usr/share/zoneinfo/UTC /etc/localtime \ && echo 'UTC' > /etc/timezone \ && rm -rf /var/lib/apt/lists/* ```

For Kubernetes, set timezone via environment variables in the pod spec:

```yaml spec: containers: - name: app image: myapp:latest env: - name: TZ value: "Europe/Amsterdam" ```

If you are running a DaemonSet that must match the host timezone dynamically, read the host's timezone in an init container:

```yaml initContainers: - name: tz-init image: busybox command: ['sh', '-c', 'cat /host-tz/timezone > /shared/TZ'] volumeMounts: - name: host-timezone mountPath: /host-tz readOnly: true - name: shared mountPath: /shared ```

If you are automating timezone configuration across Kubernetes clusters as part of a GitOps or CI/CD pipeline, tools like those at taskbotshub.ai can generate and apply ConfigMaps and environment variable patches across namespaces without manual per-deployment edits.

# Verify timezone inside a running container
docker exec -it mycontainer date
docker exec -it mycontainer cat /etc/timezone
docker exec -it mycontainer timedatectl 2>/dev/null || echo 'No systemd in container'
// advertisement

Scripting Timezone Changes for Fleet Management

When you need to standardize timezone across 50 or 500 servers, use Ansible rather than SSH loops. A role-based approach keeps changes idempotent and auditable.

```yaml # tasks/main.yml - name: Set system timezone community.general.timezone: name: "{{ system_timezone | default('UTC') }}" notify: Restart cron

- name: Ensure NTP is enabled via timedatectl command: timedatectl set-ntp true changed_when: false

- name: Verify timezone is set correctly command: timedatectl show --value -p Timezone register: current_tz changed_when: false

- name: Assert timezone is correct assert: that: - current_tz.stdout == system_timezone fail_msg: "Timezone mismatch: expected {{ system_timezone }}, got {{ current_tz.stdout }}" ```

The `community.general.timezone` module handles the `timedatectl` call, `/etc/localtime` symlink, and `/etc/timezone` file on Debian-based systems in one task. It is idempotent by default.

For bare-metal provisioning where Ansible is not yet available, a POSIX shell script works on any Linux system:

```bash #!/bin/sh set -eu

TARGET_TZ="${1:-UTC}" ZONEINFO="/usr/share/zoneinfo/${TARGET_TZ}"

if [ ! -f "${ZONEINFO}" ]; then echo "Error: ${ZONEINFO} not found. Install tzdata." >&2 exit 1 fi

if command -v timedatectl > /dev/null 2>&1; then timedatectl set-timezone "${TARGET_TZ}" else ln -sf "${ZONEINFO}" /etc/localtime printf '%s\n' "${TARGET_TZ}" > /etc/timezone fi

echo "Timezone set to: $(cat /etc/timezone 2>/dev/null || readlink /etc/localtime)" ```

Save this as `set-timezone.sh`, drop it in your kickstart or cloud-init `runcmd` block, and every provisioned machine gets a consistent timezone from day one. For cloud-init specifically:

```yaml # cloud-init user-data #cloud-config timezone: UTC ntp: enabled: true ntp_client: chrony ```

Cloud-init's `timezone` key sets `/etc/localtime` and `/etc/timezone` before any `runcmd` runs, so your application configuration scripts see the correct timezone from the start.

#!/bin/sh
set -eu
TARGET_TZ="${1:-UTC}"
ZONEINFO="/usr/share/zoneinfo/${TARGET_TZ}"
[ -f "${ZONEINFO}" ] || { echo "tzdata not installed"; exit 1; }
command -v timedatectl > /dev/null 2>&1 \
  && timedatectl set-timezone "${TARGET_TZ}" \
  || { ln -sf "${ZONEINFO}" /etc/localtime; echo "${TARGET_TZ}" > /etc/timezone; }
echo "Set: ${TARGET_TZ}"

Diagnosing Timezone Problems After the Fact

When timezone issues appear in production, the first question is always: when did the clock change? `journald` timestamps are stored in UTC internally, which means you can re-examine them in any timezone after the fact:

```bash # View logs in a specific timezone TZ=America/New_York journalctl --since '2026-08-20 09:00:00' --until '2026-08-20 10:00:00'

# Show when the timezone was last changed journalctl -u systemd-timedated --since '30 days ago' | grep -i timezone

# Check RTC vs system clock timedatectl | grep -E 'Local time|Universal time|RTC time' ```

For application logs that embed local time strings, you need to know what timezone was active when the log was written. Check `/etc/localtime` modification time as a rough indicator:

```bash stat /etc/localtime ls -la /etc/localtime ```

If the symlink target does not match what you expect and the modification time is recent, someone changed it. Cross-reference with `last` and `ausearch` if auditd is running:

```bash ausearch -f /etc/localtime ausearch -f /etc/timezone ```

For databases, timezone mismatches between the application server and the database server cause timestamp storage bugs that are extremely difficult to retroactively fix. MySQL/MariaDB stores its own timezone setting independently of the OS:

```sql SELECT @@global.time_zone, @@session.time_zone; -- 'SYSTEM' means it uses the OS timezone -- Set explicitly to avoid surprises: SET GLOBAL time_zone = '+00:00'; ```

PostgreSQL uses the OS timezone by default but can be overridden per-database or per-session:

```sql SHOW timezone; SET timezone = 'UTC'; ALTER DATABASE myapp SET timezone = 'UTC'; ```

Our recommendation: always configure databases to store and return timestamps in UTC, and handle timezone conversion in the application layer. This decouples your data from server configuration and makes migrations across regions or cloud providers far less painful.

# Full diagnostic sequence
timedatectl
ls -la /etc/localtime
cat /etc/timezone 2>/dev/null
chronyc tracking 2>/dev/null || systemctl status systemd-timesyncd
journalctl -u systemd-timedated --since '7 days ago' | grep -i 'timezone\|time zone'

Timezone in Cron Jobs and Scheduled Tasks

System cron (`/etc/crontab`, `/etc/cron.d/`) runs in the system timezone. User crontabs via `crontab -e` also inherit the system timezone unless the user's environment sets `TZ`. This is a common source of confusion when a server is in UTC but the business logic expects jobs to run at 08:00 America/New_York.

The correct fix is to set TZ at the top of the crontab:

``` TZ=America/New_York 0 8 * * * /usr/local/bin/morning-report.sh ```

Or per-job:

``` 0 8 * * * TZ=America/New_York /usr/local/bin/morning-report.sh ```

For systemd timers, timezone-aware scheduling uses `OnCalendar` with explicit timezone:

```ini [Timer] OnCalendar=Mon..Fri 08:00 America/New_York Persistent=true ```

This syntax was introduced in systemd 248. On older systems you have to calculate UTC offset manually, which breaks on DST transitions.

Check your systemd version before relying on timezone-aware `OnCalendar`:

```bash systemctl --version | head -1 # systemd 252 (252.22-1~deb12u1) ```

Debian 12 ships systemd 252, Ubuntu 24.04 ships systemd 255, RHEL 9 ships systemd 252. All support timezone-aware timers. RHEL 8 ships systemd 239, which does not - on RHEL 8 you need cron or must calculate UTC offsets statically.

For `anacron`, timezone matters less because anacron runs jobs based on elapsed days, not clock time. But the log timestamps will still reflect the system timezone, which affects monitoring integrations that parse log output.

# /etc/cron.d/morning-report
TZ=America/New_York
0 8 * * 1-5 deploy /usr/local/bin/morning-report.sh >> /var/log/morning-report.log 2>&1

# Equivalent systemd timer - /etc/systemd/system/morning-report.timer
[Unit]
Description=Morning Report Timer

[Timer]
OnCalendar=Mon..Fri 08:00 America/New_York
Persistent=true

[Install]
WantedBy=timers.target
// advertisement