Know Your Recovery Objectives Before Picking a Tool

RTO and RPO are not bureaucratic paperwork - they determine which backup architecture you actually need. A 4-hour RTO with a 1-hour RPO means you need local snapshots plus fast network access to recent backups. A 24-hour RTO with a 24-hour RPO means a nightly rsync job offsite might be enough.

Write these numbers down before touching a single cron job:

- RPO: How much data can you lose? 15 minutes? 1 hour? 1 day? - RTO: How fast must the system be back? Minutes (requires hot standby), hours (requires local backup), or days (cold archive is fine)? - Retention: How far back do you need to go? 30 days covers most ransomware detection windows. 1 year covers compliance for most regulated industries.

The mistake we see most often is sysadmins choosing a backup tool based on popularity rather than their own RTO/RPO. A team with a 15-minute RPO running nightly rsync is flying blind for 23 hours and 45 minutes at any given moment.

# Quick worksheet - paste into your runbook
echo "RPO target: ___ minutes/hours"
echo "RTO target: ___ hours"
echo "Retention: ___ days"
echo "Data size: ___ GB, change rate: ___ GB/day"
echo "Offsite required: yes/no"
echo "Encryption required: yes/no"

rsync: Still the Foundation for Most Setups

rsync 3.2.7+ is installed on virtually every Linux server and handles the majority of production backup cases well. It is not glamorous, but it is auditable, scriptable, and has zero vendor lock-in.

The basic invocation most people start with is too simple:

``` rsync -av /data/ backup@remote:/backups/data/ ```

This copies everything every time if you are not using `--link-dest`, which means it offers no true incrementals - it just overwrites the previous copy. For real incrementals with full snapshots at each point in time, use `--link-dest` to hardlink unchanged files:

The command below creates a dated snapshot directory. Unchanged files are hardlinked from the previous snapshot, so each snapshot looks complete but only new or modified blocks consume additional space.

On our test server with a 400GB data directory changing at roughly 8GB/day, each incremental snapshot consumes between 8-12GB instead of 400GB. Thirty daily snapshots occupy about 640GB total instead of 12TB.

One thing rsync does not handle well: open databases. Never rsync a live PostgreSQL data directory without first dumping it or using pg_basebackup. Rsync will produce a physically inconsistent copy that looks complete but will not restore.

For PostgreSQL, run pg_dumpall or use pg_basebackup into a staging directory first, then rsync the staging directory:

``` pg_basebackup -h localhost -U replicator -D /tmp/pg_staging -Ft -z -P rsync -a --delete /tmp/pg_staging/ backup@remote:/backups/postgres/ ```

rsync over SSH is the transport we recommend for most setups. Use key-based auth with a dedicated backup user that has read-only access to source data where possible. Do not use root-to-root rsync over SSH in production.

#!/bin/bash
# Incremental rsync snapshot with hardlinks
DATE=$(date +%Y-%m-%d_%H%M)
DEST="backup@remote:/backups/snapshots"
LATEST="$DEST/latest"
SNAPSHOT="$DEST/$DATE"

rsync -a --delete \
  --link-dest="$LATEST" \
  --exclude='*.tmp' \
  --exclude='/proc/' \
  --exclude='/sys/' \
  --log-file=/var/log/rsync-backup.log \
  /data/ \
  "$SNAPSHOT"

# Update the 'latest' symlink
ssh backup@remote "ln -snf $SNAPSHOT $LATEST"

restic: Deduplication, Encryption, and Multiple Backends

restic 0.17.x is the tool we reach for when rsync hardlink trees get unwieldy or when you need encrypted backups to an S3-compatible backend without writing your own encryption wrapper.

restic uses content-addressed chunking (CDC) for deduplication. Two files that share a 512KB block in the middle will store that block once regardless of when they were backed up. In practice on our test server, a 2TB media archive that grew by 50GB over 90 days had a restic repository size of about 1.1TB - significantly smaller than 90 copies of a 2TB directory.

Initialize a local repository and a remote S3 repository:

restic's `forget` command with `--prune` is where most people trip up. Running `forget` marks snapshots for deletion but `--prune` is required to actually reclaim the space. Without it, the repository grows indefinitely.

For checking repository integrity, run `restic check` weekly and `restic check --read-data-subset=10%` monthly. Reading 100% of data on large repositories is slow - on a 2TB repo over a 1Gbps link, `--read-data-subset=10%` takes about 22 minutes versus 3.5 hours for a full read.

restic supports backing up from stdin, which makes it clean for database dumps without a staging directory:

``` pg_dumpall -U postgres | restic backup --stdin --stdin-filename postgres-full.sql ```

The repository password must be stored somewhere accessible to the backup script but not world-readable. We use a file at `/etc/restic/password` with mode 0400 owned by root, referenced via `--password-file`.

# Initialize repositories
export RESTIC_PASSWORD_FILE=/etc/restic/password

# Local repo
restic init --repo /mnt/backup-disk/restic-repo

# S3 repo (Backblaze B2, Wasabi, MinIO, etc.)
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
restic init --repo s3:https://s3.us-east-1.amazonaws.com/mybucket/restic

# Run backup
restic backup \
  --repo s3:https://s3.us-east-1.amazonaws.com/mybucket/restic \
  --exclude='/data/tmp' \
  --tag production \
  /data

# Retention policy: keep 7 daily, 4 weekly, 12 monthly
restic forget \
  --repo s3:https://s3.us-east-1.amazonaws.com/mybucket/restic \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 12 \
  --prune
// advertisement

BorgBackup: Long Retention with Compression

BorgBackup 1.4.x sits in a different niche from restic. Where restic excels at multi-backend flexibility, Borg excels at compression ratios and fine-grained archive management for long-term retention. On our test server, Borg with `lz4` compression on log archives reduced repository size by 68% compared to uncompressed rsync snapshots. With `zstd,3` on text-heavy data, we saw 74% reduction.

Borg stores archives as named collections inside a repository. Each archive can be mounted as a FUSE filesystem for granular file-level restore without extracting the full archive:

``` borg mount /mnt/borg-repo::2026-08-20_nightly /mnt/restore-point ls /mnt/restore-point/data/configs/ umount /mnt/restore-point ```

This is the restore workflow that saves the most time in practice - mounting and browsing to find the exact file version you need, rather than extracting a multi-GB archive to find one config file.

BorgBackup's `borg check --verify-data` is more thorough than restic's equivalent on slow connections because Borg repositories are local-first by design. For remote Borg repositories, use borg serve over SSH:

``` # On the backup server - restrict what the client can do command="borg serve --restrict-to-path /backups/borg-repos" ssh-rsa AAAA...key... backup-client ```

This limits the SSH key to running borg serve only on the specified path. A compromised client cannot delete the repository using this key.

For passphrase management in Borg, we store the passphrase in `BORG_PASSPHRASE` environment variable loaded from a file sourced at script start. Never hardcode the passphrase in the backup script itself - it will end up in `ps aux` output.

#!/bin/bash
export BORG_REPO=/mnt/borg-repo
export BORG_PASSPHRASE=$(cat /etc/borg/passphrase)

# Create archive with compression
borg create \
  --compression zstd,3 \
  --stats \
  --show-rc \
  --exclude '/data/cache' \
  --exclude '*.pyc' \
  ::$(date +%Y-%m-%d_%H%M) \
  /data /etc /home

# Prune: 7 daily, 4 weekly, 6 monthly, 2 yearly
borg prune \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 6 \
  --keep-yearly 2 \
  --stats

# Compact freed space (Borg 1.2+)
borg compact

ZFS Snapshots: The Fastest Local Recovery Path

If your server runs ZFS (OpenZFS 2.2+ on Linux), snapshots are the fastest path to local recovery by a significant margin. A ZFS snapshot of a 1TB pool is instantaneous - it creates a point-in-time reference to the current block state with zero data copying. Recovery is a `zfs rollback` or a file copy from the snapshot directory.

The `/data/.zfs/snapshot/` directory is accessible directly without mounting:

``` ls /data/.zfs/snapshot/ 2026-08-23_0200 2026-08-22_0200 2026-08-21_0200

cp /data/.zfs/snapshot/2026-08-22_0200/configs/nginx.conf /data/configs/nginx.conf ```

This is sub-second recovery for accidental file deletion or overwrite, which covers the majority of day-to-day recovery requests.

For automated snapshots, zfs-auto-snapshot works but we prefer a simple cron-driven script that gives us explicit control over naming and retention. Snapshot names matter for operator sanity - a name like `tank/data@hourly-2026-08-24_1400` is immediately parseable by a human at 3am.

Sending ZFS snapshots offsite uses `zfs send | zfs receive` over SSH, and it sends only changed blocks between snapshots:

``` # Initial full send zfs send tank/data@2026-08-01 | ssh backup@remote "zfs receive backup/data"

# Incremental send (only blocks changed between snapshots) zfs send -i tank/data@2026-08-23 tank/data@2026-08-24 | \ ssh backup@remote "zfs receive backup/data" ```

On our test server, a pool with an 8GB daily change rate sends an incremental in 4-6 minutes over a 1Gbps link. The same data via rsync takes 12-18 minutes due to rsync's per-file overhead on large numbers of small files.

ZFS compression (`zfs set compression=lz4`) on the receiving pool reduces storage by 30-50% on most text-heavy workloads with negligible CPU overhead on modern hardware.

#!/bin/bash
# Automated ZFS snapshot with retention
POOL="tank/data"
TIMESTAMP=$(date +%Y-%m-%d_%H%M)
SNAP_NAME="$POOL@auto-$TIMESTAMP"

# Create snapshot
zfs snapshot "$SNAP_NAME"

# Keep last 48 hourly snapshots, delete older ones
zfs list -t snapshot -o name -s creation -r "$POOL" | \
  grep '@auto-' | \
  head -n -48 | \
  xargs -r -n1 zfs destroy

Offsite Replication and the 3-2-1 Rule Applied

The 3-2-1 rule (3 copies, 2 media types, 1 offsite) is the minimum viable backup architecture for anything important. In 2026 terms: local ZFS snapshots (copy 1), local restic or Borg repository on a separate disk or NAS (copy 2), and restic or rclone to an S3-compatible object store (copy 3, offsite).

rclone 1.68+ handles the offsite sync leg cleanly. It supports Backblaze B2, Wasabi, AWS S3, Cloudflare R2, and a dozen others with a single config syntax:

``` rclone sync /mnt/borg-repo wasabi:mybucket/borg-repo \ --transfers 8 \ --checkers 16 \ --s3-chunk-size 64M \ --progress ```

For immutability against ransomware, enable Object Lock on your S3 bucket in COMPLIANCE mode with a 30-day retention. This means even the account owner cannot delete objects before the retention period expires. Backblaze B2 and Wasabi both support this. Configure it at the bucket level, not per object, so every uploaded file inherits it automatically.

Bandwidth and cost matter for offsite replication. Wasabi has no egress fees, which matters when you need to pull 2TB back during a disaster recovery. At $6.99/TB/month for storage, a 5TB offsite backup costs $35/month. Backblaze B2 is cheaper at $6/TB/month but charges egress beyond 3x storage. For DR scenarios where you might pull large amounts back, factor in egress cost.

For teams building automated offsite pipelines, platforms like taskbotshub.ai can orchestrate multi-step backup workflows including triggering backup jobs, verifying completion via exit codes, running restore tests, and alerting on failures - which is useful when the pipeline involves five separate tools across three environments and manual coordination becomes error-prone.

# rclone config for Wasabi
[wasabi]
type = s3
provider = Wasabi
access_key_id = YOUR_KEY
secret_access_key = YOUR_SECRET
region = us-east-1
endpoint = s3.wasabisys.com

# Sync with bandwidth limit during business hours
rclone sync /mnt/local-backups wasabi:mybucket/backups \
  --bwlimit "09:00,50M 18:00,off 00:00,500M" \
  --log-file /var/log/rclone-offsite.log \
  --log-level INFO
// advertisement

Testing Restores: The Only Metric That Matters

A backup that has never been restored is a guess. Run restore tests on a schedule, not just after incidents. The minimum viable restore test schedule is: file-level restore weekly, full volume restore monthly, full server restore quarterly.

Automate the weekly file-level test. Pick a random file from the most recent backup, restore it to `/tmp/restore-test/`, and compare checksums against the live file:

For monthly volume restores, spin up a VM or container, restore the backup into it, and verify the application starts and serves a known response. For PostgreSQL, this means running `pg_restore` and then `psql -c 'SELECT count(*) FROM critical_table;'`.

For quarterly full-server restores, actually restore to bare hardware or a fresh VM from scratch using your runbook. Time it. If it takes 6 hours and your RTO is 2 hours, you have a problem to fix before an incident reveals it.

Keep restore logs. When you have 12 months of restore test logs showing consistent success times, you can give leadership a real RTO number instead of an estimate. When a test fails, you fix the backup process before a real failure forces the issue.

Naming your backup repositories, scripts, and cron jobs with consistent, parseable names matters more than it sounds. When you have 15 servers each with 3 backup jobs, a naming scheme like `[hostname]-[data-type]-[destination]` in script filenames and log entries makes debugging at 2am much faster. The same principle applies when naming associated infrastructure projects - clean, unambiguous names reduce cognitive load. nicename.me is a tool we have used when brainstorming short, clear names for infrastructure projects and associated domains when spinning up backup-specific services.

#!/bin/bash
# Weekly automated restore test
REPO=/mnt/restic-repo
PASS_FILE=/etc/restic/password
TEST_DIR=/tmp/restore-test-$(date +%Y%m%d)

mkdir -p "$TEST_DIR"

# Find a random file from the latest snapshot
RANDOM_FILE=$(restic ls --repo "$REPO" --password-file "$PASS_FILE" latest \
  | shuf -n 1 | sed 's|^/||')

# Restore it
restic restore --repo "$REPO" --password-file "$PASS_FILE" \
  --target "$TEST_DIR" \
  --include "$RANDOM_FILE" \
  latest

# Compare checksum
LIVE_SUM=$(sha256sum "/$RANDOM_FILE" | awk '{print $1}')
RESTORED_SUM=$(sha256sum "$TEST_DIR/$RANDOM_FILE" | awk '{print $1}')

if [ "$LIVE_SUM" = "$RESTORED_SUM" ]; then
  echo "PASS: restore test succeeded for $RANDOM_FILE"
else
  echo "FAIL: checksum mismatch for $RANDOM_FILE" >&2
  exit 1
fi

rm -rf "$TEST_DIR"

Monitoring Backup Jobs: Alert on Silence

Most backup monitoring alerts on failure. The problem is that a silently skipped job produces no failure - it just never runs. Alert on absence of success, not only on error.

The cleanest pattern for this is a heartbeat monitor. After a successful backup run, your script sends an HTTP GET to a monitoring service (healthchecks.io, Cronitor, or a self-hosted equivalent). If the ping does not arrive within the expected window, the monitor alerts. No ping means no backup, and you get the same alert whether the job crashed, the cron daemon died, or the server rebooted and the job was never rescheduled.

Implement it by appending a single curl call to your backup script:

``` # At end of backup script, after exit code check curl -fsS --retry 3 https://hc-ping.com/your-uuid-here > /dev/null ```

For failed jobs, send an alert with context - include the hostname, job name, last successful run time, and the tail of the log file. A bare 'backup failed' alert that requires SSH to diagnose wastes time during incidents.

Log every backup job to a structured format (JSON preferred) so you can query history with jq or ship to a log aggregator. Log these fields at minimum: timestamp, hostname, backup type, source path, destination, exit code, duration seconds, bytes transferred, and any error message.

Also monitor backup destination capacity. A restic or Borg repository that runs out of disk space fails silently on some versions - the job exits non-zero but if you are not checking that exit code in your wrapper script, the heartbeat still fires. Check exit codes explicitly with `set -e` or explicit `|| exit 1` chains.

#!/bin/bash
set -euo pipefail

HC_URL="https://hc-ping.com/your-uuid"
LOG=/var/log/backup/$(hostname)-$(date +%Y%m%d).log

# Signal start
curl -fsS "$HC_URL/start" > /dev/null

{
  echo "{\"ts\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"host\":\"$(hostname)\",\"status\":\"start\"}"
  
  # Your backup commands here
  restic backup --repo /mnt/restic-repo --password-file /etc/restic/password /data
  
  echo "{\"ts\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"host\":\"$(hostname)\",\"status\":\"success\"}"
  
  # Signal success
  curl -fsS "$HC_URL" > /dev/null

} 2>&1 | tee -a "$LOG"