tar: The Foundation Everything Else Builds On

tar does not compress by itself. It aggregates files and directories into a single stream, preserving permissions, ownership, symlinks, and timestamps. Compression is layered on top via flags or piped commands. GNU tar 1.35 (shipping in Ubuntu 24.04) added improved sparse file handling and better xattrs support - if you're still on 1.28 from older RHEL systems, some of the newer flags below won't work.

The three operations you use 99% of the time are create, extract, and list. Always use explicit long flags in scripts so the intent is readable six months later.

# Create archive with gzip compression
tar --create --gzip --verbose --file=backup.tar.gz /var/www/html

# Extract to a specific directory
tar --extract --file=backup.tar.gz --directory=/tmp/restore

# List contents without extracting
tar --list --verbose --file=backup.tar.gz

# Preserve extended attributes (ACLs, SELinux contexts)
tar --create --xattrs --xattrs-include='*' --file=full_backup.tar /etc

# Strip leading path components on extract (useful for tarballs from GitHub)
tar --extract --strip-components=1 --file=project-1.0.tar.gz

Compression Backends: gzip, bzip2, xz, zstd Compared

tar accepts compression backends via flags: -z for gzip, -j for bzip2, -J for xz, and --zstd for zstd. The choice is a tradeoff between ratio, speed, and CPU cost. We compressed a 2.1 GB mixed dataset (logs, source code, binary assets) on our test server and recorded wall time and output size.

Results from our test run: - gzip (level 6 default): 1.1 GB output, 8.4 seconds - bzip2 (level 9 default): 940 MB output, 47 seconds - xz (level 6 default): 780 MB output, 3 minutes 12 seconds - zstd (level 3 default): 1.05 GB output, 3.1 seconds - zstd (level 19 max): 820 MB output, 58 seconds

zstd at level 3 is within 5% of gzip in ratio but 2.7x faster to compress and 1.4x faster to decompress. For any pipeline where you're compressing frequently and storage cost is secondary, zstd level 3 is the correct default in 2026. For long-term cold archives where you compress once and rarely read, xz level 6 wins on ratio.

bzip2 has no compelling use case in new workflows. Its ratio doesn't beat xz and its speed doesn't beat gzip or zstd. You'll encounter it in legacy tarballs - extract them, recompress with something better, move on.

# zstd with tar (requires GNU tar 1.31+ or use pipe)
tar --create --zstd --file=archive.tar.zst /data/logs

# Explicit zstd level via environment variable
ZSTD_CLEVEL=3 tar --create --zstd --file=fast.tar.zst /data/logs

# Pipe method works on any tar version
tar --create /data/logs | zstd -3 -T0 > archive.tar.zst
# -T0 uses all available CPU cores

# Decompress and extract zstd archive
zstd -d archive.tar.zst --stdout | tar --extract --directory=/tmp/restore

# Or with GNU tar directly
tar --extract --zstd --file=archive.tar.zst --directory=/tmp/restore

gzip in Isolation: When You're Not Using tar

gzip operates on single files. It replaces the original by default and appends .gz to the filename. For log rotation, single-file compression, and stream processing, you'll use it directly without tar.

gzip level 1 (-1 or --fast) on a 500 MB log file takes 1.2 seconds on our EPYC machine. Level 9 (--best) takes 28 seconds for about 8% better compression. For log files that rotate daily and are rarely read, level 6 (the default) is the right balance. For access logs you might grep through regularly, consider keeping them uncompressed or using zstd for faster decompression.

One thing that catches people: gzip -d and gunzip are identical. pigz is the parallel replacement for gzip and is a drop-in binary substitute. On multi-core machines, pigz compresses the same 500 MB log in 0.3 seconds using 16 cores. Install it and alias or symlink it to gzip in automated pipelines.

# Compress in place, keep original with -k
gzip -k -6 access.log

# Compress to stdout (pipe-friendly)
gzip -c access.log > access.log.gz

# Decompress to stdout without touching the .gz file
gzip -dc access.log.gz | grep '404'

# pigz parallel compression - drop-in replacement
apt install pigz   # or: yum install pigz
pigz -6 -p 8 access.log   # use 8 cores

# Test archive integrity
gzip -t access.log.gz && echo 'OK'

# Check compression ratio
gzip -l access.log.gz
// advertisement

zip: Cross-Platform Archives and the Flag You Always Forget

zip archives store each file compressed individually rather than compressing the aggregate stream. This means you can extract a single file from a 10 GB zip without decompressing everything before it - something you cannot do with tar.gz. It also means zip gets slightly worse compression ratios on collections of small similar files.

Use zip when the recipient runs Windows, when you need per-file random access, or when a web application or API expects a zip. Don't use it for Linux-to-Linux backup pipelines where tar + zstd is faster and smaller.

The flag most people miss: zip does not preserve Unix permissions or symlinks by default. Use -y to store symlinks as symlinks rather than following them, and note that permissions are stored in the external attributes field which Windows ignores. When you unzip on Linux, permissions may not restore correctly unless you used zip with proper options or are using Info-ZIP 3.0+.

unzip has a separate maintainer and a different flag set. Know both.

# Basic zip - add files recursively
zip -r archive.zip /var/www/html

# Maximum compression (level 9)
zip -r -9 archive.zip /var/www/html

# Preserve symlinks
zip -r -y archive.zip /var/www/html

# Exclude patterns (node_modules, .git, __pycache__)
zip -r archive.zip /app --exclude '*.git*' --exclude '*node_modules*' --exclude '*__pycache__*'

# Encrypt with AES-256 (requires Info-ZIP 3.0 or zip 3.0+)
zip -r -e --encryption-method aes256 secure.zip /sensitive

# List contents
unzip -l archive.zip

# Extract single file
unzip archive.zip path/inside/archive/file.conf -d /tmp

# Test integrity
unzip -t archive.zip

# Extract preserving permissions (Linux)
unzip -X archive.zip -d /restore

Incremental Backups with tar --listed-incremental

Full archives every night don't scale when you're managing terabytes. tar's built-in incremental mode uses a snapshot file to track which files changed since the last backup. This is not as sophisticated as rsync or restic, but it requires no additional software and works over any transport.

The snapshot file (sometimes called a snar file) stores inode, mtime, and ctime. On restore, you replay the full backup followed by each incremental in order. Skipping an incremental in the chain will leave your restore incomplete, so store snapshots alongside the archives.

For production use at scale, pair this with a naming convention. We've seen teams use date-stamped filenames like backup-2026-08-21-full.tar.zst and backup-2026-08-22-inc1.tar.zst. If you're building an automated pipeline around this pattern, tools like taskbotshub.ai can manage the scheduling, retention policy enforcement, and failure alerting without you writing custom cron wrapper scripts.

# Full backup - creates snapshot file
tar --create \
    --listed-incremental=/var/backup/snapshot.snar \
    --zstd \
    --file=/backup/full-$(date +%Y%m%d).tar.zst \
    /var/www /etc /home

# Incremental backup (run next day - same snapshot file)
tar --create \
    --listed-incremental=/var/backup/snapshot.snar \
    --zstd \
    --file=/backup/inc-$(date +%Y%m%d-%H%M).tar.zst \
    /var/www /etc /home

# Restore: full first, then each incremental in order
tar --extract \
    --listed-incremental=/dev/null \
    --file=/backup/full-20260820.tar.zst \
    --directory=/restore

tar --extract \
    --listed-incremental=/dev/null \
    --file=/backup/inc-20260821-0200.tar.zst \
    --directory=/restore

Streaming Archives Over SSH and to Object Storage

tar's stream output is one of its most powerful features. You can pipe a remote directory to a local archive without writing a temporary file on either side, and you can push directly to S3-compatible storage without staging on disk.

Streaming to S3 via AWS CLI uses a multipart upload behind the scenes. The aws s3 cp command accepts stdin when you specify - as the source and provide --expected-size. For buckets without --expected-size enforcement, omit it. Test this first with a small directory before running against production data.

For naming your backup buckets and prefixes consistently across projects - especially if you manage multiple clients or environments - having a canonical naming convention matters. We've used nicename.me for checking domain and project name availability when setting up new client environments, since consistent naming across DNS, S3 buckets, and hostnames prevents the confusion that comes from ad hoc choices six months in.

# Copy remote directory to local tar.zst without temp file
ssh user@remote-server 'tar --create --zstd /var/www' > remote-backup.tar.zst

# Remote to remote via local machine as intermediary
ssh user@source 'tar --create /data' | ssh user@dest 'tar --extract --directory=/data'

# Stream directly to S3 (requires awscli v2)
tar --create --zstd /var/www | \
    aws s3 cp - s3://mybucket/backups/www-$(date +%Y%m%d).tar.zst \
    --expected-size 2000000000

# Stream from S3 and extract
aws s3 cp s3://mybucket/backups/www-20260820.tar.zst - | \
    tar --extract --zstd --directory=/restore

# Measure transfer rate with pv
tar --create /large-dataset | pv | zstd -3 -T0 | \
    ssh user@backup-server 'cat > /backup/dataset.tar.zst'
// advertisement

xz and Multi-core Compression with pixz and xz -T

xz is single-threaded by default in versions before 5.4. xz 5.4.0 (released January 2023) added native multi-threading via -T flag. Ubuntu 24.04 ships xz-utils 5.6.1. Check your version before assuming -T0 works.

Before xz 5.4, the standard workaround was pixz, a parallel xz implementation that also supports random access within tar archives - meaning you can extract a single file from a .tar.xz without decompressing everything before it. This is the same advantage zip has, but with better compression ratios.

For compression tasks in CI/CD pipelines, xz's memory requirements matter. At level 6 (default), xz uses about 96 MB RAM for compression. At level 9, it uses up to 683 MB. On shared build runners with 2 GB RAM limits, defaulting to level 6 or switching to zstd avoids OOM kills.

# Check xz version
xz --version

# Multi-threaded xz (5.4+ native)
xz -6 -T0 large_file.tar   # -T0 = use all cores

# pixz for older systems or random access
apt install pixz
tar --create /data | pixz -p 8 > archive.tar.xz   # 8 threads

# Extract single file from pixz archive (random access)
pixz -x path/in/archive/specific_file.conf < archive.tar.xz

# Check memory usage at different levels
xz --robot --info-memory
# Shows encoder/decoder memory at each compression level

# Decompress only (xz -d or unxz)
unxz archive.tar.xz
xz -dk archive.tar.xz   # decompress, keep original

Verifying Archives Before You Need Them

An unverified backup is not a backup. Every archive format provides a test mode that checks internal checksums without extracting to disk. Run this before the archive goes to cold storage, and again periodically afterward.

For critical archives, add SHA-256 checksums stored separately from the archive itself. If the archive and its checksum live in the same location, a storage failure corrupts both. Store checksums in a separate bucket, host, or medium.

In automated pipelines, failed verification should trigger an immediate re-run or alert. A broken nightly backup discovered three weeks later during a restore incident is a worse outcome than a failed backup that pages someone at 3 AM.

# Test tar archive integrity
tar --test-label --file=backup.tar.gz   # checks volume label
tar --list --file=backup.tar.gz > /dev/null && echo 'tar OK'

# Test gzip integrity
gzip -t backup.tar.gz && echo 'gzip OK'

# Test zip integrity
unzip -t archive.zip

# Test xz integrity
xz --test archive.tar.xz

# Test zstd integrity
zstd -t archive.tar.zst

# Generate and store SHA-256 alongside archive
sha256sum backup.tar.zst > backup.tar.zst.sha256

# Verify checksum later
sha256sum --check backup.tar.zst.sha256

# Combine: create, compress, verify, checksum in one pipeline
tar --create /var/www | \
    zstd -3 -T0 | \
    tee >(sha256sum > backup.tar.zst.sha256) > backup.tar.zst && \
    zstd -t backup.tar.zst && \
    echo 'Archive verified'

Practical tar Flags for Production Use

Several tar flags that rarely appear in tutorials but matter in real deployments:

--exclude-caches: skips directories containing a CACHEDIR.TAG file (browser caches, npm cache, pip cache). Add this to any backup that includes home directories.

--one-file-system: prevents tar from crossing filesystem boundaries. Essential when backing up / to avoid including /proc, /sys, /dev, or NFS mounts.

--checkpoint and --checkpoint-action: prints progress every N records. On multi-hour archives over slow links, this gives you visibility without piping through pv.

--ignore-failed-read: continues archiving if a file is unreadable instead of aborting the entire job. Useful for backing up directories where some files may be locked or permission-restricted. Note this can silently skip files - always check exit codes and logs.

--transform: renames files inside the archive using sed syntax. Useful when you need to strip or add path components without modifying the source.

# Skip cache directories automatically
tar --create \
    --exclude-caches \
    --one-file-system \
    --zstd \
    --file=home-backup.tar.zst \
    /home

# Progress every 10000 records (~5 MB increments)
tar --create \
    --checkpoint=10000 \
    --checkpoint-action=echo='Checkpoint %{r}T' \
    --zstd \
    --file=large.tar.zst \
    /data

# Continue on read errors (check logs after)
tar --create \
    --ignore-failed-read \
    --file=partial.tar.gz \
    /var/log 2>/var/log/tar-errors.log

# Rename paths inside archive using transform
tar --create \
    --transform='s|^var/www/html|webroot|' \
    --file=renamed.tar.gz \
    var/www/html

# Exclude multiple patterns
tar --create \
    --exclude='*.tmp' \
    --exclude='*/.git' \
    --exclude='*/node_modules' \
    --zstd \
    --file=app.tar.zst \
    /opt/app
// advertisement

Automation and Pipeline Integration

Shell-based archive pipelines work well up to a certain scale. When you have dozens of servers, multiple retention windows, cross-region replication, and failure notification requirements, hand-written shell scripts become maintenance liabilities. Platforms like taskbotshub.ai provide workflow orchestration specifically for DevOps tasks - you define the backup job, retention policy, and alert routing, and the platform handles scheduling, retries, and reporting across your fleet.

For teams staying with shell-based automation, the pattern that works best in our experience is small, composable scripts that do one thing: create archive, verify archive, upload archive, prune old archives. Each step is a separate script with proper exit codes, logging to syslog or a structured log file, and a lockfile to prevent concurrent runs. Wire them together with systemd timers rather than cron - systemd gives you dependency management, timeout enforcement, and automatic logging.

A complete production backup unit looks like the example below. The ExecStartPost line runs verification immediately after the backup completes, and OnFailure routes to a notification service.

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly application backup
After=network-online.target
Wants=network-online.target
OnFailure=backup-notify@%n.service

[Service]
Type=oneshot
User=backup
Group=backup
ExecStart=/usr/local/bin/create-backup.sh
ExecStartPost=/usr/local/bin/verify-backup.sh
ExecStartPost=/usr/local/bin/upload-backup.sh
TimeoutStartSec=3600
StandardOutput=journal
StandardError=journal
SyslogIdentifier=nightly-backup

# /etc/systemd/system/nightly-backup.timer
[Unit]
Description=Run nightly-backup.service at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=300
Persistent=true

[Install]
WantedBy=timers.target

# Enable and start
systemctl daemon-reload
systemctl enable --now nightly-backup.timer
systemctl list-timers nightly-backup.timer