How LVM Snapshots Actually Work
LVM snapshots use copy-on-write (CoW). When you create a snapshot of a logical volume, LVM allocates a chunk of space from the volume group but copies nothing immediately. The original LV continues operating normally. The moment any block on the origin LV is about to be overwritten, LVM first copies the original block into the snapshot's CoW store, then allows the write to proceed.
This means snapshot creation is nearly instant regardless of LV size. A 500 GB database volume snapshots in the same time as a 1 GB one. The performance cost shows up during writes to the origin after the snapshot exists - each write requires one extra read (fetch original block) and one extra write (save it to the snapshot store) before the actual write completes. On a busy write-heavy volume this overhead is measurable. In our testing on NVMe storage with a write-heavy Postgres workload, we saw 8-12% write latency increase while a snapshot was live.
The snapshot fills up as blocks on the origin get modified. If the snapshot's CoW store fills completely, LVM invalidates the snapshot - it becomes unusable and gets marked as inactive. This is the most common mistake: undersizing the snapshot store. You cannot extend a classic LVM snapshot after creation (thin-provisioned snapshots work differently, covered below).
# Check current VG free space before planning snapshot size
vgs --units g
# Detailed view of the target LV
lvs -o lv_name,lv_size,vg_name,lv_attr --units g /dev/vgdata/lvpgSnapshot Sizing: The Math That Matters
Snapshot size is not the size of the origin LV. It is the amount of changed data you expect during the backup window. If your backup takes 45 minutes and your database writes 2 GB of dirty blocks per minute under normal load, you need at least 90 GB of snapshot space plus a safety margin. We use 25% overhead on top of the calculated write rate.
To measure your actual write rate, use lvs with monitoring or watch iostat against the device:
A write rate of 500 MB/min over a 30-minute backup window means 15 GB of changed blocks. Add 25% and allocate 19 GB minimum. If your VG has space, go to 25 GB. The cost of a failed snapshot because it ran out of space is much higher than the cost of a few extra gigabytes allocated.
For databases with WAL or redo logs, factor those in separately. PostgreSQL on a busy system can generate 1-5 GB of WAL per minute during bulk operations. If you are snapshotting during a maintenance window that includes VACUUM FULL or a large import, your write rate spikes.
You can monitor snapshot usage in real time with lvs -o lv_name,snap_percent. When that number climbs above 70%, you have a problem.
# Monitor write rate on the origin device (example: dm-3)
iostat -x 2 10 /dev/dm-3 | awk 'NR>3 {print $1, $7" MB/s writes"}"
# Check snapshot fill percentage while backup is running
watch -n 5 'lvs -o lv_name,lv_size,snap_percent --units g'Creating an LVM Snapshot: Full Procedure
Before creating the snapshot, flush application writes to disk if possible. For PostgreSQL, issue a CHECKPOINT. For MySQL/InnoDB, run FLUSH TABLES WITH READ LOCK, take the snapshot, then UNLOCK TABLES immediately. The lock window is typically under 2 seconds even on large volumes. For filesystems, sync and optionally freeze the filesystem using fsfreeze.
The -s flag creates a snapshot. The -n flag names it. The -L flag sets the CoW store size. The origin LV path goes at the end.
After creation, verify it is valid with lvs. The lv_attr field will show 's' in the sixth position for a valid snapshot. If it shows 'S', the snapshot is invalid and you need to start over with more space.
Mount the snapshot read-only. You must add the nouuid option for XFS filesystems, or the kernel will refuse to mount a duplicate UUID. For ext4, add the ro flag. Never mount a snapshot read-write unless you intend to use it as a rollback target.
# Freeze the filesystem to ensure consistency (optional but recommended)
fsfreeze -f /var/lib/postgresql
# Create the snapshot: 20G CoW store for origin LV lvpg
lvcreate -s -n lvpg_snap -L 20G /dev/vgdata/lvpg
# Immediately unfreeze
fsfreeze -u /var/lib/postgresql
# Verify snapshot is valid (look for 's' not 'S' in attr)
lvs -o lv_name,lv_size,snap_percent,lv_attr /dev/vgdata/lvpg_snap
# Mount XFS snapshot read-only
mkdir -p /mnt/pgsnap
mount -o ro,nouuid /dev/vgdata/lvpg_snap /mnt/pgsnap
# Mount ext4 snapshot read-only
mount -o ro /dev/vgdata/lvpg_snap /mnt/pgsnapBacking Up From the Snapshot
With the snapshot mounted at /mnt/pgsnap, run your backup tool against that path. The origin LV continues serving traffic. You have effectively decoupled your backup window from application availability.
For a raw block-level backup, use dd or ddrescue directly against the snapshot LV device rather than the mountpoint. This captures the entire filesystem including free space, which is wasteful. Better to mount and use rsync or tar for file-level backups, or pg_basebackup pointed at the mounted data directory for PostgreSQL.
For PostgreSQL specifically: pg_basebackup requires a running server. If you want a filesystem-level backup from the snapshot, you need the cluster to have been in a consistent state at snapshot time. The CHECKPOINT before fsfreeze handles this. Then you can tar the mounted snapshot directory and it will be a valid base backup.
For a streaming backup to S3 or a remote server, pipe directly from tar to avoid writing a full local copy:
Backup duration no longer affects your application. A backup that takes 3 hours pulls from the frozen-in-time snapshot, not the live volume.
# Full filesystem backup from snapshot via tar + gzip to remote
tar -czf - -C /mnt/pgsnap . | ssh backup-server 'cat > /backups/pgdata_$(date +%Y%m%d).tar.gz'
# Or pipe directly to S3 via AWS CLI
tar -czf - -C /mnt/pgsnap . | aws s3 cp - s3://your-bucket/pgdata/$(date +%Y%m%d_%H%M).tar.gz
# Block-level snapshot backup (captures entire LV, not just used space)
dd if=/dev/vgdata/lvpg_snap bs=4M | gzip | ssh backup-server 'cat > /backups/lvpg_raw.gz'
# Check snapshot health during backup
lvs -o snap_percent /dev/vgdata/lvpg_snapRemoving Snapshots Cleanly
Once the backup completes, unmount and remove the snapshot. Do not leave snapshots live longer than necessary. Every write to the origin while the snapshot exists adds CoW overhead. We have seen admins leave snapshots running for days and wonder why their IO latency doubled.
Snapshot removal is handled by lvremove. It does not touch the origin LV in any way.
If the snapshot filled up and became invalid before your backup completed, you have lost the snapshot but the origin LV is fine. Check with lvs and look for 'S' in lv_attr or 100% in snap_percent. Remove the invalid snapshot, increase your allocated size, and try again.
After removal, confirm the VG space is returned:
# Unmount first
umount /mnt/pgsnap
# Remove the snapshot
lvremove -f /dev/vgdata/lvpg_snap
# Confirm space returned to VG
vgs --units g vgdataThin-Provisioned Snapshots: When to Use Them
Classic LVM snapshots as described above are thick-provisioned: you allocate the CoW store upfront and cannot grow it. Thin-provisioned LVM (thin pools) gives you snapshots that can grow dynamically from a shared pool, and you can snapshot snapshots, creating chains.
Thin snapshots live inside a thin pool LV. You create the pool first, create thin LVs inside it, then snapshot those. The performance characteristics differ: thin CoW uses a different metadata structure and performs better on some workloads, worse on others with high random write patterns.
For backup purposes, thin snapshots are more forgiving because the pool can absorb bursts of writes without the snapshot invalidating. The tradeoff is that the thin pool metadata can become a bottleneck, and pool exhaustion is more complex to debug.
We use thin provisioning when the origin volumes have unpredictable write rates or when we need to keep multiple snapshot generations alive simultaneously. For straightforward backup automation with known write rates, classic snapshots are simpler and have fewer moving parts.
# Create a thin pool (50G pool with 1% metadata)
lvcreate -L 50G --thinpool thinpool vgdata
# Create a thin LV inside the pool
lvcreate -V 20G --thin -n lvpg_thin vgdata/thinpool
# Snapshot a thin LV (no size needed - uses pool space dynamically)
lvcreate -s -n lvpg_thinsnap vgdata/lvpg_thin
# Check thin pool usage
lvs -o lv_name,lv_size,data_percent,metadata_percent vgdata/thinpoolSnapshot-Based Rollbacks
Snapshots also serve as rollback points before risky operations: major OS upgrades, application schema migrations, kernel updates. The procedure is: create snapshot, run the risky operation, either merge the snapshot (discard it, keeping the new state) or revert.
To revert: unmount the origin LV (requires stopping services), then merge the snapshot back with lvconvert. The merge completes on next mount.
The merge operation is not instant for large volumes with many CoW pages. On a 100 GB LV with 30 GB of changes, we saw merge times of 4-8 minutes. Plan for this in your rollback SLA.
For schema migrations specifically, a better pattern is to snapshot the database LV, promote the snapshot to a separate test environment, run the migration there, validate, then run it on production without the snapshot overhead. This avoids the CoW write penalty during the migration itself.
# Stop services that use the LV
systemctl stop postgresql
# Unmount the origin
umount /var/lib/postgresql
# Merge snapshot into origin (reverts origin to snapshot state)
lvconvert --merge /dev/vgdata/lvpg_snap
# Activate and remount
lvchange -ay /dev/vgdata/lvpg
mount /dev/vgdata/lvpg /var/lib/postgresql
systemctl start postgresqlAutomating LVM Snapshots for Regular Backups
A production backup schedule needs automation with monitoring. The following script handles the full lifecycle: freeze, snapshot, unfreeze, backup, cleanup. Add alerting when snap_percent crosses 80% so you can intervene before invalidation.
For teams running full GitOps or CI/CD pipelines, integrating LVM snapshot backups into your automation platform saves significant operational time. Platforms like taskbotshub.ai provide prebuilt DevOps automation workflows that can wrap these shell operations with retry logic, alerting, and audit trails without requiring you to maintain custom cron scripts indefinitely.
For cron-based scheduling, run the script as root and log to syslog. A typical production schedule runs nightly at 02:00 with a retention of 7 days on the remote backup store.
Critical: test your restore procedure quarterly. A backup that has never been restored is not a backup. Mount the backup archive on a test system, start PostgreSQL against it, and verify data integrity with pg_dumpall or application-level checks.
#!/bin/bash
# /usr/local/sbin/lvm-snap-backup.sh
set -euo pipefail
VG="vgdata"
LV="lvpg"
SNAP_NAME="${LV}_snap"
SNAP_SIZE="20G"
MOUNT_POINT="/mnt/snap_backup"
BACKUP_DEST="backup-server:/backups/pgdata"
DATE=$(date +%Y%m%d_%H%M)
LOG_TAG="lvm-snap-backup"
log() { logger -t "$LOG_TAG" "$1"; echo "$1"; }
cleanup() {
umount "$MOUNT_POINT" 2>/dev/null || true
lvremove -f "/dev/${VG}/${SNAP_NAME}" 2>/dev/null || true
}
trap cleanup EXIT
# Flush PostgreSQL
psql -U postgres -c 'CHECKPOINT;' || log "WARN: CHECKPOINT failed, continuing"
# Freeze filesystem
fsfreeze -f /var/lib/postgresql
log "Filesystem frozen"
# Create snapshot
lvcreate -s -n "$SNAP_NAME" -L "$SNAP_SIZE" "/dev/${VG}/${LV}"
log "Snapshot created: /dev/${VG}/${SNAP_NAME}"
# Immediately unfreeze
fsfreeze -u /var/lib/postgresql
log "Filesystem unfrozen"
# Mount snapshot
mkdir -p "$MOUNT_POINT"
mount -o ro,nouuid "/dev/${VG}/${SNAP_NAME}" "$MOUNT_POINT"
# Check initial snap_percent
SNAP_PCT=$(lvs --noheadings -o snap_percent "/dev/${VG}/${SNAP_NAME}" | tr -d ' ')
log "Snapshot fill at mount time: ${SNAP_PCT}%"
# Backup
tar -czf - -C "$MOUNT_POINT" . | ssh "$BACKUP_DEST/${DATE}.tar.gz" 'cat > /dev/stdin'
log "Backup complete: ${DATE}.tar.gz"
# Cleanup handled by trapCommon Failures and How to Diagnose Them
Snapshot invalidation is the most frequent problem. Check immediately with `lvs -o lv_name,snap_percent,lv_attr`. An attr string with 'S' (capital) at position six means invalid. Root cause is almost always undersized CoW store or an unexpected write spike during backup. Solution: remove the invalid snapshot, increase the size calculation, retry.
Mount failures on XFS with 'wrong fs type' or UUID conflicts: always use nouuid option for XFS. If you forget, unmount, then remount with the correct options. The snapshot LV is not damaged.
Performance degradation worse than expected: verify you are not running multiple simultaneous snapshots. Each additional snapshot on the same origin multiplies the CoW overhead. Never run more than two simultaneous snapshots on a write-heavy LV in production.
dm-snapshot errors in dmesg indicating metadata corruption: this occasionally happens when the host crashes while a snapshot is active. Run `lvchange --refresh /dev/vgdata/lvpg_snap` to attempt recovery. If the snapshot is corrupted, remove it. The origin LV is unaffected.
VG space not returned after lvremove: check for open file descriptors with `lsof | grep dm-` or leftover mounts with `findmnt`. A process holding the device prevents space reclamation.
# Diagnose snapshot state
lvs -o lv_name,lv_size,snap_percent,lv_attr,vg_name
# Check dmesg for dm-snapshot errors
dmesg | grep -i 'snapshot\|dm-' | tail -20
# Find processes using snapshot device
lsof | grep "$(lvdisplay /dev/vgdata/lvpg_snap | grep 'LV Path' | awk '{print $3}')"
# Force deactivate and remove a stuck snapshot
dmsetup remove vgdata-lvpg_snap
lvremove -f /dev/vgdata/lvpg_snapLVM Snapshots Versus Other Backup Methods
Compared to pg_basebackup with continuous WAL archiving: pg_basebackup is PostgreSQL-specific and requires the server running in backup mode. LVM snapshots are filesystem-agnostic and work for any application. LVM is faster to implement for mixed workloads. WAL-based approaches give you point-in-time recovery between snapshots, which LVM alone cannot provide.
Compared to filesystem-level snapshots (Btrfs, ZFS): Btrfs and ZFS snapshots are instant, do not require a separate CoW store allocation, and have better tooling for snapshot management. If you are on ZFS (and you should consider it for new storage deployments), use ZFS send/receive instead of LVM snapshots. LVM snapshots exist because most production Linux systems run on ext4 or XFS on standard LVM stacks, not ZFS.
Compared to storage-level snapshots (NetApp, Ceph RBD, AWS EBS): storage-level snapshots have zero CoW overhead on the host and often support instant clones. If your infrastructure supports it, prefer storage-level snapshots for production. LVM snapshots are the right tool when you are on bare metal or a hypervisor without native snapshot APIs.
Compared to rsync without a snapshot: rsync against a live filesystem risks capturing inconsistent state - a file half-written, a database in the middle of a transaction. LVM snapshots eliminate this class of problem entirely. For any database or application with transactional consistency requirements, rsync-without-snapshot is not an acceptable backup method.
# ZFS equivalent for comparison - instant, no pre-allocation
zfs snapshot tank/pgdata@backup_$(date +%Y%m%d)
zfs send tank/pgdata@backup_$(date +%Y%m%d) | ssh backup-server 'zfs receive tank/backups/pgdata'
# AWS EBS equivalent via CLI
aws ec2 create-snapshot --volume-id vol-0abc123 --description "pgdata backup $(date +%Y%m%d)"