Confirming Your OpenZFS Version and Module State

Before touching a single disk, verify what version of OpenZFS is running and that the kernel modules are loaded.

On a fresh FreeBSD 14.1-RELEASE install, zfs.ko loads automatically if vfs.root.mountfrom references a ZFS pool. On systems where the root filesystem is UFS and you are adding ZFS for data, load the module manually and make it persistent.

# Check loaded modules
kldstat | grep zfs

# Load manually if needed
kldload zfs

# Make persistent across reboots
echo 'zfs_enable="YES"' >> /etc/rc.conf

# Confirm OpenZFS version
zpool version
# Expected output on FreeBSD 14.1: zpool version 5000
# This maps to OpenZFS 2.2.x feature flags

zfs version
# zfs-2.2.x-FreeBSD_g....

Disk Identification and Labeling Before Pool Creation

Never use raw device names like /dev/da0 in a zpool create command. If you add or remove a disk, device enumeration can shift and you end up with a corrupt pool config. Use GPT labels or disk IDs instead.

FreeBSD's camcontrol and gpart tools handle this. Run camcontrol devlist to see physical disks, then label each one before pool creation.

# List all SCSI/SATA/NVMe devices
camcontrol devlist

# Create a GPT partition table on each disk (destroys existing data)
gpart create -s gpt da1
gpart create -s gpt da2
gpart create -s gpt da3
gpart create -s gpt da4

# Add a single freebsd-zfs partition spanning the whole disk
gpart add -t freebsd-zfs -l zfs-disk0 da1
gpart add -t freebsd-zfs -l zfs-disk1 da2
gpart add -t freebsd-zfs -l zfs-disk2 da3
gpart add -t freebsd-zfs -l zfs-disk3 da4

# Verify labels
ls -la /dev/gpt/
# Output: zfs-disk0  zfs-disk1  zfs-disk2  zfs-disk3

Creating Pools: Mirror, RAIDZ1, RAIDZ2

Pool topology is permanent. You can add vdevs to a pool but you cannot change the redundancy level of an existing vdev without recreating it (RAIDZ expansion is available in OpenZFS 2.2 but still experimental on FreeBSD as of 14.1). Choose correctly the first time.

For 4-disk setups, a 2-way mirror striped pair outperforms RAIDZ1 on IOPS and has simpler resilver mechanics. RAIDZ2 on 6 disks is the standard choice for bulk storage where capacity matters more than write IOPS.

The -o ashift=12 flag is mandatory for any drive with 4K physical sectors, which includes virtually all drives manufactured after 2015. Misaligned ashift causes permanent performance degradation.

For NVMe pools, use ashift=13 if the drive reports 8K optimal transfer size, or verify with: diskinfo -v /dev/nvme0ns1 | grep sectorsize

# 2-way mirrored stripe (4 disks = 2x mirror vdevs)
zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  tank mirror /dev/gpt/zfs-disk0 /dev/gpt/zfs-disk1 \
       mirror /dev/gpt/zfs-disk2 /dev/gpt/zfs-disk3

# RAIDZ2 on 6 disks
zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  datapool raidz2 \
  /dev/gpt/zfs-disk0 /dev/gpt/zfs-disk1 /dev/gpt/zfs-disk2 \
  /dev/gpt/zfs-disk3 /dev/gpt/zfs-disk4 /dev/gpt/zfs-disk5

# Verify pool status
zpool status tank
zpool list -v tank
// advertisement

Adding L2ARC and SLOG Devices

The ARC (Adaptive Replacement Cache) lives in RAM. L2ARC extends read cache to an SSD. SLOG (Separate Intent Log) offloads synchronous write acknowledgment to a fast device, which matters for databases and NFS exports using sync=standard.

On our test server with 64GB RAM, ARC peaked at 48GB before the kernel reclaimed memory under pressure. Adding a 480GB SSD as L2ARC gave measurable improvement only on workloads with a working set larger than RAM. Do not add L2ARC to pools doing sequential writes - it wastes SSD write endurance.

SLOG devices must be redundant. A single SLOG failure causes the pool to lose the intent log device but the pool itself survives. However, unmirrored SLOG is poor practice for anything holding live data. Use a mirrored pair of NVMe or Intel Optane slices.

# Add a mirrored SLOG (two NVMe partitions)
zpool add tank log \
  mirror /dev/gpt/slog-nvme0 /dev/gpt/slog-nvme1

# Add a single L2ARC device (acceptable - L2ARC loss is not data loss)
zpool add tank cache /dev/gpt/l2arc-ssd0

# Verify the vdev layout
zpool status tank
# You should see:
#   logs
#     mirror  ONLINE
#       gpt/slog-nvme0  ONLINE
#       gpt/slog-nvme1  ONLINE
#   cache
#     gpt/l2arc-ssd0  ONLINE

# Monitor L2ARC hit rate
arc_summary | grep -A5 'L2 ARC'

Dataset Layout and Property Inheritance

ZFS datasets are not partitions. They are namespace entries that inherit properties from their parent unless overridden. Structure your dataset hierarchy to exploit this - set common properties at the pool root and override only where necessary.

A practical layout for a FreeBSD application server separates system data, user home directories, databases, and backup targets. This lets you apply different snapshot schedules, compression algorithms, and quota policies per dataset without duplicating config.

The recordsize property is critical for database workloads. The default 128K recordsize is optimal for sequential reads but wasteful for PostgreSQL which uses 8K pages. Set recordsize=8K on database datasets before writing any data - changing it after the fact only affects newly written blocks.

# Create a structured dataset hierarchy
zfs create tank/apps
zfs create tank/apps/nginx
zfs create tank/apps/postgres
zfs create tank/home
zfs create tank/backups

# Override recordsize for PostgreSQL
zfs set recordsize=8K tank/apps/postgres

# Enable dedup only where it makes sense (check ram ratio first)
# Rule: dedup table needs ~320MB RAM per 1TB of deduplicated data
zfs set dedup=on tank/backups

# Set quota and reservation on user home dirs
zfs set quota=50G tank/home
zfs set reservation=10G tank/home

# Verify inherited and local properties
zfs get all tank/apps/postgres | grep -E 'recordsize|compression|atime'

# List all datasets with key properties
zfs list -o name,used,avail,refer,mountpoint,compression,recordsize tank

Snapshot Management and Automated Schedules

Snapshots on ZFS are instantaneous and space-efficient until data changes. On our test server, creating a snapshot of a 200GB dataset takes under 50 milliseconds. The space cost begins only when blocks in the snapshot diverge from the live dataset.

For automated snapshots, FreeBSD's base system includes periodic(8) integration but the most reliable tool for production is zfstools or zrepl. We use zrepl on our production servers because it handles both local snapshots and replication in a single daemon with a clean YAML config.

Install via pkg: pkg install zrepl

Manual snapshot operations are straightforward. The naming convention matters - use ISO 8601 timestamps so snapshots sort lexicographically.

# Create a snapshot manually
zfs snapshot tank/apps/postgres@2026-06-23T14:00:00

# Recursive snapshot of entire pool hierarchy
zfs snapshot -r tank@2026-06-23T14:00:00

# List snapshots
zfs list -t snapshot -o name,used,refer,creation tank/apps/postgres

# Roll back to a snapshot (destroys data written after snapshot)
zfs rollback tank/apps/postgres@2026-06-23T14:00:00

# Destroy a single snapshot
zfs destroy tank/apps/postgres@2026-06-23T14:00:00

# Destroy all snapshots older than a pattern (use with care)
zfs list -t snapshot -H -o name tank/apps/postgres | \
  grep '2026-06-22' | xargs -n1 zfs destroy

# Clone a snapshot (creates a writable copy)
zfs clone tank/apps/postgres@2026-06-23T14:00:00 tank/apps/postgres-clone
// advertisement

ZFS Replication with zfs send and zfs receive

zfs send | zfs receive is the standard mechanism for pool-to-pool replication, both local and over SSH. The incremental form (-i for single increment, -I for full range) transfers only changed blocks, making it practical for continuous offsite backup.

For a remote backup server at 192.168.1.50, the initial seed transfer uses the full stream. Subsequent transfers use incremental. Pipe through mbuffer if the network has variance - it absorbs throughput spikes and prevents the sender from blocking.

Install mbuffer first: pkg install mbuffer

In our experience, adding lz4 compression via ssh -C is counterproductive on already-compressed ZFS streams. The data is already compressed by ZFS before it hits the pipe. Skip ssh -C and let ZFS compression handle it.

# Initial full replication (seed)
zfs send -R tank/apps/postgres@2026-06-23T14:00:00 | \
  mbuffer -s 128k -m 1G | \
  ssh backup@192.168.1.50 \
  'mbuffer -s 128k -m 1G | zfs receive -F backup/postgres'

# Incremental replication (subsequent runs)
zfs send -i \
  tank/apps/postgres@2026-06-22T14:00:00 \
  tank/apps/postgres@2026-06-23T14:00:00 | \
  mbuffer -s 128k -m 1G | \
  ssh backup@192.168.1.50 \
  'mbuffer -s 128k -m 1G | zfs receive backup/postgres'

# Verify the remote snapshot exists
ssh backup@192.168.1.50 \
  'zfs list -t snapshot backup/postgres'

# Resume an interrupted send (OpenZFS 2.x)
# On receiver, get the resume token:
ssh backup@192.168.1.50 \
  'zfs get receive_resume_token backup/postgres'
# Then on sender:
zfs send -t  | \
  ssh backup@192.168.1.50 'zfs receive backup/postgres'

Tuning the ARC, ZIL, and Module Parameters

FreeBSD's OpenZFS ARC defaults to using up to half of installed RAM. On a dedicated storage server with 128GB RAM, this means 64GB for ARC, which is appropriate. On a hypervisor running multiple VMs, you need to cap it to leave memory for guests.

ZFS tunable parameters live in /boot/loader.conf on FreeBSD. Changes to most arc parameters require a reboot. The vfs.zfs namespace is the relevant one - do not confuse it with the Solaris-era zfs_arc_max sysctl path.

The vfs.zfs.zil_slog_bulk controls how much data goes to SLOG vs ARC-backed ZIL. The default 786432 (768KB) is conservative. For write-heavy workloads with a fast SLOG device, increase it.

If you are running ZFS on a server that also runs jails or bhyve VMs, consider using zfs-stats or arc_summary from the sysutils/py-zfs-stats package to monitor ARC hit rates before adjusting limits.

# /boot/loader.conf tuning entries

# Cap ARC at 32GB on a 64GB machine running other workloads
vfs.zfs.arc_max="34359738368"

# Set ARC minimum (prevent OS from reclaiming too aggressively)
vfs.zfs.arc_min="8589934592"

# Increase SLOG threshold to 4MB for write-heavy postgres
vfs.zfs.zil_slog_bulk="4194304"

# Disable prefetch on random-IO workloads (databases)
vfs.zfs.prefetch_disable="1"

# Tune transaction group timeout (default 5s, lower = more frequent syncs)
vfs.zfs.txg.timeout="3"

# Apply sysctl changes live (arc_max can be changed without reboot)
sysctl vfs.zfs.arc_max=34359738368

# Monitor current ARC size
sysctl vfs.zfs.arc_size
arc_summary

Scrubs, Resilver Monitoring, and Pool Health Checks

A monthly scrub is the minimum acceptable maintenance cadence. Weekly is better for pools holding irreplaceable data. Add a periodic(8) entry or a cron job. On our test server with 4x4TB in RAIDZ2, a full scrub completes in approximately 6 hours at ~280MB/s.

During resilver after a disk replacement, monitor progress with zpool status. Resilver speed on spinning disks is typically 60-120MB/s, meaning a 4TB replacement takes 10-18 hours. Do not interrupt it. If the server must reboot, resilver resumes automatically from a checkpoint in OpenZFS 2.x.

For disk replacement, the correct procedure is: offline the failed device, physically swap it, bring it online or use zpool replace. The order matters - offling first prevents a split-brain state.

# Manual scrub
zpool scrub tank

# Monitor scrub progress
watch -n 5 zpool status tank

# Add weekly scrub via cron
# Add to /etc/crontab:
# 0 2 * * 0 root /sbin/zpool scrub tank

# Disk replacement procedure
zpool offline tank /dev/gpt/zfs-disk2
# (physically swap the drive, re-label the new disk)
gpart create -s gpt da2
gpart add -t freebsd-zfs -l zfs-disk2 da2

zpool replace tank /dev/gpt/zfs-disk2

# Monitor resilver
zpool status tank
# Look for: resilver in progress, X.XX% done

# Check pool I/O statistics during resilver
zpool iostat -v tank 5

# Export pool status to a file for monitoring integration
zpool status -x tank >> /var/log/zpool-health.log
// advertisement

Encryption with Native ZFS

OpenZFS 2.x native encryption uses AES-256-GCM by default. It operates at the dataset level, not the pool level, which means you can have unencrypted datasets in the same pool as encrypted ones. Encryption keys are not stored on disk - you provide a passphrase or keyfile at boot or mount time.

For servers that need to auto-mount encrypted datasets after reboot (common for headless systems), store the key as a file on a separate volume (tmpfs populated from a network KMS, or a USB token) and reference it via keylocation.

In our experience, native ZFS encryption adds approximately 5-8% CPU overhead on modern hardware with AES-NI. On a Xeon Gold 6338 running our benchmark, we measured 2.1GB/s write throughput unencrypted vs 1.94GB/s encrypted - negligible for most workloads.

# Create an encrypted dataset with passphrase
zfs create \
  -o encryption=aes-256-gcm \
  -o keyformat=passphrase \
  -o keylocation=prompt \
  tank/apps/secrets

# Create with a keyfile (for automated mounting)
dd if=/dev/urandom of=/root/tank-secrets.key bs=32 count=1
chmod 400 /root/tank-secrets.key

zfs create \
  -o encryption=aes-256-gcm \
  -o keyformat=raw \
  -o keylocation=file:///root/tank-secrets.key \
  tank/apps/secrets-auto

# Load key and mount on boot (add to /etc/rc.local or rc.d script)
zfs load-key tank/apps/secrets-auto
zfs mount tank/apps/secrets-auto

# Check encryption status
zfs get encryption,keystatus,keylocation tank/apps/secrets-auto

# Rotate the encryption key
zfs change-key \
  -o keylocation=file:///root/tank-secrets-new.key \
  tank/apps/secrets-auto

Integrating ZFS with Jails and Bhyve

FreeBSD jails benefit from ZFS datasets as their root filesystems. Each jail gets its own dataset, enabling per-jail snapshots, quotas, and instant cloning for staging environments. This is functionally equivalent to container layer filesystems on Linux but implemented at the storage layer.

For bhyve virtual machines, use zvols instead of datasets. A zvol is a block device backed by ZFS, giving the VM raw block access while retaining ZFS snapshot capabilities. Use volblocksize=16K for VMs running Linux guests with 4K sectors, and volblocksize=8K for Windows guests.

If you are building a DevOps pipeline that provisions and tears down bhyve VMs programmatically, tools like taskbotshub.ai can automate the zfs snapshot, zfs clone, and bhyve startup sequence, reducing manual provisioning work to a configuration file.

# Create a jail dataset
zfs create -o mountpoint=/jails tank/jails
zfs create tank/jails/www01

# Set resource limits on the jail dataset
zfs set quota=20G tank/jails/www01
zfs set reservation=2G tank/jails/www01

# Snapshot before jail updates
zfs snapshot tank/jails/www01@before-update-2026-06-23

# Create a bhyve zvol
zfs create \
  -V 50G \
  -o volblocksize=16K \
  -o compression=lz4 \
  tank/vms/debian12

# The zvol appears as a block device
ls /dev/zvol/tank/vms/debian12

# Snapshot the VM disk before major changes
zfs snapshot tank/vms/debian12@pre-kernel-update

# Clone a VM for a staging environment
zfs clone \
  tank/vms/debian12@pre-kernel-update \
  tank/vms/debian12-staging

# List zvols
zfs list -t volume tank/vms

Monitoring and Alerting in Production

ZFS pool events surface via devd(8) on FreeBSD. You can hook into pool state changes, checksum errors, and resilver completion without polling. Configure /etc/devd.conf to run a script when pool status degrades.

For SMART monitoring of the underlying disks, use smartmontools. pkg install smartmontools, then run smartd with the -d auto flag to auto-detect drive types including NVMe.

For teams running centralized monitoring, export zpool status output as structured data to Prometheus via the node_exporter ZFS collector, or parse the output with a shell script pushing metrics to InfluxDB. The sysutils/node_exporter port includes ZFS stats collection when compiled with --collector.zfs on FreeBSD.

# /etc/devd.conf snippet for ZFS pool event alerting
notify 100 {
    match "system"          "ZFS";
    match "type"            "ESC";
    match "subsystem"       ".*";
    action "logger -t devd-zfs 'ZFS event: $subsystem $type on $device' && \
            /usr/local/sbin/zfs-alert.sh '$subsystem' '$device'";
};

# Restart devd after config change
service devd restart

# Enable SMART monitoring
pkg install smartmontools
echo 'smartd_enable="YES"' >> /etc/rc.conf
service smartd start

# Manual SMART test on a disk
smartctl -t short /dev/da1
smartctl -a /dev/da1 | grep -E 'Reallocated|Pending|Uncorrectable|Temperature'

# Quick pool health check script for cron
zpool status -x
# Exits 0 if all pools healthy, non-zero if any pool has issues
# Use in cron: zpool status -x || mail -s 'ZFS ALERT' admin@example.com < /dev/null
// advertisement