What Each Tool Actually Measures
Before running anything, understand the layer each tool operates at. iostat reads from /proc/diskstats, which the kernel updates per request completion. It gives you aggregated counters over a sampling interval - throughput in KB/s, I/O operations per second, average queue depth, and service time. It is device-centric, not process-centric.
iotop uses the taskstats netlink interface (CONFIG_TASK_IO_ACCOUNTING must be compiled in, which it is on every major distro since kernel 2.6.20). It maps I/O bytes to individual PIDs and threads. The accounting is cumulative per-task, so iotop computes deltas between polls. It answers 'who is doing the I/O' but not 'what exactly are they asking for at the block level'.
blktrace sits below the filesystem and above the device driver, instrumenting the block I/O scheduler via debugfs and the relay filesystem. It records every request lifecycle event - request queued (Q), merged (M), issued to driver (D), completed (C), and more. The output is binary and requires blkparse or btt to read. It answers 'what is the I/O pattern, where is time being spent in the stack'. Combining all three gives you process, device, and request-level visibility simultaneously.
# Verify task I/O accounting is available
grep CONFIG_TASK_IO_ACCOUNTING /boot/config-$(uname -r)
# Should return: CONFIG_TASK_IO_ACCOUNTING=y
# Check blktrace kernel support
ls /sys/kernel/debug/block/iostat: Reading the Output Correctly
iostat ships in the sysstat package. On RHEL/Rocky/Alma 9: `dnf install sysstat`. On Debian/Ubuntu: `apt install sysstat`. The version on our test server was sysstat 12.7.5.
The basic invocation that most sysadmins use is wrong for diagnosis. Running `iostat -x 2` gives extended statistics every 2 seconds, but the first output block is averages since boot - discard it. Your real data starts from the second output block onward.
The columns that matter most in extended mode: `r/s` and `w/s` are read and write operations per second. `rMB/s` and `wMB/s` are throughput. `await` is average time in milliseconds for I/O requests to be served, including queue wait time. `svctm` was deprecated in sysstat 12.0 and is unreliable - ignore it. `%util` is the percentage of time the device had at least one request outstanding. For spinning disks, %util above 80% is a concern. For SSDs and NVMe, %util is largely meaningless because these devices handle parallel queues internally - a NVMe at 10% util can still be saturated.
The metric that actually tells you whether an SSD is saturated is `aqu-sz` (average queue size). If aqu-sz is consistently above the device's queue depth, you are backed up. Check `cat /sys/block/nvme0n1/queue/nr_requests` for the configured queue depth.
On our PostgreSQL server during a checkpoint storm, we saw await jump to 45ms with aqu-sz of 18 on an NVMe device rated for 32 queue depth. That indicated the device was handling it but the application was generating bursts beyond what the scheduler was smoothing out.
# Skip first output, show extended stats every 2s, device nvme0n1 only
iostat -x -d nvme0n1 2 | awk 'NR>6'
# Machine-readable output for scripting (JSON, sysstat 12.3+)
iostat -x -o JSON nvme0n1 2 1
# Show statistics per partition, useful for LVM setups
iostat -xp sda 2
# Persistent logging via sadc (sysstat's collection daemon)
# Edit /etc/sysconfig/sysstat or /etc/default/sysstat
# Then review with sar -d
sar -d -f /var/log/sa/sa$(date +%d) | grep nvme0n1iostat Metrics That Require Context
The `%iowait` metric shown in the CPU section of iostat is one of the most misread numbers in Linux performance monitoring. iowait means a CPU was idle while at least one I/O was outstanding system-wide. It does not mean the CPU was waiting for I/O. A heavily threaded application can show 0% iowait while being completely I/O bound, because other threads keep the CPU busy. Conversely, a single-threaded process waiting on a fast SSD might show 20% iowait simply because nothing else is running.
For latency distribution - something iostat cannot show - you need either blktrace or BPF tools. But for quick triage, the ratio of `await` to `(1000 / (r/s + w/s))` gives you a rough sense of whether latency is coming from queue depth or from actual device response time. If `await` is much larger than the theoretical time per operation, you have a queuing problem, not a device problem.
On systems running dm-multipath or LVM thin pools, iostat output for the underlying physical devices is often more useful than the mapper device. Run `iostat -x -d $(ls /dev/sd? | tr '\n' ' ') 2` to watch all spindles simultaneously during a load test.
# Watch all physical block devices, skip mapper and loop
iostat -x 2 | grep -E '^(sd|nvme|xvd|vd)'
# Calculate IOPS headroom vs manufacturer spec
# Example: device rated 100K IOPS, current r/s + w/s
iostat -x nvme0n1 1 1 | awk '/nvme0n1/{printf "IOPS: %.0f\n", $4+$5}'iotop: Finding the Process Responsible
iotop requires root. The standard invocation shows a live top-like view, but for scripting and logging, batch mode is what you want. iotop is available as the original Python implementation (`iotop`) or the newer C rewrite (`iotop-ng`, packaged as `iotop-c` on some distros). We used iotop-ng 1.24 in our tests - it is noticeably faster and supports filtering by cgroup.
In interactive mode, press `o` to show only processes with active I/O, press `a` to switch from current bandwidth to accumulated totals, and press `p` to toggle between process and thread view. The thread view is critical for identifying which specific thread inside a Java or PostgreSQL process is generating I/O.
The DISK READ and DISK WRITE columns show actual block device I/O. The SWAPIN column shows time waiting for swap reads, which is distinct from general I/O wait. If SWAPIN is nonzero for application processes, you have a memory pressure problem that is manifesting as I/O.
For PostgreSQL-specific diagnosis: PostgreSQL uses separate background worker processes. iotop will show `postgres: bgwriter` and `postgres: walwriter` and `postgres: checkpointer` as distinct entries. If checkpointer is dominating I/O, adjust `checkpoint_completion_target` (increase toward 0.9) and consider `max_wal_size`. If walwriter dominates, your synchronous_commit setting or fsync behavior is the issue.
# Batch mode, 5 samples, 2s interval, only show processes with I/O
iotop -b -n 5 -d 2 -o
# Log to file for post-incident analysis
iotop -b -n 60 -d 5 -o >> /var/log/iotop-$(date +%Y%m%d).log &
# iotop-ng with cgroup filtering (systemd services)
iotop-c --cgroup /system.slice/postgresql.service
# Find PID of the worst offender from a single snapshot
iotop -b -n 1 -o | sort -k 4 -rn | head -5blktrace: Block Layer Request Tracing
blktrace is the correct tool when you need to understand I/O patterns at the request level: sequential vs random, read/write mix, request size distribution, and where in the I/O stack latency is being introduced. It requires debugfs mounted and the `blktrace` package installed.
blktrace records events using a set of single-letter action codes. The critical ones: Q (queued to the scheduler), G (get request struct), M (merged with existing request), D (issued to device driver), C (completed by driver). The time between Q and D is scheduler latency. The time between D and C is device service time. If D-to-C is high, the device is slow. If Q-to-D is high, the scheduler is holding requests.
Running blktrace directly produces binary per-CPU files. Always use it with a time limit or you will fill a filesystem fast. On a busy NVMe under load, blktrace can generate 200MB/s of trace data. Use `-w` to set a duration in seconds.
After capture, `blkparse` converts binary to human-readable text. The `btt` (blktrace time analysis) tool post-processes blkparse output into statistical summaries including Q2D (queue to dispatch), D2C (dispatch to complete), and Q2C (total latency). This is the output you actually want for latency profiling.
# Trace device nvme0n1 for 30 seconds, write to /tmp/trace
blktrace -d /dev/nvme0n1 -w 30 -o /tmp/trace
# Parse binary output to human-readable (creates /tmp/trace.blktrace.0 etc)
blkparse -i /tmp/trace -o /tmp/trace.parsed
# Generate latency statistics with btt
btt -i /tmp/trace.parsed
# One-liner: trace, parse, and summarize in pipeline
blktrace -d /dev/sda -w 10 -o - | blkparse -i - -o - | btt -i -
# Check if debugfs is mounted (required)
mount | grep debugfs
# If not: mount -t debugfs debugfs /sys/kernel/debugReading blkparse and btt Output
blkparse output is dense. Each line shows: device, CPU, sequence number, timestamp, PID, action code, direction (R/W), offset, and size. The offset tells you whether I/O is sequential or random. If consecutive D events show wildly different LBA offsets, you have random I/O. Sequential I/O shows monotonically increasing offsets.
btt produces several output sections. The most useful is the per-device latency summary table. Look for the Q2C row - this is total round-trip time from the moment the process issued the I/O request to completion. On a healthy NVMe under moderate load, Q2C should be under 200 microseconds. On a spinning disk doing random I/O, 10-15ms is normal. If Q2C is high, decompose it: if D2C is high, the device is the bottleneck. If Q2D is high, the I/O scheduler or kernel overhead is the bottleneck.
btt also outputs a seek distance histogram and a request size histogram. The request size histogram is particularly valuable. If 90% of your requests are 4K, you have a small-block random I/O pattern. If requests are 128K or larger, your application is doing large sequential I/O or the page cache is effectively merging requests before they hit the block layer.
On our test PostgreSQL server, btt showed Q2D averaging 180 microseconds with D2C at 85 microseconds. The queue-to-dispatch time being twice the actual device service time pointed us to `mq-deadline` scheduler overhead. Switching to `none` scheduler (appropriate for NVMe with internal queuing) reduced Q2D to under 20 microseconds.
# Switch I/O scheduler for NVMe (none is appropriate for modern NVMe)
echo none > /sys/block/nvme0n1/queue/scheduler
# Make permanent via udev rule
cat /etc/udev/rules.d/60-ioscheduler.rules
# ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# ACTION=="add|change", KERNEL=="sd[a-z]*", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"
# Verify current scheduler
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq
# Generate per-process I/O breakdown from blktrace
blkparse -i /tmp/trace -d /tmp/trace.bin
btt -i /tmp/trace.bin -p /tmp/trace.procsCombining All Three Tools: A Real Workflow
The effective workflow is layered. Start with iostat to confirm there is a device-level anomaly. Then use iotop to identify which process or service is responsible. Then use blktrace to characterize the I/O pattern and find where latency accumulates.
In our PostgreSQL investigation, the sequence was: iostat showed nvme0n1 await spiking to 40-50ms during application response time degradation. iotop identified the checkpointer process generating 180 MB/s of writes. blktrace + btt showed 4K write requests with high Q2D time, pointing to scheduler overhead. The fix was a combination of scheduler change and PostgreSQL checkpoint tuning.
For systems where you are doing this investigation repeatedly, consider automating the data collection. Teams using DevOps automation platforms like taskbotshub.ai can wire these tools into triggered collection workflows - for example, automatically capturing a 60-second blktrace when a Prometheus alert fires on disk latency, then attaching the btt summary to the incident ticket. This removes the human reaction time from trace collection and captures the issue while it is still happening.
For databases specifically, also check `pg_stat_bgwriter` in PostgreSQL or equivalent engine-level I/O stats alongside these system tools. System-level tools show what the kernel sees; the database has its own buffer pool and its view of I/O differs from what reaches the block layer.
# Three-terminal workflow for active investigation
# Terminal 1: iostat watching the device
watch -n 2 'iostat -x nvme0n1 1 1 | grep nvme0n1'
# Terminal 2: iotop finding the process
iotop -b -d 2 -o -k | tee /tmp/iotop-live.log
# Terminal 3: blktrace when you have identified the suspect timeframe
blktrace -d /dev/nvme0n1 -w 30 -o /tmp/incident-trace
# Then analyze
btt -i /tmp/incident-trace.blktrace.0 | head -40BPF Tools as Complement to the Classic Three
biolatency and biotop from BCC (BPF Compiler Collection) or bpftrace one-liners give you histogram-based latency data that neither iostat nor blktrace provides out of the box. On kernel 5.8+ with BTF enabled, you can run these without kernel headers.
biolatency shows a histogram of I/O completion latencies in microseconds and milliseconds. It answers 'what is my p99 I/O latency' in a way that iostat's average `await` cannot. A device might show 2ms average await while p99 is 200ms - a bimodal distribution that averages hide completely.
biotop is effectively iotop implemented in BPF, with lower overhead and better accuracy for short-lived processes that iotop might miss between polling intervals.
These tools complement but do not replace blktrace. blktrace captures every individual request. BPF tools aggregate into histograms in-kernel, reducing overhead but losing per-request detail. For routine monitoring, BPF tools are better. For deep pattern analysis, blktrace is necessary.
# biolatency: histogram of block I/O latency (requires bcc-tools)
biolatency -d nvme0n1 10
# biotop: top-like per-process block I/O
biotop 2 5
# bpftrace one-liner for I/O latency histogram
bpftrace -e '
kprobe:blk_account_io_start { @start[arg0] = nsecs; }
kprobe:blk_account_io_done /@start[arg0]/ {
@usecs = hist((nsecs - @start[arg0]) / 1000);
delete(@start[arg0]);
}'
# Install bcc on RHEL 9 / Rocky 9
dnf install bcc-tools
export PATH=$PATH:/usr/share/bcc/toolsPersistent I/O Monitoring and Alerting Setup
For production systems, point-in-time tools are insufficient. You need time-series data to correlate I/O spikes with application events.
sar from the sysstat package is the standard solution. Enable the sysstat collection daemon, configure the interval in `/etc/cron.d/sysstat` or the systemd timer, and use `sadc` to write binary data files. Default collection is every 10 minutes; for I/O-intensive systems we recommend every 1-2 minutes. Retention defaults to 28 days. Analyze with `sar -d` for device stats and `sar -b` for block I/O summary.
For Prometheus-based monitoring, the node_exporter exposes all /proc/diskstats metrics under the `node_disk_*` namespace. Key metrics to alert on: `node_disk_io_time_weighted_seconds_total` (reflects queue depth over time - alert if rate exceeds 0.9), and `node_disk_read_time_seconds_total` / `node_disk_writes_merged_total`. Build a Grafana panel with the rate of `node_disk_io_time_weighted_seconds_total` divided by the scrape interval to get a normalized utilization percentage comparable to iostat's `%util`.
If you manage multiple servers and want to standardize your monitoring stack naming or project organization, clear naming conventions matter at scale - teams we have talked to that used services like nicename.me for their internal tooling and dashboard naming have found it easier to maintain consistency across hundreds of monitored hosts. That might seem like a small detail, but grep-able, consistent naming in Prometheus labels and Grafana folder structures saves real time during incidents.
Set retention policy intentionally. sar data in /var/log/sa compresses well but still grows. On a server with 20 block devices sampled every 2 minutes, expect roughly 50MB/day before compression.
# Enable sysstat collection (systemd)
systemctl enable --now sysstat.service
systemctl enable --now sysstat-collect.timer
systemctl enable --now sysstat-summary.timer
# Configure 2-minute collection interval
cat /etc/sysstat/sysstat
# HISTORY=28
# COMPRESSAFTER=10
# SADC_OPTIONS="-S DISK"
# Change systemd timer to 2-minute intervals
systemctl edit sysstat-collect.timer
# [Timer]
# OnCalendar=*:0/2
# Replay I/O stats from yesterday at 14:00-15:00
sar -d -f /var/log/sa/sa$(date -d yesterday +%d) -s 14:00:00 -e 15:00:00
# Prometheus: query for disk utilization equivalent to %util
# rate(node_disk_io_time_seconds_total{device="nvme0n1"}[5m])Common Misdiagnoses and How to Avoid Them
The most common mistake we see is treating high %util on NVMe as proof of saturation. As noted, NVMe devices with multi-queue support handle parallel I/O internally. A NVMe at 100% util might be processing 400K IOPS perfectly. Look at await and aqu-sz instead.
The second common mistake is ignoring the filesystem layer. All three tools measure at or below the filesystem. If your problem is metadata operations - millions of small file creates, inode lookups, directory scans - the block I/O you see may be modest while the application stalls on filesystem locks. For filesystem-level insight, use `strace -p PID -e trace=openat,read,write,fsync,stat` to catch where a process spends time before I/O even reaches the block layer.
Third: confusing buffered and direct I/O. iostat and blktrace see only I/O that actually hits the block layer. If your application writes buffered data that the kernel holds in the page cache and writes later, iotop will show the write I/O attributed to the kernel's `kworker` or `jbd2` threads, not to your application. Use `iotop`'s accumulated mode (`a` key) and watch over a longer window, or check `/proc/PID/io` directly which accounts for bytes written to the page cache before they flush.
# Read /proc/PID/io directly to see all I/O including buffered writes
cat /proc/$(pgrep -n postgres)/io
# rchar: bytes read (includes cache hits)
# wchar: bytes written (includes buffered, not yet flushed)
# read_bytes: bytes read from block device
# write_bytes: bytes written to block device (actual disk I/O)
# cancelled_write_bytes: writes cancelled due to truncation
# Watch a specific process's block-level I/O over time
watch -n 2 'cat /proc/$(pgrep -n java)/io | grep _bytes'
# Check if application uses O_DIRECT (bypasses page cache)
strace -p $(pgrep -n postgres) -e trace=openat 2>&1 | grep -E '(O_DIRECT|O_SYNC)'