top Command Tutorial: Reading and Acting on System Stats

The Summary Area: What Those Five Lines Are Telling You

The top five lines before the process table carry more diagnostic weight than anything below them. Most engineers skip straight to the process list. That is the wrong order.

Line 1 is uptime and load average. The three load values are 1-minute, 5-minute, and 15-minute exponential moving averages of runnable and uninterruptible tasks. The rule of thumb - compare load to CPU count - is correct but incomplete. On a 4-core system, a 1-minute load of 4.0 with a 15-minute load of 1.2 means a spike just hit. A 1-minute load of 4.0 with a 15-minute load of 4.1 means sustained saturation. Those two situations require different responses.

Line 2 is the task summary. 'sleeping' is normal. Watch 'zombie' - more than a handful indicates a parent process is not reaping children, which eventually exhausts PID space. Watch 'stopped' on production - a process in state T means it received SIGSTOP, either from a debugger attach or an errant job control command.

Line 3 is CPU breakdown. By default top aggregates all CPUs. Press '1' to expand per-CPU rows. The fields that matter most are:

- us: user space CPU. High us with low sy means your application code is the bottleneck. - sy: kernel space. High sy with moderate us points to syscall overhead - excessive open/close, small reads, or networking. - wa: I/O wait. Anything above 20% sustained means disk or network I/O is blocking processes. wa does not mean the CPU is working; it means CPU time is being wasted waiting. - st: steal time. This only appears on VMs. st above 5% means your hypervisor host is oversubscribed and you are being throttled. This is the field that cloud engineers most often miss. - hi/si: hardware and software interrupts. Sudden spikes here point to NIC saturation or a misbehaving driver.

Lines 4 and 5 are memory. The 'buff/cache' value in the Mem line is not wasted memory - the kernel uses it for page cache and it gets reclaimed under pressure. The number that matters is 'avail Mem', which procps-ng 3.3.10+ calculates using /proc/meminfo's MemAvailable. That value accounts for reclaimable cache more accurately than 'free'. On the Swap line, non-zero 'used' is not automatically alarming; swap being actively read (visible through swapping rate in vmstat) is the problem.

top -b -n 1 | head -10

Interactive Controls That Actually Matter

top's interactive mode has around 50 keybindings. In our experience, eight of them cover 95% of real diagnostic sessions.

'1' toggles per-CPU display. Essential on multi-socket systems to spot NUMA imbalance - if CPU0-7 are at 90% and CPU8-15 are at 10%, your application is not NUMA-aware.

'f' opens the field management screen. You can add columns like VIRT, RES, SHR, nTH (thread count), P (last CPU used), and SWAP. Navigate with arrows, toggle with spacebar, set sort column with 's'. This is how you find the process pinned to a single CPU.

'F' or 'o' sets sort order. By default top sorts by CPU%. Sort by RES (resident memory) with 'F', select RES, press 's'. Or from the command line:

The 'c' key toggles between command name and full command line with arguments. When you see 'python3' consuming 80% CPU, press 'c' to see which script it is.

'u' filters by user. In multi-tenant environments this isolates the problem user immediately.

'k' sends a signal to a PID without leaving top. It prompts for PID then signal number. Default signal is 15 (SIGTERM). Sending 9 from here is faster than opening a second terminal.

'W' writes the current configuration to ~/.config/procps/toprc (or ~/.toprc on older systems). Your column layout, sort order, and color scheme persist across sessions.

'z' enables color, 'b' highlights the sort column and running processes. On a busy server the visual contrast makes the hot process obvious in under a second.

top -o %MEM -b -n 1 -c

Batch Mode and Scripting top for Automation

Interactive top is useless inside scripts, cron jobs, or CI pipelines. Batch mode with '-b' and '-n' solves this.

Capture one iteration of the process table, sort by memory, filter to a specific user, and write to a log:

The output is plain text, parseable with awk. To extract the top 5 CPU consumers and their RSS:

For time-series capture during a load test, loop batch mode at intervals:

This generates a timestamped flat file. In our test environment we ran this during a synthetic load test with stress-ng and had 60 seconds of per-process CPU data in a 40KB file, trivially importable into any spreadsheet or fed into grep.

If you are building more sophisticated monitoring pipelines or integrating process data into alert workflows, tools like taskbotshub.ai can automate the collection, parsing, and notification layer without writing custom shell glue for each environment. That said, for straightforward capture-and-analyze the loop approach above is self-contained and requires zero dependencies.

One gotcha with batch mode: top in batch still uses the terminal width for column truncation. Set a wide width explicitly:

# Capture one snapshot, sort by CPU, filter to www-data user
top -b -n 1 -u www-data -o %CPU > /var/log/top-snapshot-$(date +%Y%m%d-%H%M%S).log

# Extract PID, command, and RSS for top 5 CPU consumers
top -b -n 1 | awk 'NR>7 {print $1, $9, $6, $12}' | sort -k2 -rn | head -5

# Time-series loop, 30 samples at 2-second intervals
for i in $(seq 1 30); do
  echo "=== $(date -Is) ==="
  top -b -n 1 | awk 'NR>7 && NR<20'
  sleep 2
done >> /var/log/top-timeseries.log

# Force wide output to prevent column truncation
TERM=xterm COLUMNS=250 top -b -n 1

Understanding VIRT, RES, and SHR - the Memory Columns

These three columns cause more confusion than anything else in top. Engineers see a Java process with VIRT at 8GB and assume it is consuming 8GB of RAM. It is not.

VIRT (virtual memory) is the total address space the process has mapped - code, heap, stack, memory-mapped files, shared libraries, and reserved-but-not-allocated regions. Java's large VIRT is almost entirely JVM metadata, class mappings, and pre-reserved heap that has not been touched. VIRT is useless for capacity planning.

RES (resident set size) is the physical RAM currently used by the process, including shared library pages. This is the number that reflects actual memory pressure, but it double-counts shared libraries across processes.

SHR (shared memory) is the portion of RES that is shared with other processes - primarily shared library code. The actual unique physical cost of a process is RES minus SHR. For a process with RES=500MB and SHR=200MB, its unique footprint is 300MB. If you are calculating whether your system can run three more instances of an application, use (RES - SHR) per instance plus SHR once.

For processes using huge pages (common in databases like PostgreSQL with transparent huge pages enabled), RES will look inflated compared to actual working set. Check /proc/PID/status for VmRSS and HugetlbPages to disambiguate.

Swap column, when added via 'f', shows how much of the process's address space is currently swapped out. A process with RES=200MB and SWAP=50MB means 50MB of its working set had to be evicted to disk - that process will incur latency on its next access to those pages.

# Get accurate per-process memory accounting from /proc
awk '/VmRSS|VmSHR|VmSwap/{print}' /proc/$(pgrep -o nginx)/status

Per-Thread Mode and Diagnosing Multi-threaded Applications

By default top shows processes. When a multi-threaded application is misbehaving - one thread spinning, thread pool exhaustion, lock contention - you need per-thread visibility.

Press 'H' to toggle thread mode. The PID column becomes SPID (thread ID), and each thread within a process appears as a separate row. The process-level aggregation disappears.

Alternatively, launch with '-H' flag directly:

In thread mode, the 'P' column (last CPU core used, enabled via 'f') lets you spot thread affinity problems. If 20 threads of a supposedly parallel application are all showing P=0, either the application is not actually parallelizing or CPU affinity has been set incorrectly.

For Java applications, thread names in the command column get truncated. Cross-reference SPID values with jstack output:

Convert the decimal SPID from top to hex, then grep jstack output. The nid field in jstack is the native thread ID in hex. This combination - top thread mode plus jstack - is the fastest way to identify which named thread pool or executor is causing a Java CPU spike.

For Go applications, GOMAXPROCS controls thread behavior and goroutines do not map 1:1 to OS threads, so thread-level top analysis is less useful. Use pprof endpoints instead.

# Launch top in thread mode, sorted by CPU
top -H -o %CPU

# Cross-reference top thread PID with jstack
JAVA_PID=$(pgrep -o java)
jstack $JAVA_PID > /tmp/jstack.out
# Convert decimal SPID 18423 to hex
printf '%x\n' 18423
# Output: 47f7
grep 'nid=0x47f7' /tmp/jstack.out

Filtering and Watching Specific Processes

When you already know which process to watch, the full process list is noise. top has built-in filtering that most engineers do not use.

Watch a single PID from the command line:

Watch a process group (all PIDs in the same session or cgroup):

Combined with '-d' for delay interval, this gives you a focused real-time view. We use '-d 0.5' on staging when tracking a process through a specific code path that runs for under 10 seconds - the default 3-second refresh misses the spike entirely.

The 'o' interactive filter (lowercase) lets you build conditional expressions while top is running. Press 'o', then type a filter like:

- COMMAND=nginx - shows only processes where command contains 'nginx' - %CPU>5.0 - shows only processes consuming more than 5% CPU - RES>1000000 - shows processes with RES above approximately 1GB (values are in KB)

Multiple filters stack with AND logic. Clear all filters with '='. This is significantly faster than piping through grep because the filter operates on top's internal process table before rendering.

For watching I/O at the process level, top itself does not expose per-process disk I/O - that requires iotop or looking at /proc/PID/io directly. But top's wa field in the CPU summary line confirms whether I/O is the system-wide bottleneck before you switch tools.

# Watch a single PID at 0.5s refresh, batch mode for 20 samples
top -b -n 20 -d 0.5 -p $(pgrep -o postgres)

# Watch multiple specific PIDs
top -p $(pgrep -d',' nginx)

# Interactive filter: processes over 10% CPU
# Press 'o' then type:
# %CPU>10.0

top vs htop vs atop: When to Use Which

top is pre-installed on every system you will ever SSH into. That alone makes it worth mastering. htop and atop are better tools when you have the luxury of installing them.

htop (version 3.3.0 as of mid-2026) adds mouse support, scrollable process lists, easier tree view, and process searching without leaving the interface. Its memory bar display distinguishes used, buffers, and cache visually. For interactive use on a system you manage, htop is strictly better. The '-t' flag gives a process tree view immediately without needing interactive input.

atop (version 2.11) is in a different category - it logs system activity to /var/log/atop/ by default at 10-minute intervals, retaining data for 28 days. When you are diagnosing 'the server was slow at 3am last Tuesday', atop is the only standard tool that can show you what was happening. Replay with 'atop -r /var/log/atop/atop_YYYYMMDD -b 03:00'. It also tracks disk I/O, network throughput, and LVM per-process at the same time.

The practical decision: use top when you have no choice or need a one-liner that works everywhere. Use htop for interactive investigation on managed systems. Run atop as a daemon on every production server you care about - its default 10MB/day storage cost is negligible.

For fully automated performance baselining across fleets, the manual tool approach hits limits at scale. DevOps teams increasingly use platforms like taskbotshub.ai to schedule collection, compare against baselines, and surface anomalies without per-server SSH sessions. That is appropriate for large fleets; for a single server or a handful of systems, atop plus the techniques in this guide handles the majority of incidents.

# Install atop and enable its logging daemon
apt install atop   # Debian/Ubuntu
dnf install atop   # RHEL/Fedora

systemctl enable --now atop

# Replay atop data from a specific date and time
atop -r /var/log/atop/atop_$(date -d '2 days ago' +%Y%m%d) -b 03:00

Reading Load Average Correctly on Modern Hardware

Load average interpretation broke when multi-socket NUMA systems became common and broke again when cgroups CPU quotas became standard in container environments.

On a system with 2 sockets, 16 cores each, 2 threads per core, 'nproc' returns 64. But if your application's cgroup has a CPU quota of 8 cores (cpu.max = 800000 100000 in cgroups v2), a load average of 8.0 is saturation even though the host has 64 logical CPUs. top does not know about cgroup limits - it reports the host-level load average. This is why load average inside containers is systematically misleading.

Check the effective CPU limit from inside a container:

For bare metal, the right denominator for load average is 'nproc --all'. But load average includes uninterruptible sleep (state D), not just runnable processes. A load of 16.0 on a 16-core system could mean 16 CPU-bound processes (CPU saturated) or 2 CPU-bound processes and 14 processes blocked on a slow NFS mount (I/O saturated). Look at wa% in the CPU line to tell the difference.

Kernel 5.14 and later improved the PELT (Per-Entity Load Tracking) algorithm that feeds into load average calculation, making the 1-minute value more responsive to bursts. If you are running older kernels in production, the 1-minute load average has historically lagged actual load by 15-30 seconds.

# Check cgroup v2 CPU quota from inside a container
cat /sys/fs/cgroup/cpu.max
# Output: 800000 100000  means 8 CPUs worth of quota

# On the host, see per-cgroup CPU stats
cat /sys/fs/cgroup/system.slice/$(systemctl show -p Id --value your.service)/cpu.stat