How Linux Represents Processes

Every process on Linux has a PID (process ID), a PPID (parent PID), and a set of file descriptors, memory mappings, and scheduling attributes exposed under /proc//. The kernel scheduler tracks processes in one of five main states: R (running or runnable), S (interruptible sleep), D (uninterruptible sleep - usually blocked on I/O), Z (zombie), and T (stopped by signal).

Zombie processes are already dead but their parent has not called wait() to collect the exit status. They consume a PID slot but no memory or CPU. If you see hundreds of zombies, the parent process has a bug - killing the zombies directly does nothing. Kill the parent or fix its signal handling. Uninterruptible D-state processes are the dangerous ones: you cannot kill them, they are waiting on a kernel resource, usually a hung NFS mount or a failing disk.

The /proc filesystem is the authoritative source. Everything ps and top display comes from reading files in /proc. Understanding this matters when ps output looks wrong - check /proc//status directly to verify.

cat /proc/$(pgrep -o nginx)/status | grep -E 'Name|Pid|State|VmRSS|Threads'

ps: The Right Flags for Real Work

The default `ps` output with no flags shows only processes in your current terminal session. That is almost never what you want. The two formats you need to know are BSD style (no hyphens) and UNIX style (with hyphens). On Linux both work, but they do not mix cleanly.

`ps aux` is BSD style: a = all users, u = user-oriented format, x = include processes without a controlling terminal. This gives you USER, PID, %CPU, %MEM, VSZ, RSS, TTY, STAT, START, TIME, COMMAND. VSZ is virtual memory size in KB. RSS is resident set size - actual physical RAM in use. For memory troubleshooting, RSS is your number.

`ps -ef` is UNIX style: -e = every process, -f = full format. Adds PPID and STIME, which is useful for tracing process trees. Neither aux nor -ef shows thread count by default. Add -L for threads: `ps -eLf` lists every thread as a separate line with LWP (light weight process) ID.

Custom output format with -o is where ps becomes surgical. You pick exactly the columns you need.

# Show PID, PPID, RSS in MB, CPU%, state, and full command for all processes
ps -eo pid,ppid,rss,pcpu,stat,comm --sort=-rss | head -20

# Find all child processes of nginx master
ps --ppid $(pgrep -o nginx) -o pid,stat,rss,comm

# Processes in uninterruptible sleep (potential I/O hang)
ps aux | awk '$8 ~ /^D/ {print $0}'

# Show process start time and elapsed time
ps -eo pid,lstart,etime,comm | grep postgres

Filtering and Searching with ps and pgrep

Piping ps to grep works but has one persistent annoyance: grep matches its own process in the output. The classic workaround `grep [n]ginx` uses a character class to prevent the pattern from matching itself. A cleaner solution is pgrep, which was built for this.

pgrep returns PIDs only by default. Add -l to include the process name, -a to include the full command line, and -f to match against the full command line instead of just the process name. The -f flag is essential when you have multiple instances of the same binary running with different arguments - for example, multiple Python workers each running a different script.

pgrep also supports -u for user filtering and -P for filtering by parent PID. For scripts, pgep's exit code is useful: it returns 0 if at least one matching process exists, 1 if none found. Use this in health check scripts instead of parsing ps output.

On systems with many Java or Python processes where the process name is just `java` or `python3`, always use `pgrep -af` to match on arguments.

# Match on full command line, useful for Python/Java workers
pgrep -af 'celery worker -Q high_priority'

# Find processes by user running more than 10 threads
ps -u appuser -L -o pid,lwp,comm | awk 'NR>1{count[$1]++} END{for(p in count) if(count[p]>10) print p, count[p]}'

# List all unique process names consuming more than 100MB RSS
ps -eo rss,comm | awk '$1 > 102400 {print $2}' | sort -u

# Check if a service is running in a script (exit code approach)
if pgrep -x haproxy > /dev/null; then echo "haproxy running"; fi
// advertisement

kill and Signal Handling

The name `kill` is misleading. It sends signals to processes, and most signals are not fatal. There are 31 standard signals plus real-time signals. The ones you use operationally are a short list.

SIGTERM (15) is the default when you run `kill `. It asks the process to terminate gracefully. Well-written daemons catch SIGTERM, finish in-flight requests, flush buffers, and exit. SIGKILL (9) cannot be caught or ignored - the kernel terminates the process immediately. Never send SIGKILL as your first move on a production process. You will leave incomplete writes, locked files, and corrupt state.

SIGHUP (1) is traditionally used to tell a daemon to reload its configuration without restarting. nginx, sshd, and rsyslog all support this. SIGUSR1 and SIGUSR2 are application-defined - nginx uses SIGUSR1 to reopen log files, PostgreSQL uses SIGUSR1 to trigger a checkpoint. Check the application documentation before sending these.

SIGSTOP (19) pauses a process - it cannot be caught. SIGCONT (18) resumes it. This is useful for temporarily pausing a CPU-heavy batch job without killing it.

killall matches by name. pkill matches by name and supports the same -f, -u, -P flags as pgrep. For scripted process management, pkill is more reliable than parsing ps and calling kill.

# Graceful shutdown, then force if needed after 10 seconds
kill -TERM $(pgrep gunicorn) && sleep 10 && kill -KILL $(pgrep gunicorn) 2>/dev/null

# Reload nginx config without dropping connections
kill -HUP $(cat /var/run/nginx.pid)

# Pause and resume a CPU-intensive process
kill -STOP $(pgrep ffmpeg)
kill -CONT $(pgrep ffmpeg)

# Send SIGUSR1 to all workers of a specific user
pkill -USR1 -u worker_user python3

# List all signals
kill -l

top and htop: Interactive Monitoring

The `top` command from procps-ng updates every 3 seconds by default. The header shows load averages (1, 5, 15 minutes), total tasks, CPU breakdown, and memory stats. The load average numbers mean different things depending on CPU count - a load of 8.0 on a 16-core system is fine; the same number on a single-core system means every process is waiting.

The CPU line splits into: us (user space), sy (kernel), ni (nice), id (idle), wa (I/O wait), hi (hardware IRQ), si (software IRQ), st (stolen, relevant in VMs). High wa means I/O is the bottleneck. High sy means kernel activity - check for excessive system calls, interrupts, or context switching. High st in a VM means your hypervisor host is overloaded.

Interactive commands inside top worth memorizing: `M` sorts by memory, `P` sorts by CPU, `T` sorts by cumulative CPU time (finds long-running hogs), `k` prompts to kill a process by PID, `r` renice, `1` toggles per-CPU breakdown, `H` toggles thread view, `u` filters by user, `V` shows process tree view.

For batch output from top - useful in scripts and cron jobs - use `top -bn1`. The -b flag is batch mode, -n1 runs one iteration.

# Single snapshot from top for scripting
top -bn1 | head -20

# Watch top output for a specific PID
top -p $(pgrep -o postgres)

# Top sorted by memory, showing only first 15 processes
top -bn1 -o %MEM | head -22

# htop with tree view and highlight for a specific user (if htop installed)
htop -u appuser --tree

Process Priority: nice and renice

Linux scheduling priority runs from -20 (highest priority) to 19 (lowest). The default nice value is 0. Only root can set negative nice values. Regular users can only increase the nice value (lower priority) for their own processes.

Use `nice` to start a process with a specific priority. Use `renice` to change the priority of a running process. For batch jobs, backups, and compilation tasks that run alongside production workloads, setting nice 15-19 prevents them from starving real traffic.

The relationship between nice value and actual CPU scheduling is not linear - it is logarithmic. The difference between nice 0 and nice 5 is large; the difference between nice 15 and nice 19 is small. In practice, setting a background job to nice 10 is usually enough to keep it from interfering with foreground processes on a loaded system.

For I/O priority, nice does not help - you need ionice. Class 3 (idle) means the process only gets I/O when no other process needs it. This is the right setting for backup jobs.

# Start a compression job at low priority
nice -n 15 tar czf /backup/archive.tar.gz /var/data/

# Renice a running process
renice -n 10 -p $(pgrep -o mysqldump)

# Set idle I/O class for a backup process
ionice -c 3 -p $(pgrep rsync)

# Start a job at low CPU and I/O priority simultaneously
nice -n 19 ionice -c 3 restic backup /var/data --repo /mnt/backup
// advertisement

Process Groups, Sessions, and Job Control

Every process belongs to a process group. Process groups receive signals together - when you press Ctrl+C in a terminal, SIGINT goes to the entire foreground process group, not just the shell. This is why a shell pipeline like `cat bigfile | gzip | split` gets interrupted cleanly when you hit Ctrl+C: all three processes are in the same process group.

Sessions are collections of process groups tied to a controlling terminal. When the terminal closes, SIGHUP goes to the session leader (your shell), which propagates to the process group. This is why processes started in SSH sessions die when the connection drops - unless you use nohup, screen, tmux, or systemd-run.

`nohup` redirects stdout and stderr to nohup.out and sets SIGHUP to be ignored. It does not daemonize the process. For proper daemonization, use systemd: `systemd-run --unit=myjob --uid=appuser /usr/bin/myapp`. This creates a transient systemd unit that survives terminal exit and gets proper cgroup accounting.

Job control with `bg`, `fg`, and `jobs` is useful interactively but do not rely on it in scripts. Background jobs in scripts can produce unexpected behavior when the script exits.

# Detach a running process from terminal with systemd
systemd-run --unit=import-job --uid=importer \
  --setenv=DB_HOST=10.0.1.5 \
  /usr/local/bin/import_data.py --source /mnt/data

# Check its status and follow logs
systemctl status import-job.service
journalctl -u import-job.service -f

# Send signal to entire process group
kill -TERM -$(ps -o pgid= -p $(pgrep gunicorn) | head -1 | tr -d ' ')

Automating Process Management

Manual process monitoring does not scale past a handful of servers. For fleet-scale process management, you need either systemd service definitions with RestartPolicy, or an external monitoring layer.

Systemd handles process lifecycle better than any hand-rolled script. Define the process as a service unit with Restart=on-failure, RestartSec=5, and StartLimitIntervalSec=60. If your process dies more than StartLimitBurst times within StartLimitIntervalSec seconds, systemd stops trying and alerts. Combine with systemd watchdog (WatchdogSec=) for processes that support sd_notify.

For teams building DevOps pipelines that need to monitor process state across multiple hosts and trigger automated responses - auto-scaling, alerting, runbook execution - platforms like taskbotshub.ai provide AI-driven automation that can integrate with Linux process events through webhook triggers and scheduled checks, reducing the toil of writing and maintaining custom monitoring scripts.

For local process supervision without systemd (containers, minimal systems), consider s6-overlay or runit. Both are simpler than systemd and have predictable behavior in PID 1 environments.

On the scripting side, always use PID files carefully. A stale PID file from a crashed process will break your restart logic. Check that the PID in the file actually corresponds to the expected process before deciding a service is running.

# Check if PID file is stale
PIDFILE=/var/run/myapp.pid
if [ -f "$PIDFILE" ]; then
  PID=$(cat "$PIDFILE")
  if ! kill -0 "$PID" 2>/dev/null; then
    echo "Stale PID file, removing"
    rm -f "$PIDFILE"
  elif ! grep -q myapp /proc/"$PID"/comm 2>/dev/null; then
    echo "PID $PID is not myapp, removing stale file"
    rm -f "$PIDFILE"
  fi
fi

# Minimal systemd service with watchdog and restart limits
# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target postgresql.service

[Service]
User=appuser
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3
WatchdogSec=30

[Install]
WantedBy=multi-user.target

Diagnosing Specific Problems: OOM, Zombies, and Stuck Processes

The OOM killer (Out Of Memory killer) terminates processes when the kernel cannot allocate memory. It logs to dmesg. When a production process gets OOM-killed unexpectedly, check `dmesg | grep -i 'oom\|killed process'` and `journalctl -k | grep -i oom`. The OOM killer scores processes using /proc//oom_score - higher score means more likely to be killed. You can protect critical processes by setting oom_score_adj to -1000.

For zombie cleanup, find the parent: `ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/ {print}'`. Then send SIGCHLD to the parent (`kill -CHLD `) to prompt it to collect the zombie. If that fails, the parent process itself is broken - restart it.

For processes stuck in D state: verify the mount points with `df -h` and check `cat /proc//wchan` to see which kernel function the process is waiting in. Common culprits are nfs, nfsd, and blk_wait_io. Try umounting the stuck filesystem forcibly with `umount -l` (lazy unmount) to detach it from the namespace while letting existing file handles drain.

High voluntary context switches (check /proc//status, lines VCS and NVCS) indicate a process frequently yielding the CPU - typical of I/O-bound or lock-contended code. High involuntary context switches mean the process is being preempted, suggesting CPU contention.

# Protect a critical process from OOM killer
echo -1000 > /proc/$(pgrep -o haproxy)/oom_score_adj

# Find zombies and their parents
ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/ {print "zombie:", $1, "parent:", $2, $4}'

# Check what kernel function a D-state process is blocked on
cat /proc/$(pgrep -o nfsd)/wchan

# Context switch stats for a process
grep -E 'voluntary_ctxt|nonvoluntary_ctxt' /proc/$(pgrep -o myapp)/status

# Review recent OOM kills
dmesg --ctime | grep -A5 'Out of memory'
// advertisement