What Triggers the OOM Killer
The OOM killer fires when the kernel fails to allocate a memory page after exhausting page reclaim. The path is: allocation request -> try direct reclaim -> if reclaim fails, check if overcommit allows waiting -> if not, call out_of_memory() in mm/oom_kill.c.
Overcommit policy controls whether the kernel even reaches this point. Three modes exist, set via /proc/sys/vm/overcommit_memory:
0 (default): heuristic overcommit. The kernel allows most allocations but refuses obviously impossible ones. 1: always overcommit. Every malloc() succeeds regardless of available memory. OOM killer fires more frequently because physical limits are hit at runtime, not at allocation time. 2: strict overcommit. Total committed memory cannot exceed swap + a fraction of RAM defined by overcommit_ratio (default 50) or overcommit_kbytes. Allocations fail cleanly with ENOMEM rather than succeeding and dying later.
On our test server running kernel 6.8, mode 2 with overcommit_ratio=80 eliminated OOM kills entirely for a Java workload - at the cost of some allocation failures that the application had to handle. For most production systems, mode 0 is correct. Mode 1 is only appropriate if you have verified that your application stack handles ENOMEM gracefully everywhere it matters.
cat /proc/sys/vm/overcommit_memory
sysctl vm.overcommit_memory
sysctl vm.overcommit_ratio
# Show current committed memory vs limit
grep -E 'Committed_AS|CommitLimit' /proc/meminfoHow the OOM Score Is Calculated
Every process has an oom_score visible at /proc/PID/oom_score, ranging from 0 to 1000. Higher score means more likely to be killed. The kernel calculates it in oom_badness() using two components: a points value based on the process's proportional memory use, and an adjustment from oom_score_adj.
The base calculation is roughly: (rss + pagetable + swap usage) / total_memory * 1000. A process consuming 10% of RAM gets approximately 100 base points. The adjustment from /proc/PID/oom_score_adj is added directly - it ranges from -1000 to +1000. Setting oom_score_adj to -1000 makes the process immune; the kernel skips it entirely. Setting it to +1000 makes it the preferred kill target.
Children inherit their parent's oom_score_adj value at fork time unless they explicitly change it. This matters for daemons that spawn workers - all workers start with the same adj value as the parent. You can verify what the kernel actually computed by reading oom_score after the process is running.
# Check score and adjustment for a process
pid=$(pgrep postgres | head -1)
echo "PID: $pid"
cat /proc/$pid/oom_score
cat /proc/$pid/oom_score_adj
# List top 10 processes by OOM score
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
score=$(cat /proc/$pid/oom_score 2>/dev/null)
comm=$(cat /proc/$pid/comm 2>/dev/null)
echo "$score $pid $comm"
done | sort -rn | head -10Protecting Critical Processes with oom_score_adj
Setting oom_score_adj to -1000 makes a process immune from OOM killing. The kernel explicitly checks for this value in oom_badness() and returns LONG_MIN, excluding it from selection. This is how systemd protects itself - it sets its own oom_score_adj to -1000 at startup.
For a PostgreSQL primary, you typically want to protect the postmaster process. When postmaster dies, all backends die with it. Setting oom_score_adj to -900 gives it strong protection while still allowing the kernel to kill it if literally nothing else is available. We prefer -900 over -1000 for databases because a database process consuming all available memory is genuinely a problem you want addressed - just not first.
For Redis, we typically use -500. Redis can tolerate a restart and reload from RDB/AOF much more gracefully than PostgreSQL.
The adjustment survives exec() but not across a new process tree. If your service manager restarts the process, the new instance inherits the systemd service's oom_score_adj setting, not the one you wrote manually to /proc.
# Protect a running process (write directly)
echo -900 > /proc/$(pgrep -x postgres)/oom_score_adj
# Set via systemd unit (persistent across restarts)
# /etc/systemd/system/postgresql.service.d/oom.conf
[Service]
OOMScoreAdjust=-900
# Apply without full restart
systemctl daemon-reload
systemctl restart postgresql
# Verify
cat /proc/$(pgrep -x postgres)/oom_score_adjSystemd OOMPolicy and the oom_killer_disable Knob
Since systemd 253, the OOMPolicy= directive controls what systemd does when the OOM killer terminates a process in a service. Three values: continue (default, service keeps running if other processes survive), stop (systemd stops the entire service unit), and kill (systemd kills all remaining processes in the service's cgroup).
For a multi-process service like a web server with workers, continue is wrong - a killed worker leaves the service in a degraded state. Use stop or kill, then rely on systemd's Restart= to bring it back cleanly.
The kernel also exposes /proc/sys/vm/oom_kill_allocating_task. Default is 0, meaning the OOM killer picks the highest-scoring process even if it is not the one that triggered the allocation. Setting it to 1 kills the task that actually ran out of memory. This is useful for debugging but counterproductive in production - the allocating task is often a small utility, not the actual memory hog.
# /etc/systemd/system/nginx.service.d/oom.conf
[Service]
OOMPolicy=stop
Restart=always
RestartSec=2s
OOMScoreAdjust=200
# Check oom_kill_allocating_task
cat /proc/sys/vm/oom_kill_allocating_task
# Temporarily kill the allocating task (debug only)
echo 1 > /proc/sys/vm/oom_kill_allocating_taskcgroup v2 Memory Limits as OOM Prevention
The most effective OOM killer tuning is preventing runaway processes from consuming all memory in the first place. cgroup v2 memory.max is a hard limit - when a process in the cgroup hits it, the kernel tries reclaim within the cgroup, and if that fails, kills a process inside that cgroup rather than going system-wide.
This is the key architectural insight: cgroup-level OOM events are contained. A Java application hitting its memory.max limit triggers an OOM kill scoped to that cgroup. Your PostgreSQL instance on the same host is not affected.
memory.high is the soft limit. When usage exceeds memory.high, the kernel throttles allocation speed for that cgroup and applies memory pressure to encourage reclaim. This gives the process time to release memory before hitting the hard wall. Set memory.high to roughly 80-90% of memory.max.
memory.swap.max controls swap usage within the cgroup. Setting it to 0 disables swap for that cgroup entirely, forcing OOM events to happen sooner but making memory behavior more predictable.
On our test server running kernel 6.8 with systemd 256, every containerized service runs with these limits set via systemd's MemoryMax= and MemoryHigh= directives. OOM events that used to take down unrelated services are now contained to the offending unit.
# Systemd unit with cgroup memory limits
# /etc/systemd/system/myapp.service
[Service]
MemoryMax=4G
MemoryHigh=3500M
MemorySwapMax=0
OOMScoreAdjust=300
OOMPolicy=stop
Restart=on-failure
# Check live cgroup memory stats
systemctl status myapp
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max
cat /sys/fs/cgroup/system.slice/myapp.service/memory.eventsReading OOM Kill Events in the Kernel Log
When the OOM killer fires, the kernel logs a detailed report. Knowing how to read it quickly is essential during an incident. The log includes the triggering process, a snapshot of all processes with their memory usage, and the selected victim.
The key fields in the process table are: 'oom_score_adj' shows the adjustment at time of kill, 'anon-rss' is anonymous RSS in kilobytes, 'file-rss' is file-backed RSS, 'shmem-rss' is shared memory RSS. The line starting with 'Killed process' shows who died and how much memory was reclaimed.
The 'oom_score_adj' column in the kernel log and the value you read from /proc/PID/oom_score_adj match. Use this to verify your adjustments took effect before an incident, not during one.
For automated alerting, parsing dmesg or /var/log/kern.log for 'Out of memory' or 'Killed process' is the standard approach. If you are running DevOps automation pipelines, tools like those available at taskbotshub.ai can be used to trigger remediation workflows when OOM events are detected in log streams.
# View recent OOM events
dmesg -T | grep -E 'Out of memory|Killed process|oom_kill'
# On systemd systems
journalctl -k | grep -E 'Out of memory|Killed process|oom_score'
# Get full OOM report from last boot
journalctl -k -b -1 | grep -A 50 'Out of memory'
# Watch for OOM events in real time
dmesg -w | grep --line-buffered -E 'Out of memory|Killed process'vm.panic_on_oom and When to Use It
Setting /proc/sys/vm/panic_on_oom to 1 causes the kernel to panic and reboot instead of invoking the OOM killer. This is appropriate for systems where killing any process would leave the system in an unusable or inconsistent state - certain real-time systems, some HPC configurations, or cluster nodes where you would rather trigger a failover than limp along with killed services.
Value 2 panics even if oom_cpuset or oom_mempolicy would have allowed killing only a subset of processes. For most general-purpose servers, leaving this at 0 is correct.
A middle path: set panic_on_oom to 0 but configure the OOM killer to kill aggressively with vm.oom_kill_allocating_task=0 and ensure your process supervision (systemd, runit, s6) restarts killed services quickly. This recovers more gracefully than a full reboot while still clearing the memory pressure.
# Current setting
cat /proc/sys/vm/panic_on_oom
# Enable panic on OOM (HPC/cluster nodes only)
sysctl vm.panic_on_oom=1
# Make persistent
echo 'vm.panic_on_oom=1' > /etc/sysctl.d/99-oom.conf
sysctl --system
# Verify all relevant OOM settings
sysctl vm.panic_on_oom vm.overcommit_memory vm.overcommit_ratio vm.oom_kill_allocating_taskPer-Process Tuning Reference for Common Services
Here are the oom_score_adj values we use in production, with reasoning for each. These are starting points - adjust based on your actual service priorities.
PostgreSQL postmaster: -900. Killing postmaster kills all backends. Recovery is slow. Protect it aggressively.
Redis (AOF enabled): -500. Can reload from disk. Brief downtime is tolerable.
Nginx worker processes: +200. They are stateless and restart fast. Sacrificing them to protect databases is the right trade.
Java application servers: +100 to +300 depending on restart time and state. Stateless microservices get +300. Anything with session state gets +100.
sshd listener: -900. Losing sshd access to a memory-pressured server makes recovery much harder. Protect it the same as your database.
Batch jobs and cron tasks: +500 to +800. These are the intended first victims during memory pressure.
Prometheus and monitoring agents: -200. Killing your monitoring during an OOM event is exactly when you need it most.
For any service, the first question is: how long does restart take, and what state is lost? Services with long restart times and significant state loss get negative adjustments. Stateless, fast-restarting services get positive adjustments.
# Apply a full production baseline via script
set_oom_adj() {
local name=$1
local adj=$2
for pid in $(pgrep -x "$name"); do
echo $adj > /proc/$pid/oom_score_adj
echo "Set $name ($pid) oom_score_adj=$adj"
done
}
set_oom_adj postgres -900
set_oom_adj redis-server -500
set_oom_adj sshd -900
set_oom_adj prometheus -200
set_oom_adj nginx 200earlyoom and nohang: Userspace OOM Daemons
The kernel OOM killer only fires when memory is truly exhausted. By that point, the system is already in a degraded state - swap thrashing, unresponsive shells, minutes of latency on every operation. Userspace OOM daemons act earlier, before the kernel reaches the limit.
earlyoom monitors free memory and swap in a configurable loop (default 1 second). When free memory drops below a threshold (default 10% RAM and 10% swap), it sends SIGTERM to the highest oom_score process, then SIGKILL if it does not exit within a timeout. On kernel 6.8, earlyoom 1.8.2 reliably prevents the kernel from reaching its OOM state entirely on our test server.
nohang is a similar tool with more configuration options, including the ability to set per-process thresholds and actions. It supports ignoring specific processes, sending custom signals, and running scripts on OOM events.
Both tools respect oom_score_adj values - processes with lower adjusted scores are spared. The interaction is clean: tune your oom_score_adj settings for the kernel OOM killer, and earlyoom will honor the same relative priorities.
# Install earlyoom (Debian/Ubuntu)
apt install earlyoom
# Configuration: /etc/default/earlyoom
EARLYOOM_ARGS="-m 5 -s 10 --avoid '(^|/)(init|Xorg|ssh)$' --prefer '(^|/)(java|python)$'"
# Enable and start
systemctl enable --now earlyoom
# Watch its decisions
journalctl -fu earlyoom
# nohang installation from source
git clone https://github.com/hakavlad/nohang
cd nohang && sudo make install
systemctl enable --now nohangDiagnosing OOM Issues Before They Happen
Reactive OOM tuning is the wrong approach. The right workflow is: establish memory baselines, identify which processes would be killed under pressure, adjust before an incident.
The oom_score for all processes can be scraped and graphed. On systems running Prometheus with node_exporter, the process_oom_score metric is available from node_exporter's --collector.processes flag (disabled by default). Enable it and alert when any protected process has an oom_score above a threshold.
For capacity planning, /proc/meminfo's MemAvailable field is the most accurate indicator of how much memory can actually be freed for new allocations. It accounts for reclaimable caches and buffers. When MemAvailable drops below 500MB on a system with 32GB RAM, you are approaching danger territory. Set your alerting threshold there, not at MemFree.
The vm.min_free_kbytes setting controls how much memory the kernel reserves and never allocates. Default is usually 67MB on a 32GB system. Increasing this gives the kernel more room to operate under pressure. We set it to 256MB on database servers.
# Enable process OOM score collection in node_exporter
# /etc/default/prometheus-node-exporter
ARGS="--collector.processes"
# Check MemAvailable trend
watch -n 5 'grep -E "MemTotal|MemFree|MemAvailable|SwapFree" /proc/meminfo'
# Set min_free_kbytes
sysctl vm.min_free_kbytes=262144
echo 'vm.min_free_kbytes=262144' >> /etc/sysctl.d/99-memory.conf
# Quick script: show all processes sorted by OOM score with adj
ps aux --sort=-%mem | head -20 | awk '{print $2}' | xargs -I{} sh -c \
'echo -n "PID {} score="; cat /proc/{}/oom_score 2>/dev/null; echo -n " adj="; cat /proc/{}/oom_score_adj 2>/dev/null; echo -n " cmd="; cat /proc/{}/comm 2>/dev/null'