cgroups v1 vs v2: What Actually Changed
cgroups v1 allowed each resource controller - cpu, memory, blkio, net_cls - to have its own independent hierarchy. A process could be in /sys/fs/cgroup/cpu/app and /sys/fs/cgroup/memory/db simultaneously, which made consistent accounting nearly impossible. Coordination between controllers was a known design failure.
cgroups v2 uses a single unified tree. Every process lives at exactly one node in the hierarchy, and all controllers you attach to that node act on the same group. This makes accounting correct: when you see memory.current for a cgroup, it accounts for every thread in that group, regardless of which controller you are looking at.
To verify which mode your kernel is running, check the mount table:
mount | grep cgroup
# v2 only:
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
# Mixed v1+v2 (hybrid, older distros):
# tmpfs on /sys/fs/cgroup type tmpfs ...
# cgroup on /sys/fs/cgroup/memory type cgroup (rw,...,memory)The cgroupfs Interface: Direct Kernel Control
Everything under /sys/fs/cgroup is a pseudo-filesystem exposed by the kernel. Creating a cgroup is as simple as mkdir. Deleting it is rmdir. No daemons required. This is where you learn the actual semantics before letting systemd abstract them.
Let us create a cgroup named 'batch' under the root, enable the memory and CPU controllers, move a process into it, and set a 512 MB memory hard limit:
# Create the cgroup directory
mkdir /sys/fs/cgroup/batch
# Enable memory and cpu controllers (must be done from parent)
echo '+memory +cpu' > /sys/fs/cgroup/cgroup.subtree_control
# Move shell's own PID into the group (or any PID)
echo $$ > /sys/fs/cgroup/batch/cgroup.procs
# Set 512 MB hard memory limit
echo $((512 * 1024 * 1024)) > /sys/fs/cgroup/batch/memory.max
# Set a soft limit (kernel tries to reclaim before OOM)
echo $((384 * 1024 * 1024)) > /sys/fs/cgroup/batch/memory.high
# Verify current usage
cat /sys/fs/cgroup/batch/memory.currentCPU Throttling with cpu.max
In cgroups v2, CPU quota is expressed as a quota/period pair in the cpu.max file. The default is 'max 100000', meaning no limit. To allow a cgroup to use at most 2 full CPU-equivalents on an 8-core machine, write '200000 100000' - 200ms of CPU time per 100ms period.
This is the same mechanism Kubernetes uses for CPU limits. When a container hits its cpu.max quota, the kernel scheduler throttles it until the next period. You can see throttle events in cpu.stat:
# Allow 200% CPU (2 cores equivalent) out of 8 available
echo '200000 100000' > /sys/fs/cgroup/batch/cpu.max
# Check throttle statistics after running a load
cat /sys/fs/cgroup/batch/cpu.stat
# nr_periods 8523
# nr_throttled 1204
# throttled_usec 48291033
# nr_bursts 0
# burst_usec 0
# CPU weight (nice-equivalent, replaces cpu.shares)
# Default is 100, range is 1-10000
echo 50 > /sys/fs/cgroup/batch/cpu.weightI/O Control with io.max and io.weight
Block I/O control in cgroups v2 uses the io controller, which replaces the old blkio controller. You reference devices by their major:minor number. Get that from lsblk or stat.
We tested this on an NVMe drive capable of 3.5 GB/s reads. Capping a backup job to 200 MB/s write bandwidth dropped its impact on concurrent database workload from measurable latency spikes to baseline noise:
# Find device major:minor
lsblk -o NAME,MAJ:MIN
# nvme0n1 259:0
# Set max read/write BPS and IOPS for device 259:0
echo '259:0 rbps=524288000 wbps=209715200 riops=10000 wiops=5000' \
> /sys/fs/cgroup/batch/io.max
# Proportional weight (100 = default, range 1-10000)
echo '259:0 100' > /sys/fs/cgroup/batch/io.weight
# Read current I/O stats
cat /sys/fs/cgroup/batch/io.stat
# 259:0 rbytes=0 wbytes=4194304 rios=0 wios=4 dbytes=0 dios=0systemd Slice and Scope Integration
Manually writing to cgroupfs works for testing, but in production systemd owns the cgroup tree. Every systemd unit lives in a slice: system.slice for system services, user.slice for user sessions, machine.slice for containers and VMs. You do not need to touch /sys/fs/cgroup directly for most production scenarios.
To constrain an existing service, use systemctl set-property. Changes are persistent across reboots when you use --runtime=false (the default). The example below limits the nginx service to 1 CPU core and 256 MB RAM:
# Set CPU and memory limits on a running service (persistent)
systemctl set-property nginx.service CPUQuota=100% MemoryMax=256M
# Equivalent unit file directives in /etc/systemd/system/nginx.service.d/limits.conf
[Service]
CPUQuota=100%
CPUWeight=80
MemoryMax=256M
MemoryHigh=200M
IOWeight=50
# Apply without full restart
systemctl daemon-reload
systemctl restart nginx.service
# Inspect cgroup placement and limits
systemctl status nginx.service | grep -E 'CGroup|Memory|CPU'
cat /sys/fs/cgroup/system.slice/nginx.service/memory.maxCreating Custom Slices for Application Tiers
For multi-tier applications - web, API, workers, batch - isolating them into separate slices lets you apply proportional resource guarantees. If your web tier and batch processing share the same system.slice, a runaway batch job can starve web requests. Separate slices prevent this at the kernel scheduler level.
Create /etc/systemd/system/app-web.slice and /etc/systemd/system/app-batch.slice, then assign services to them with the Slice= directive:
# /etc/systemd/system/app-web.slice
[Unit]
Description=Web Application Slice
Before=slices.target
[Slice]
CPUWeight=800
MemoryHigh=4G
MemoryMax=6G
---
# /etc/systemd/system/app-batch.slice
[Unit]
Description=Batch Processing Slice
Before=slices.target
[Slice]
CPUWeight=100
MemoryHigh=1G
MemoryMax=2G
IOWeight=20
---
# In your worker service unit:
[Service]
Slice=app-batch.slice
ExecStart=/usr/bin/python3 /opt/worker/main.py
# Reload and verify the tree
systemctl daemon-reload
systemctl start app-batch.slice
systemd-cgls /sys/fs/cgroup/app-batch.sliceMemory OOM Behavior and oom_kill_policy
When a cgroup hits memory.max, the kernel OOM killer fires within that cgroup first before touching the rest of the system. This is a critical safety property of v2 - a misbehaving service gets killed, not a random other process. You can tune which process within the cgroup gets selected via /proc/PID/oom_score_adj, range -1000 to 1000.
In our experience, setting oom_score_adj to 500 on worker processes and -200 on the main service process ensures the workers die first if the group runs out of memory. The main process stays up and can restart workers cleanly.
You can also enable memory.oom.group so that when OOM fires, the entire cgroup is killed together rather than individual processes, which prevents partial-state crashes in multi-process apps:
# Make OOM kill the entire cgroup atomically
echo 1 > /sys/fs/cgroup/app-batch.slice/memory.oom.group
# Bias OOM selection: protect main process (-200), sacrifice workers (+500)
pid_main=12345
pid_worker=12346
echo -200 > /proc/$pid_main/oom_score_adj
echo 500 > /proc/$pid_worker/oom_score_adj
# Monitor OOM events without waiting for logs
# memory.events updates atomically
watch -n 1 cat /sys/fs/cgroup/app-batch.slice/memory.events
# anon 0
# file 0
# kernel_stack 0
# pagetables 0
# sec_pagetables 0
# shmem 0
# file_mapped 0
# oom 0
# oom_kill 0Monitoring cgroup Metrics in Production
The kernel exposes per-cgroup pressure stall information (PSI) via memory.pressure, cpu.pressure, and io.pressure. PSI measures the fraction of time tasks are stalled waiting for a resource. A cpu.pressure value of 'some avg10=15.00' means 15% of the time in the last 10 seconds, at least one task in that cgroup was runnable but not running.
Prometheus users can scrape these via node_exporter with the --collector.cgroups flag (available from node_exporter 1.7.0). For teams running automated remediation workflows, tools like taskbotshub.ai can consume these PSI metrics via webhook and trigger scaling or alerting actions without manual intervention.
For quick CLI inspection without Prometheus:
# PSI for a systemd service
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.pressure
# some avg10=0.00 avg60=0.12 avg300=0.08 total=4821034
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0
cat /sys/fs/cgroup/system.slice/nginx.service/memory.pressure
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0
# Aggregate stats for a slice
cat /sys/fs/cgroup/app-batch.slice/memory.stat | grep -E '^(anon|file|slab) '
# List all cgroups consuming more than 100MB
find /sys/fs/cgroup -name memory.current 2>/dev/null \
| xargs -I{} awk -v f={} '$1>104857600{print f, $1/1024/1024 "MB"}' {}Delegating cgroup Control to Unprivileged Users
Delegation lets a non-root process manage its own sub-tree. This is how rootless containers work in Podman and how user-space container runtimes avoid needing setuid binaries. Systemd sets up delegation automatically when you use User= and Delegate=yes in a service unit.
For manual delegation, the owner of the cgroup directory can write to the control files within that subtree. The key rule: you can only enable controllers in a child that are already enabled in the parent's cgroup.subtree_control.
# systemd unit with delegation for rootless container runtime
[Service]
User=containeruser
Delegate=yes
DelegateSubgroup=init
ExecStart=/usr/bin/podman run --rm myapp:latest
# Verify delegation attributes
cat /sys/fs/cgroup/user.slice/user-1001.slice/user@1001.service/cgroup.delegate
# 1
cat /sys/fs/cgroup/user.slice/user-1001.slice/user@1001.service/cgroup.controllers
# cpuset cpu io memory pids
# A delegated process can then write to its own subtree
# mkdir /sys/fs/cgroup/user.slice/.../myapp
# echo $$ > /sys/fs/cgroup/user.slice/.../myapp/cgroup.procscpuset Controller: NUMA and Core Pinning
On NUMA machines with multiple CPU sockets or when running latency-sensitive workloads, pinning a cgroup to specific cores and memory nodes eliminates cross-NUMA memory access penalties. The cpuset controller handles this.
On a 2-socket server with cores 0-31 on NUMA node 0 and cores 32-63 on NUMA node 1, pinning a database to node 0 and a batch job to node 1 cuts inter-socket memory latency out of the critical path entirely. We measured 12-18% throughput improvement on PostgreSQL 16 under OLTP load by doing this on a dual-socket EPYC 9654 system.
# Enable cpuset controller (must be enabled at root first)
echo '+cpuset' > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/db-tier
echo '+cpuset' > /sys/fs/cgroup/db-tier/cgroup.subtree_control 2>/dev/null || true
# Pin to cores 0-15 and NUMA node 0
echo '0-15' > /sys/fs/cgroup/db-tier/cpuset.cpus
echo '0' > /sys/fs/cgroup/db-tier/cpuset.mems
# Verify effective assignment (accounts for hotplug)
cat /sys/fs/cgroup/db-tier/cpuset.cpus.effective
# 0-15
# Move postgres main process
pg_pid=$(pgrep -o postgres)
echo $pg_pid > /sys/fs/cgroup/db-tier/cgroup.procs
# For systemd integration
systemctl set-property postgresql.service AllowedCPUs=0-15 AllowedMemoryNodes=0Troubleshooting Common cgroup Problems
The most common issue we see is writing to cgroup.subtree_control before the cgroup has a controller listed in cgroup.controllers. If a controller is not in the parent's subtree_control, it will not appear in the child's controllers at all.
A second common failure is trying to rmdir a cgroup that still has processes. Use cgroup.procs to verify it is empty before removal. Attempting to remove a non-empty cgroup returns EBUSY.
A third failure mode specific to systemd environments: directly writing to /sys/fs/cgroup paths for systemd-managed units works but systemd may overwrite your values on service restart. Use systemctl set-property or unit file directives for persistence.
# Debug: why can't I enable a controller?
cat /sys/fs/cgroup/batch/cgroup.controllers
# cpu memory <-- io is missing
# It must be enabled in the parent first
cat /sys/fs/cgroup/cgroup.subtree_control
# cpu memory <-- io not here
echo '+io' > /sys/fs/cgroup/cgroup.subtree_control
echo '+io' > /sys/fs/cgroup/batch/cgroup.subtree_control
# Debug: EBUSY on rmdir
cat /sys/fs/cgroup/batch/cgroup.procs
# 4821 <-- still has processes
cat /sys/fs/cgroup/batch/cgroup.threads
# Move processes out first
echo 4821 > /sys/fs/cgroup/cgroup.procs
rmdir /sys/fs/cgroup/batch
# Check if systemd owns a cgroup
systemd-cgls --all | grep batch
cgroupstatus /sys/fs/cgroup/batch 2>/dev/null || systemctl status --property=ControlGroup --value yourservice