How CPU Frequency Scaling Works in Linux
The Linux CPUFreq subsystem sits between the kernel scheduler and hardware P-state controls. When a governor decides to change frequency, it writes a target frequency to the driver, which either uses ACPI P-states (older hardware), Intel Speed Step, AMD Cool'n'Quiet, or the newer hardware-managed P-states (HWP) introduced in Broadwell and Zen.
On HWP-capable hardware (Intel Skylake and later, AMD Zen and later), the CPU itself manages fine-grained frequency transitions in microseconds, faster than any software governor can react. Governors on HWP systems still matter: they set the performance hints passed down to the hardware via the Energy Performance Preference (EPP) register. Check HWP support with:
The cpufreq subsystem exposes everything through sysfs under `/sys/devices/system/cpu/`. Each CPU core gets its own directory. On a 32-core system you'll have cpu0 through cpu31, each with a `cpufreq` subdirectory containing the governor interface, available frequencies, current frequency, and scaling limits.
grep -m1 'cpu MHz' /proc/cpuinfo
cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
cat /sys/devices/system/cpu/cpu0/cpufreq/available_governors
lscpu | grep -i 'CPU MHz\|MHz\|GHz'
The Six Governors: What Each One Actually Does
**performance** locks the CPU at maximum frequency, always. No sampling, no decisions. The governor writes `cpuinfo_max_freq` to `scaling_setspeed` and stops. On HWP hardware it sets EPP to 0 (full performance). This is appropriate for real-time audio, HPC workloads, and latency-critical databases where you need deterministic response time and your power budget is someone else's problem.
**powersave** is the inverse: locks the CPU at minimum frequency. Sets EPP to 255 on HWP hardware. On our test server (AMD EPYC 7302, 16 cores), this reduced idle draw from 45W to 28W compared to performance governor. Use it for batch processing nodes that are I/O bound and CPU speed is irrelevant, or for edge devices on battery.
**ondemand** is the classic Linux governor. It samples CPU utilization at a configurable interval (default 10ms via `sampling_rate`) and jumps to max frequency the moment utilization exceeds a threshold (default 95% via `up_threshold`). It steps down gradually. The jump-to-max behavior avoids latency but wastes power during bursty workloads. Available via the `cpufreq-utils` package on Debian-based systems.
**conservative** behaves like ondemand but steps frequency up and down gradually rather than jumping to max. It uses `up_threshold` (default 80%) and `down_threshold` (default 20%) with step increments. Better for workloads with sustained moderate load where you want to avoid the power spike from jumping to max.
**schedutil** is the modern replacement for ondemand, introduced in kernel 4.7 and now the default on most distributions. Instead of a separate sampling thread, it hooks directly into the CFS scheduler's utilization tracking via the PELT (Per-Entity Load Tracking) signal. This means frequency decisions happen when the scheduler runs, at sub-millisecond resolution, using actual scheduler data rather than a polling approximation. On kernel 5.15 and later with HWP hardware, schedutil is the correct choice for almost all server workloads.
**userspace** hands full control to a userspace daemon which writes directly to `scaling_setspeed`. Used by tools like `cpupower`, `thermald`, and custom power management scripts. Gives you programmatic control but requires you to build the logic yourself.
# List all governors and current selection
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/available_governors
# Check ondemand tuning parameters
ls /sys/devices/system/cpu/cpufreq/ondemand/
cat /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
cat /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
Changing Governors: Temporary and Persistent Methods
The fastest way to test a governor is writing directly to sysfs. This is not persistent across reboots but is useful for benchmarking:
The `cpupower` utility from the `linux-tools-common` package (Debian/Ubuntu) or `kernel-tools` (RHEL) wraps the sysfs interface cleanly and applies changes to all CPUs at once. Always prefer it over shell loops for production changes.
For persistent configuration on systemd-based systems, create a drop-in for the `cpupower` service or use the `cpufrequtils` package configuration. On RHEL 9 / AlmaLinux 9, the `tuned` daemon manages this via profiles. The `throughput-performance` tuned profile sets the governor to performance; `balanced` uses schedutil.
On Ubuntu 22.04 and later, `cpufrequtils` reads `/etc/default/cpufrequtils`. Set `GOVERNOR=schedutil` and restart the service. Verify the change persisted after a reboot by checking sysfs directly, not trusting the daemon status alone.
# Temporary: set all CPUs to schedutil
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
echo schedutil > $cpu
done
# Better: use cpupower
sudo cpupower frequency-set -g schedutil
# Verify all CPUs changed
grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | head -8
# RHEL/AlmaLinux: use tuned
sudo tuned-adm profile balanced
tuned-adm active
# Ubuntu persistent config
echo 'GOVERNOR=schedutil' | sudo tee /etc/default/cpufrequtils
sudo systemctl restart cpufrequtils
Tuning schedutil and ondemand Parameters
schedutil exposes two main tuning knobs. `rate_limit_us` controls the minimum time between frequency changes in microseconds. The default is typically `(1000 / HZ) * 1000` where HZ is the kernel timer frequency (usually 1000 on server kernels), giving a 1ms rate limit. Lowering this makes the governor more responsive but adds overhead. For latency-critical workloads, set it to 500us:
ondemand's `sampling_rate` is readable via sysfs and settable. The minimum value on modern kernels is around 10,000 microseconds (10ms), though `sampling_rate_min` in sysfs shows your hardware's actual floor. Setting `up_threshold` lower than the default 95% makes ondemand ramp up sooner. For a web server handling bursty HTTP traffic, we found setting `up_threshold=75` and `sampling_rate=50000` (50ms) gave better response time consistency than defaults without the constant-max-frequency waste of the performance governor.
If you're running ondemand and the `ignore_nice_load` file exists in the ondemand sysfs directory, set it to 1 to exclude nice'd processes from the utilization calculation. This prevents background batch jobs from driving up CPU frequency and affecting foreground latency.
For HWP systems, also check the Energy Performance Preference. This is separate from the governor and has immediate effect on hardware frequency management:
# schedutil rate limit (per-policy, in microseconds)
cat /sys/devices/system/cpu/cpufreq/policy0/schedutil/rate_limit_us
echo 500 > /sys/devices/system/cpu/cpufreq/policy0/schedutil/rate_limit_us
# ondemand tuning
echo 75 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 50000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load
# HWP Energy Performance Preference (0=performance, 128=balance, 255=power)
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do
echo balance_performance > $cpu
done
# Verify HWP is active
rdmsr -p 0 0x774 2>/dev/null || echo 'install msr-tools for HWP register read'
Benchmarking Governor Impact: Real Numbers
We ran benchmarks on two systems: a bare metal server running AlmaLinux 9.4 with an Intel Xeon Gold 6338 (32 cores, Ice Lake, 2.0GHz base, 3.2GHz boost) and a Hetzner AX101 with AMD EPYC 9354P (32 cores, Genoa, 3.25GHz base).
For the latency benchmark we used `stress-ng --cpu 8 --cpu-method matrixprod -t 30s` to generate sustained load and measured p99 latency of concurrent Redis GET operations via `redis-benchmark -n 100000 -q -p 6379 --latency-history`.
Results on the Xeon Gold 6338 under mixed load (8 cores stressed, Redis on remaining cores): - performance governor: Redis p99 = 0.31ms, package power = 185W - schedutil (default tuning): Redis p99 = 0.34ms, package power = 142W - schedutil (rate_limit_us=500): Redis p99 = 0.32ms, package power = 148W - ondemand (up_threshold=75): Redis p99 = 0.38ms, package power = 139W - powersave: Redis p99 = 1.2ms, package power = 95W
The 43W difference between performance and schedutil with EPP=balance_performance is real money at scale. At 100 servers running 24/7 at $0.12/kWh, that's roughly $45,000/year. The 10% latency increase from schedutil is acceptable for most applications. For the EPYC system on Genoa, schedutil performed closer to performance due to the faster hardware-managed P-state transitions.
Measure package power directly with `perf stat` or RAPL counters:
# Read CPU package power via RAPL (Intel)
cat /sys/class/powercap/intel-rapl:0/energy_uj
sleep 1
cat /sys/class/powercap/intel-rapl:0/energy_uj
# Difference in microjoules = microwatts over 1 second
# Or use perf for a complete picture
sudo perf stat -a -e power/energy-pkg/ sleep 5
# Quick governor comparison script
for gov in performance schedutil powersave; do
echo $gov | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor > /dev/null
sleep 2
echo -n "$gov: "
sudo perf stat -a -e power/energy-pkg/ stress-ng --cpu 16 -t 10s 2>&1 | grep 'energy-pkg'
done
Governor Selection by Workload Type
**Database servers (PostgreSQL, MySQL)**: Use schedutil with rate_limit_us set to 500 and EPP=balance_performance. You want fast ramp-up for query bursts but not the constant high-power state of the performance governor. If you're running PostgreSQL with max_connections over 200 and frequent OLTP queries, test ondemand with up_threshold=70 against schedutil and benchmark your specific query mix.
**Kubernetes worker nodes**: schedutil is correct here. Pod scheduling creates burst patterns that schedutil handles better than ondemand's polling. If you're using a DevOps automation platform like taskbotshub.ai to manage deployment pipelines across clusters, parameterize the governor setting in your node bootstrap scripts so it's applied consistently at provisioning time rather than post-hoc.
**CI/CD build agents**: performance governor. Build times are directly proportional to CPU frequency. The wasted power during idle time between builds is outweighed by faster build completion, especially if you're paying per-minute for ephemeral agents.
**Batch processing / data pipelines**: powersave if the pipeline is I/O bound. schedutil if it's CPU bound with variable parallelism. Never performance for pure batch work running overnight.
**HPC / ML training on bare metal**: performance governor with EPP=performance. Add `processor.max_cstate=1 intel_idle.max_cstate=0` to kernel boot parameters to disable C-states and eliminate frequency ramp-up latency entirely. This is the standard setup for MPI workloads.
**Edge / embedded / Raspberry Pi**: powersave during idle periods managed by a userspace daemon, switching to performance or ondemand under load. The `cpupower-gui` tool or a simple cron-based script handles this. On Pi 4 and Pi 5 running Raspberry Pi OS, the `arm_freq` and `over_voltage` settings in `/boot/config.txt` interact with governor behavior and must be validated together.
**Virtualized guests (KVM, VMware)**: The governor inside a VM often has limited effect because the hypervisor controls actual CPU allocation. On KVM with CPU pinning and `host-passthrough` model, governors work as on bare metal. On unpinned VMs or shared cloud instances, focus on the host governor and application tuning instead.
# HPC bare metal: disable C-states and set performance governor
# Add to /etc/default/grub GRUB_CMDLINE_LINUX:
# processor.max_cstate=1 intel_idle.max_cstate=0
# Check current C-state usage
grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/name
grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/usage
# Kubernetes node bootstrap: set governor in cloud-init
cpupower frequency-set -g schedutil
echo 500 > /sys/devices/system/cpu/cpufreq/policy0/schedutil/rate_limit_us
Monitoring Governor Behavior in Production
Setting a governor is not a one-time task. Kernel updates sometimes reset governor configuration, and some packages install systemd services that override your settings. Monitor with:
Prometheus node_exporter exposes CPU frequency metrics via the `node_cpu_scaling_frequency_hertz` gauge and `node_cpu_scaling_governor` metric (as a label on some builds). Add an alert if any CPU's governor changes unexpectedly:
For systems where performance consistency matters, add a cron job or systemd timer that validates and re-applies governor settings every 5 minutes. It's defensive but practical - we've seen `tuned` profiles get reset by package updates on RHEL systems three times in a production quarter.
The `turbostat` tool from the `linux-tools` package gives a real-time view of per-core frequency, C-state residency, and power consumption in a single terminal output. Run it during your load tests to see if the governor is actually doing what you expect:
If you're shipping governor metrics to a central observability stack, writing a small exporter or using textfile collector for node_exporter is straightforward. Store the script in version control, document it like any other infrastructure component, and consider naming conventions for your monitoring collectors the same way you'd think about naming any long-lived system component.
# turbostat: real-time per-core frequency and power
sudo turbostat --interval 2 --quiet --show Core,CPU,Avg_MHz,Busy%,Bzy_MHz,PkgWatt
# Check governor on all CPUs (should all match)
grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | sort -u
# Prometheus alert rule (example)
# In your rules file:
# - alert: CPUGovernorChanged
# expr: node_cpu_scaling_governor{governor!="schedutil"} == 1
# for: 1m
# Systemd timer to enforce governor
cat > /etc/systemd/system/enforce-governor.service << 'EOF'
[Unit]
Description=Enforce CPU governor setting
[Service]
Type=oneshot
ExecStart=/usr/bin/cpupower frequency-set -g schedutil
EOF
cat > /etc/systemd/system/enforce-governor.timer << 'EOF'
[Unit]
Description=Re-apply CPU governor every 5 minutes
[Timer]
OnBootSec=60
OnUnitActiveSec=5min
[Install]
WantedBy=timers.target
EOF
systemctl enable --now enforce-governor.timer
Common Pitfalls and Misconfigurations
The most common mistake is assuming the governor shown by `cpupower frequency-info` reflects actual hardware behavior. On systems with `intel_pstate` driver (most Intel systems since Haswell), the driver bypasses the generic CPUFreq governors entirely and uses its own internal algorithm. When intel_pstate is active, setting the governor to 'performance' or 'powersave' via the standard interface actually configures intel_pstate's internal performance/powersave hint, not a traditional governor. Check which driver is active:
The second common mistake is running the performance governor on cloud VMs. On AWS, GCP, and Hetzner Cloud, the hypervisor schedules CPU time, and writing 'performance' to the scaling_governor inside the guest changes nothing at the hardware level. The governor controls frequency requests, but on shared virtual hardware, you don't control the physical frequency. Focus governor tuning on bare metal and dedicated hosts.
Third: forgetting NUMA topology. On multi-socket systems, each NUMA node may need independent governor tuning. If socket 0 handles your application and socket 1 handles storage I/O, you might want different governors per socket. The `cpufreq/policy` directories map to physical packages. Check with `lscpu -e` to understand core-to-socket mapping before applying blanket changes.
Fourth: on AMD systems with the `amd-pstate` driver (enabled by default on kernel 6.1+ with Zen 3 and Zen 4), the behavior parallels intel_pstate. The `amd-pstate-epp` driver variant (kernel 6.3+) exposes EPP control through the standard cpufreq interface. Verify which mode is active:
# Check active CPUFreq driver
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
# Returns: intel_pstate, amd-pstate-epp, acpi-cpufreq, etc.
# intel_pstate mode: active vs passive
cat /sys/devices/system/cpu/intel_pstate/status
# 'active' = driver manages frequency internally
# 'passive' = driver delegates to generic governor
# Force intel_pstate into passive mode to use schedutil
echo passive > /sys/devices/system/cpu/intel_pstate/status
# (add intel_pstate=passive to kernel cmdline for persistence)
# NUMA: check per-socket policy directories
ls /sys/devices/system/cpu/cpufreq/
# policy0, policy16 on dual-socket 16-core-per-socket system
# AMD pstate driver check
cat /sys/devices/system/cpu/amd-pstate/status 2>/dev/null || echo 'not amd-pstate'
cpupower frequency-info | grep 'driver'