How sysctl Works
sysctl reads and writes to the /proc/sys virtual filesystem. Every file under /proc/sys maps directly to a kernel parameter. The dotted notation sysctl uses is just the directory path with slashes replaced by dots: net.ipv4.tcp_rmem corresponds to /proc/sys/net/ipv4/tcp_rmem.
You can read any parameter directly with cat, which is useful when you want to inspect a value inside a script without spawning a sysctl process:
``` cat /proc/sys/net/ipv4/tcp_rmem ```
Changes written to /proc/sys take effect immediately and apply to the running kernel. They are not persistent across reboots. Persistence requires writing to /etc/sysctl.conf or a drop-in file under /etc/sysctl.d/. The kernel version matters here: parameters introduced in Linux 5.15 will not exist on a 4.19 system, and writing an unknown parameter causes sysctl to error out with 'sysctl: setting key ... No such file or directory'. Always verify the parameter exists before deploying a sysctl.d file to heterogeneous fleets.
# List all current parameters and their values
sysctl -a
# Read a single parameter
sysctl net.ipv4.tcp_rmem
# Read the raw file directly
cat /proc/sys/net/ipv4/tcp_rmemThe sysctl.d Drop-in Pattern
Never edit /etc/sysctl.conf directly on a managed system. Use drop-in files under /etc/sysctl.d/ with a numeric prefix to control load order. Files are read in lexicographic order, so 99-custom.conf overrides values set by 10-default.conf.
Distributions ship their own defaults in /usr/lib/sysctl.d/. On Debian-based systems this is typically /usr/lib/sysctl.d/50-pid-max.conf and similar. On RHEL 9, systemd ships /usr/lib/sysctl.d/50-default.conf with a handful of security-relevant defaults. Your custom files in /etc/sysctl.d/ always win over /usr/lib/sysctl.d/ because sysctl-d processes /etc/ last.
After editing a file, apply it without rebooting:
``` sysctl --system ```
This loads all sysctl.d files in the correct order. To apply a single file explicitly:
``` sysctl -p /etc/sysctl.d/99-network.conf ```
For Ansible-managed systems, use the sysctl module with the reload: true option rather than templating the file and running sysctl manually. The module handles ordering and idempotency. If you are automating sysctl rollouts across multiple environments, DevOps tooling like taskbotshub.ai can pipeline parameter validation against kernel version checks before deployment.
# /etc/sysctl.d/99-network.conf
# Apply with: sysctl -p /etc/sysctl.d/99-network.conf
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216TCP Network Tuning: What Actually Matters
The default TCP socket buffers on Linux are sized for a network with 100ms RTT and a few hundred megabits of throughput. On 10GbE or 40GbE links, or on high-latency WAN paths, those defaults throttle throughput. The bandwidth-delay product is the key figure: a 10 Gbps link with 10ms RTT needs a buffer of 10,000,000,000 * 0.010 / 8 = 12.5 MB per connection to keep the pipe full. The default net.core.rmem_max of 212,992 bytes (208 KB) is insufficient by a factor of 60.
net.ipv4.tcp_rmem and net.ipv4.tcp_wmem each take three values: minimum, default, and maximum. The kernel auto-tunes between default and maximum based on available memory. Set net.core.rmem_max and net.core.wmem_max as the absolute ceiling the kernel is allowed to allocate per socket.
net.ipv4.tcp_congestion_control is worth checking. On kernels 4.9 and later, BBR is available and in our testing on servers with bufferbloat-prone paths, it improved sustained throughput by 30-40% over Cubic. On LAN paths with no bufferbloat, Cubic is fine. Check what is available:
``` sysctl net.ipv4.tcp_available_congestion_control ```
Two parameters that are almost always worth enabling: net.ipv4.tcp_fastopen and net.ipv4.tcp_mtu_probing. tcp_fastopen=3 enables TFO for both client and server connections, reducing round-trip overhead on connection setup. tcp_mtu_probing=1 enables path MTU discovery, which helps on VPN and tunnel paths where the MTU is lower than the default 1500.
net.ipv4.tcp_tw_reuse=1 allows TIME_WAIT sockets to be reused for new outbound connections when safe. This matters on servers making very high rates of short-lived outbound connections (proxies, scrapers, API gateways). Do not confuse this with tcp_tw_recycle, which was removed in Linux 4.12 because it broke connections through NAT.
# /etc/sysctl.d/99-tcp.conf
# Buffer sizing for 10GbE / high-latency paths
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 1048576 134217728
net.ipv4.tcp_wmem = 4096 1048576 134217728
# Congestion control - use BBR on kernel >= 4.9
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq
# Connection reuse and path MTU
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_fastopen = 3
# Increase connection backlog for high-traffic servers
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535Memory Management Parameters
vm.swappiness controls how aggressively the kernel moves anonymous memory pages to swap. The default is 60 on most distributions, meaning the kernel will start swapping when free memory drops to a level that most application servers would consider healthy. For database servers or any workload where you want application data in RAM rather than on disk, set this to 1 (not 0). Setting to 0 does not disable swap entirely on kernels before 3.5; it just makes the kernel swap only under out-of-memory pressure. On kernel 3.5 and later, 0 means the kernel will OOM-kill processes instead of swapping. Value of 1 is the safe production choice for latency-sensitive workloads.
vm.dirty_ratio and vm.dirty_background_ratio control when the kernel starts writing dirty pages to disk. dirty_background_ratio is the percentage of total RAM at which background writeback starts. dirty_ratio is the hard limit at which processes are blocked until writeback completes. On a server with 128 GB RAM, the defaults (10% and 20%) mean up to 25 GB of dirty data can accumulate before processes stall. For databases that do their own I/O management (PostgreSQL, MySQL with InnoDB), reduce these significantly:
``` vm.dirty_background_ratio = 3 vm.dirty_ratio = 10 ```
For write-heavy workloads on fast NVMe storage, you can push background_ratio to 5 to allow larger write coalescing. On slower spinning disks, keep it low to prevent write spikes.
vm.overcommit_memory has three modes. 0 (default) uses heuristics. 1 always allows overcommit, used by some in-memory databases and Redis. 2 refuses allocation that would exceed physical RAM plus swap, which prevents OOM kills but can cause legitimate allocations to fail. In our experience, the default 0 is correct for most workloads. Set it to 1 only if you have a specific application requirement and you understand the OOM implications.
vm.max_map_count defaults to 65530. Elasticsearch, some JVMs, and applications using large numbers of memory-mapped files will hit this limit and fail to start. Elasticsearch's own documentation requires 262144. Set it to at least 262144 on any system running JVM-based services.
# /etc/sysctl.d/99-memory.conf
# Reduce swap aggressiveness for application servers
vm.swappiness = 1
# Reduce dirty page accumulation (important for database hosts)
vm.dirty_background_ratio = 3
vm.dirty_ratio = 10
# Increase memory map limit (required for Elasticsearch, some JVMs)
vm.max_map_count = 262144
# Keep overcommit heuristic for most workloads
vm.overcommit_memory = 0Security-Relevant Kernel Parameters
Several sysctl parameters are directly relevant to security hardening. CIS Benchmarks and the DISA STIG both mandate specific values for these. We cover the most impactful ones.
kernel.dmesg_restrict = 1 prevents unprivileged users from reading dmesg output. dmesg can leak kernel addresses, memory layout, and hardware information useful for local privilege escalation. This is set by default on Ubuntu 22.04 and later but not on all distributions.
kernel.kptr_restrict = 2 hides kernel symbol addresses from /proc/kallsyms even for privileged users. Value 1 hides addresses from unprivileged users. Value 2 hides them from everyone including root. Use 2 in high-security environments, 1 is acceptable elsewhere.
net.ipv4.conf.all.rp_filter = 1 enables reverse path filtering, which drops packets where the source address is not reachable via the interface the packet arrived on. This blocks basic IP spoofing. Mode 1 is strict, mode 2 is loose. Use strict (1) unless you have asymmetric routing, in which case use loose (2).
net.ipv4.conf.all.accept_redirects = 0 and net.ipv6.conf.all.accept_redirects = 0 disable ICMP redirect acceptance. ICMP redirects can be used to poison routing tables. Disable on all servers that are not acting as routers.
kernel.randomize_va_space = 2 enables full ASLR. This should already be 2 on any modern distribution, but verify it. Value 1 enables partial ASLR (stack, VDSO), value 2 adds heap randomization.
fs.suid_dumpable = 0 prevents core dumps from setuid processes. A core dump of a setuid binary can contain sensitive data. The default on most distributions is 0, but some application install scripts set it to 2 for debugging and forget to revert it.
# /etc/sysctl.d/99-security.conf
# Restrict kernel information leakage
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
# Full ASLR
kernel.randomize_va_space = 2
# No setuid core dumps
fs.suid_dumpable = 0
# IP spoofing protection - strict reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
# Disable source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
# Log martian packets (spoofed source addresses)
net.ipv4.conf.all.log_martians = 1File Descriptor and Process Limits
fs.file-max is the system-wide limit on open file descriptors. The default is typically computed as RAM_in_KB / 10, which gives around 800,000 on an 8 GB system. For servers running many concurrent connections (load balancers, proxies, databases), increase this. The per-process limit is controlled separately through /etc/security/limits.conf or systemd unit LimitNOFILE settings.
Do not conflate fs.file-max with the per-user ulimit. A common production mistake is setting fs.file-max to 1,000,000 and wondering why nginx still hits the descriptor limit. The answer is the per-process NOFILE limit, which defaults to 1,024 on many distributions. Check what a running process actually sees:
``` cat /proc/$(pgrep -f nginx | head -1)/limits | grep 'open files' ```
For systemd-managed services, set LimitNOFILE in the unit file or override:
``` systemctl edit nginx ```
Then add:
``` [Service] LimitNOFILE=65535 ```
kernel.pid_max defaults to 32768 on 32-bit systems and 4194304 on 64-bit. Container-heavy hosts running thousands of short-lived processes can exhaust the PID namespace. If you are seeing 'fork: retry: Resource temporarily unavailable' errors on a system with available RAM, check pid_max:
``` sysctl kernel.pid_max echo 4194304 > /proc/sys/kernel/pid_max ```
kernel.threads-max is a separate limit on the number of threads. On systems running heavily threaded Java applications or Python multiprocessing workloads, you may hit this before pid_max.
# Check current system-wide fd usage vs limit
cat /proc/sys/fs/file-nr
# Output: open_fds 0 fs.file-max
# Increase system-wide file descriptor limit
sysctl -w fs.file-max=2097152
# Check per-process limit for a running service
cat /proc/$(pgrep -f postgres | head -1)/limits | grep 'open files'
# Persistent in /etc/sysctl.d/99-limits.conf
fs.file-max = 2097152
kernel.pid_max = 4194304Verifying and Auditing Parameters
After applying changes, always verify the kernel accepted them. sysctl -a after a --system reload will show current values. For auditing, diff the output against a known-good baseline:
``` sysctl -a 2>/dev/null | sort > /var/lib/sysctl-baseline-$(date +%Y%m%d).txt ```
Store that file in version control. On subsequent deployments, run the diff:
``` sysctl -a 2>/dev/null | sort | diff /var/lib/sysctl-baseline-20260101.txt - ```
For continuous compliance checking, systemd's systemd-analyze cat-config sysctl.conf sysctl.d shows the merged effective configuration that systemd will apply at boot, in load order. This is the canonical way to debug why a parameter set in 10-something.conf is being overridden:
``` systemd-analyze cat-config sysctl.d ```
If a parameter is not taking effect, check whether the relevant kernel module is loaded. Some parameters under net.netfilter.* require nf_conntrack to be loaded. Others under net.bridge.* require br_netfilter. Verify:
``` lsmod | grep nf_conntrack modprobe nf_conntrack sysctl -p /etc/sysctl.d/99-conntrack.conf ```
For nf_conntrack specifically, if you are running Docker or Kubernetes, the conntrack table can fill up causing 'nf_conntrack: table full, dropping packet' errors. The default nf_conntrack_max is often 65536. Increase it:
``` net.netfilter.nf_conntrack_max = 1048576 ```
You can watch conntrack table utilization in real time:
``` watch -n1 'cat /proc/sys/net/netfilter/nf_conntrack_count' ```
# Dump all parameters to a dated baseline file
sysctl -a 2>/dev/null | sort > /var/lib/sysctl-baseline-$(date +%Y%m%d).txt
# Show merged effective sysctl configuration (systemd)
systemd-analyze cat-config sysctl.d
# Watch conntrack table fill
watch -n1 'echo "$(cat /proc/sys/net/netfilter/nf_conntrack_count) / $(cat /proc/sys/net/netfilter/nf_conntrack_max)"'
# Apply all sysctl.d files in order
sysctl --systemNamespaced Parameters in Containers
Containers complicate sysctl management. Kubernetes and Docker both present a namespaced view of /proc/sys. Parameters under net.* and fs.mqueue.* are namespaced per network namespace. Parameters under kernel.* are mostly not namespaced and require host-level changes.
In Kubernetes, pod-level sysctl settings are configured in the pod spec under securityContext.sysctls. Only safe sysctls are allowed by default without additional kubelet flags. Safe sysctls are those that are properly namespaced: net.ipv4.ip_local_port_range, net.ipv4.tcp_syncookies, net.ipv4.ping_group_range, net.ipv4.ip_unprivileged_port_start, and net.ipv4.ip_local_reserved_ports.
Unsafe sysctls (net.core.somaxconn, vm.*, kernel.*) require the kubelet to be started with --allowed-unsafe-sysctls. Alternatively, apply them at the node level via a DaemonSet that runs sysctl commands in a privileged init container:
``` initContainers: - name: sysctl image: busybox securityContext: privileged: true command: ['sysctl', '-w', 'vm.max_map_count=262144'] ```
For Docker, pass sysctl at container start:
``` docker run --sysctl net.ipv4.tcp_tw_reuse=1 myimage ```
A practical point: if you are tuning a Kubernetes cluster for Elasticsearch and setting vm.max_map_count, the DaemonSet approach is the only reliable method. Pod-level securityContext sysctls will not cover this because vm.* is not namespaced.
# Kubernetes pod spec - safe sysctls only in securityContext
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
securityContext:
sysctls:
- name: net.ipv4.ip_local_port_range
value: "1024 65535"
- name: net.ipv4.tcp_syncookies
value: "0"
containers:
- name: app
image: myapp:latest