How File Descriptor Limits Are Actually Structured

Linux enforces two distinct per-process limits: the soft limit and the hard limit. The soft limit is the value the kernel actually enforces right now. A process can raise its own soft limit up to the hard limit without any privileges. Raising the hard limit requires CAP_SYS_RESOURCE, which in practice means root.

On top of per-process limits, the kernel maintains a system-wide ceiling: the total number of file descriptors any process on the entire system can hold open simultaneously. This is controlled by fs.file-max. On a modern 64-bit server with 32GB RAM, the default is typically around 9 million. You rarely hit that. The per-process limits are almost always the bottleneck.

The third layer is the per-user limit enforced by PAM via /etc/security/limits.conf. This applies to processes spawned by login sessions - shell logins, SSH sessions, su. It does not apply to services started by systemd, which is the most common source of confusion.

The fourth layer is the systemd unit file itself. If your service is managed by systemd, the LimitNOFILE directive in the unit file is what actually controls the file descriptor limit for that process, regardless of what /etc/security/limits.conf says.

# Check current soft and hard limits for the running shell
ulimit -Sn   # soft
ulimit -Hn   # hard

# Check limits for a specific running process (PID 1234)
cat /proc/1234/limits | grep 'open files'

# System-wide kernel maximum
sysctl fs.file-max

# Current system-wide file descriptor usage
cat /proc/sys/fs/file-nr
# output: open_fds  free_fds  max_fds

Reading the Numbers: What /proc/sys/fs/file-nr Tells You

The file /proc/sys/fs/file-nr outputs three numbers. The first is the total number of allocated file descriptors across all processes. The second is the number of allocated but unused file descriptors (this is almost always 0 on Linux kernels 2.6 and later, because Linux only allocates on demand). The third is fs.file-max.

On our test server running PostgreSQL 16 and Nginx 1.27 under moderate load, we saw values like '87432 0 9223372036854775807'. The max there is effectively unlimited at the kernel level. The real constraint was the per-process ulimit set to 1024 for the postgres user because nobody had touched the PAM config.

To find which processes are holding the most file descriptors open, do not guess. Run the following to get a ranked list by PID.

# Rank processes by open file descriptor count
for pid in /proc/[0-9]*; do
  pid_num=${pid##*/}
  count=$(ls /proc/$pid_num/fd 2>/dev/null | wc -l)
  echo "$count $pid_num $(cat /proc/$pid_num/comm 2>/dev/null)"
done | sort -rn | head -20

# Or with lsof (slower but more detail)
lsof 2>/dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head -20

Setting Limits for Login Sessions with ulimit and limits.conf

The ulimit command sets limits for the current shell and all processes it spawns. Changes made with ulimit -n are not persistent across sessions. They also do not affect already-running processes.

For permanent per-user limits on login sessions, edit /etc/security/limits.conf. The format is four columns: domain, type, item, value. Domain can be a username, a group prefixed with @, or * for all users. Type is either soft or hard.

A typical production configuration for a database or application server user looks like the block below. We use 65536 as the standard starting point for busy services. For very high connection count systems - Redis handling 50k+ clients, or a busy HAProxy instance - push it to 1048576.

After editing limits.conf, the change takes effect on the next login. It does not affect the current session and it does not affect systemd-managed services.

# /etc/security/limits.conf
# Format:    

# Set limits for the postgres user
postgres         soft    nofile    65536
postgres         hard    nofile    65536

# Set limits for all users in the www-data group
@www-data        soft    nofile    65536
@www-data        hard    nofile    65536

# Set for all users (catch-all)
*                soft    nofile    65536
*                hard    nofile    65536

# Verify the file is loaded by PAM - check this line exists
grep pam_limits /etc/pam.d/common-session
# Should output: session required pam_limits.so
// advertisement

Why limits.conf Does Not Work for systemd Services

This is the issue we see most often on support escalations. A sysadmin sets nofile to 65536 in limits.conf, restarts the service, checks /proc/PID/limits, and still sees 1024. The problem is that systemd does not go through PAM for services. It starts them directly, so pam_limits.so never runs.

For systemd-managed services, you have two options. The first is adding LimitNOFILE to the unit file. The second is setting a system-wide default in /etc/systemd/system.conf or /etc/systemd/user.conf.

For per-service configuration, use a drop-in override rather than editing the package-provided unit file directly. Drop-ins survive package updates.

# Create a drop-in override for a specific service (e.g., nginx)
mkdir -p /etc/systemd/system/nginx.service.d/
cat > /etc/systemd/system/nginx.service.d/limits.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

# Reload and restart
systemctl daemon-reload
systemctl restart nginx

# Verify the limit took effect
systemctl show nginx | grep LimitNOFILE
# Also check the live process
cat /proc/$(pgrep -f 'nginx: master')/limits | grep 'open files'

# Set system-wide default for all systemd services
# Edit /etc/systemd/system.conf
grep DefaultLimitNOFILE /etc/systemd/system.conf
# Uncomment and set:
# DefaultLimitNOFILE=65536

Raising the Kernel-Level Maximum with sysctl

fs.file-max is the system-wide hard ceiling on total open file descriptors. On modern kernels with a 64-bit address space, the default is already in the billions and you almost never need to change it. The cases where you do are embedded systems with constrained memory, or very old kernels where the default was set conservatively.

A related and more commonly relevant sysctl is fs.nr_open. This is the per-process maximum that the kernel will allow, and it caps what any ulimit or LimitNOFILE can actually set. On Linux 6.x kernels, fs.nr_open defaults to 1048576 (1M). If you need per-process limits above 1M, you must raise fs.nr_open first, then raise LimitNOFILE.

In our testing on Ubuntu 24.04 LTS with kernel 6.8, setting LimitNOFILE=2000000 in a systemd unit without first raising fs.nr_open caused the service to silently fall back to the kernel default rather than fail with an explicit error. Always verify with /proc/PID/limits after changes.

# Check current values
sysctl fs.file-max
sysctl fs.nr_open

# Apply temporarily (survives until next reboot)
sysctl -w fs.file-max=2097152
sysctl -w fs.nr_open=2097152

# Make permanent across reboots
cat > /etc/sysctl.d/99-file-descriptors.conf << 'EOF'
fs.file-max = 2097152
fs.nr_open = 2097152
EOF

# Apply without rebooting
sysctl --system

# Verify
sysctl fs.file-max fs.nr_open

Practical Limits for Common Services

Rather than abstract recommendations, here are the values we use in production environments based on specific workload profiles.

For PostgreSQL 16 with max_connections=500, set nofile to at least 65536 for the postgres user. Each connection uses multiple file descriptors (socket, WAL files, data files). With autovacuum workers and background processes, 500 connections can consume 3000-5000 file descriptors easily. We set 65536 as the floor and have never seen it be the bottleneck at that connection count.

For Nginx serving static files under high concurrency (10k+ simultaneous connections), each connection is a file descriptor, and each file being served is another. Set worker_rlimit_nofile in nginx.conf to match your LimitNOFILE. We use 65536 for most deployments and 262144 for CDN-edge nodes.

For Redis 7.x, the default maxclients is 10000. Each client connection is a file descriptor. Redis itself warns you at startup if the system limit is too low. Set nofile to at least 65536 for any Redis instance with more than a handful of clients.

For Java applications (Elasticsearch, Kafka, Cassandra), the JVM opens files aggressively. Elasticsearch 8.x requires a minimum of 65536 and will refuse to start in production mode if it cannot get at least that. We set 1048576 for Elasticsearch nodes.

If you are automating these configurations across fleets of servers - especially if you are deploying microservices or configuring infrastructure dynamically - the tooling at taskbotshub.ai includes pre-built automation scripts for kernel tuning and ulimit configuration that integrate with Ansible and Terraform pipelines, which cuts the manual configuration work significantly.

# nginx.conf - match this to your LimitNOFILE
worker_processes auto;
worker_rlimit_nofile 65536;

events {
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

# PostgreSQL: check current fd usage per backend
# Run inside psql as superuser
SELECT pid, usename, application_name,
       pg_stat_get_backend_client_addr(pid) as client_addr
FROM pg_stat_activity
ORDER BY pid;

# Redis: check at runtime
redis-cli INFO server | grep maxclients
redis-cli INFO server | grep connected_clients
// advertisement

Diagnosing EMFILE Errors in Production

When a process hits its file descriptor limit, the kernel returns EMFILE (error code 24: too many open files). Applications handle this differently. Nginx logs 'accept() failed (24: Too many open files)'. PostgreSQL logs 'could not open file: too many open files'. Node.js throws an EMFILE exception. Java throws java.io.IOException: Too many open files.

The diagnosis process is straightforward. First, confirm the process is actually hitting its limit.

If the current open count is close to the limit shown in /proc/PID/limits, that is your problem. If there is headroom, the EMFILE is coming from somewhere else - a library making syscalls that fail, or a subprocess inheriting a tighter limit.

One subtlety: inotify watches also consume file descriptors at the kernel level. Each inotify instance uses one fd, and each watch within that instance uses kernel memory but not an fd. However, some applications create thousands of inotify instances rather than one instance with many watches. Check with 'cat /proc/sys/fs/inotify/max_user_instances' and 'cat /proc/sys/fs/inotify/max_user_watches'.

Another source of fd exhaustion is socket TIME_WAIT accumulation. Sockets in TIME_WAIT hold a file descriptor until the timeout expires. On a busy server with short-lived connections, this can accumulate tens of thousands of sockets. Use 'ss -s' to count states. If TIME_WAIT is in the thousands, tune net.ipv4.tcp_tw_reuse before raising fd limits.

# Check open fd count for a process vs its limit
PID=$(pgrep -f 'your-process-name' | head -1)
OPEN=$(ls /proc/$PID/fd 2>/dev/null | wc -l)
LIMIT=$(cat /proc/$PID/limits | awk '/open files/ {print $4}')
echo "Open: $OPEN / Limit: $LIMIT"

# Count socket states system-wide
ss -s

# Check inotify limits
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches

# Find processes with inotify fds
lsof 2>/dev/null | grep inotify | awk '{print $1, $2}' | sort | uniq -c | sort -rn

# Strace a process to catch EMFILE in real time
strace -e trace=open,openat,accept,accept4 -p $PID 2>&1 | grep -E 'EMFILE|too many'

Container and Kubernetes Environments

Containers add another layer of complexity. The host kernel enforces file descriptor limits, and containers share the host kernel. A container process sees its own /proc view, but the underlying limits come from the container runtime configuration.

In Docker, you set the ulimit at container start time with --ulimit nofile=65536:65536. You can also set daemon-level defaults in /etc/docker/daemon.json. These settings propagate to the init process of the container, which then passes them to child processes.

In Kubernetes, the resources.limits block does not cover file descriptors - that field does not exist in the Pod spec as of Kubernetes 1.32. File descriptor limits in Kubernetes pods come from the container runtime (containerd or CRI-O). You configure these via RuntimeClass or by modifying the runtime configuration directly.

For containerd, edit /etc/containerd/config.toml and set the default ulimits. For CRI-O, the configuration is in /etc/crio/crio.conf under [crio.runtime]. Both require restarting the runtime daemon, which will restart all running containers on that node - plan accordingly.

Alternatively, the application can call setrlimit(2) directly at startup to raise its own soft limit up to the hard limit. Many well-engineered servers do this. If your application runs as root inside a container, it can raise both soft and hard limits up to fs.nr_open without any external configuration.

# Docker: set ulimits at run time
docker run --ulimit nofile=65536:65536 your-image

# Docker: set daemon-wide defaults
# /etc/docker/daemon.json
{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65536,
      "Soft": 65536
    }
  }
}

# containerd: check current default ulimits
cat /etc/containerd/config.toml | grep -A5 'default_rlimits'

# Add to containerd config under [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
# Or under [plugins."io.containerd.grpc.v1.cri"]
# default_ulimits = [{"type":"RLIMIT_NOFILE","hard":65536,"soft":65536}]

# Verify inside a running container
docker exec container_name cat /proc/1/limits | grep 'open files'

Scripting and Auditing Limits Across a Fleet

On more than a handful of servers, manual configuration does not scale. You need a way to audit current limits across systems and enforce desired state.

For auditing, a simple SSH loop works for small fleets. For anything more than 20 nodes, push this into your config management system. The Ansible uri module or a simple shell task can collect /proc/sys/fs/file-nr and /proc/1/limits from every host and report anomalies.

For enforcement, Ansible's pam_limits module handles /etc/security/limits.conf changes cleanly. For systemd unit overrides, use the Ansible template module to write drop-in files and trigger daemon-reload handlers. For sysctl values, the Ansible sysctl module is idempotent and handles the /etc/sysctl.d/ file creation correctly.

If you are building automation pipelines for this kind of kernel tuning work across cloud instances or bare metal, taskbotshub.ai offers prebuilt DevOps task bots that handle fleet-wide sysctl enforcement, ulimit configuration, and post-change verification as discrete, auditable tasks. That is worth evaluating if you are managing more than a few dozen nodes with similar tuning requirements.

For a quick fleet audit without pulling in Ansible for a one-off, the script below collects the four key numbers from a list of hosts.

#!/bin/bash
# fleet-fd-audit.sh - check fd limits across multiple hosts
# Usage: ./fleet-fd-audit.sh host1 host2 host3

HOSTS=("$@")
PRINTF_FMT="%-20s %-12s %-12s %-15s %-15s\n"

printf "$PRINTF_FMT" "HOST" "FILE-MAX" "NR-OPEN" "PROC1-SOFT" "PROC1-HARD"
printf '%s\n' "$(printf '%.0s-' {1..80})"

for HOST in "${HOSTS[@]}"; do
  DATA=$(ssh -o ConnectTimeout=5 -o BatchMode=yes "$HOST" \
    "sysctl -n fs.file-max fs.nr_open; \
     awk '/open files/ {print \$4, \$5}' /proc/1/limits" 2>/dev/null)
  
  if [ $? -eq 0 ]; then
    FILE_MAX=$(echo "$DATA" | sed -n '1p')
    NR_OPEN=$(echo "$DATA" | sed -n '2p')
    SOFT=$(echo "$DATA" | sed -n '3p' | awk '{print $1}')
    HARD=$(echo "$DATA" | sed -n '3p' | awk '{print $2}')
    printf "$PRINTF_FMT" "$HOST" "$FILE_MAX" "$NR_OPEN" "$SOFT" "$HARD"
  else
    printf "$PRINTF_FMT" "$HOST" "ERROR" "ERROR" "ERROR" "ERROR"
  fi
done
// advertisement