How the Hierarchy Is Defined and Where to Read It
The authoritative source is the FHS specification at refspecs.linuxfoundation.org. On any systemd-based distro you can also pull the hierarchy from the man page:
`man hier`
On our test server running Debian 12.5, `man hier` gives a concise description of every standard directory. The spec divides directories along two axes: shareable vs. unshareable, and static vs. variable. Shareable content can be mounted read-only and served to multiple hosts. Static content does not change without administrator action. That matrix matters when you are designing NFS exports or read-only root images for embedded or immutable systems.
Distributions are allowed to extend the hierarchy, and they do. Red Hat adds /etc/sysconfig. Alpine moves init to /sbin/init rather than symlinking through /bin. macOS ships with /private/etc and symlinks /etc to it. When you write a deployment script, always resolve symlinks with `realpath` rather than assuming the path you see is canonical.
man hier
realpath /etc
# On macOS: /private/etc
# On Linux: /etcThe Root Filesystem: / and What Must Live There
Everything hangs off the root mount point `/`. The FHS requires that root contain only the files needed to boot the system and mount other filesystems. That is a stricter rule than it sounds. If `/usr` lives on a separate partition that is not mounted in the initramfs, then any binary that calls into `/usr` will fail during early boot.
In practice, most modern distributions have merged `/bin`, `/sbin`, and `/lib` into their `/usr` equivalents and left symlinks at the old paths. On Debian 12:
The merge means your boot toolchain must be available in the initramfs image. Verify with:
If you are building a minimal root filesystem for a container or embedded target, you need exactly the files that the init process and its direct dependencies reference. Use `ldd` on each binary and chase every shared library.
The rule we enforce on our production container images: the root filesystem is read-only. Anything that needs to write at runtime goes to a tmpfs or a named volume mounted at a specific path. That path must be in the FHS-defined variable directories.
ls -la /bin /sbin /lib
# lrwxrwxrwx 1 root root 7 /bin -> usr/bin
# lrwxrwxrwx 1 root root 8 /sbin -> usr/sbin
# lrwxrwxrwx 1 root root 7 /lib -> usr/lib
# Check what your initramfs contains
lsinitramfs /boot/initrd.img-$(uname -r) | grep -E '^(bin|sbin|lib)/' | head -20/usr: The Bulk of the System
/usr is the second major hierarchy and the largest directory on most systems. It is shareable and static. On a standard Debian 12 install, `/usr` accounts for roughly 3.5 GB of the base system.
Key subdirectories:
- `/usr/bin` - all non-essential user binaries (everything from `bash` to `python3`) - `/usr/sbin` - non-essential system administration binaries (`adduser`, `useradd`, `sshd`) - `/usr/lib` - libraries and internal binaries not meant for direct execution - `/usr/lib64` - 64-bit libraries on multilib systems - `/usr/include` - C/C++ header files - `/usr/share` - architecture-independent data: man pages, locale files, documentation - `/usr/local` - the administrator's own software, not managed by the package manager - `/usr/src` - kernel source and other source packages
The `/usr/local` tree is critical for sysadmins. Anything you compile and install by hand goes here. Its own sub-hierarchy mirrors `/usr`: `bin`, `lib`, `include`, `share`. The package manager never touches it. When you run `make install` without a prefix override, most autoconf-based projects default to `/usr/local`.
Before you drop a custom binary into `/usr/local/bin`, check whether your PATH puts it before or after `/usr/bin`. Shadowing a system binary unintentionally has broken more systems than most sysadmins want to admit.
du -sh /usr
# 3.4G /usr
# See what is in /usr/local on a fresh system vs a production system
find /usr/local/bin -type f -executable | sort
# Check PATH ordering
echo $PATH | tr ':' '\n'/etc: Configuration, Not Binaries, Not Data
/etc is host-specific, static configuration. The FHS is explicit: no binaries go in /etc. Configuration files that control system behavior live here. Package managers own most of what is here, and many tools follow a drop-in pattern where `/etc/programname.d/` holds fragment files that override or extend the main configuration.
Files and directories you interact with constantly:
- `/etc/passwd`, `/etc/shadow`, `/etc/group` - user and group databases - `/etc/fstab` - filesystem mount configuration - `/etc/hosts`, `/etc/resolv.conf`, `/etc/nsswitch.conf` - name resolution - `/etc/systemd/` - systemd unit overrides and system configuration - `/etc/ssh/sshd_config` - SSH daemon configuration - `/etc/cron.d/`, `/etc/cron.daily/` - scheduled tasks - `/etc/sudoers` and `/etc/sudoers.d/` - privilege escalation rules
The drop-in pattern is the right way to manage configuration in automation. Never edit the base package configuration file. Create a file in `.d/`. That way a package upgrade does not clobber your changes:
Version control /etc. On our infrastructure, we use a bare git repository in `/etc/.git` initialized with `git init` and commit every change with the responsible engineer's name in the message. Some teams prefer etckeeper, which hooks into the package manager and auto-commits after every apt or dnf transaction.
If you are running a DevOps automation platform, tools like taskbotshub.ai can manage configuration drift detection across multiple hosts by comparing live /etc state against a known-good snapshot stored in your repository.
# Drop-in override for sshd - never edit sshd_config directly
mkdir -p /etc/ssh/sshd_config.d/
cat > /etc/ssh/sshd_config.d/99-hardening.conf << 'EOF'
PermitRootLogin no
PasswordAuthentication no
X11Forwarding no
EOF
chmod 600 /etc/ssh/sshd_config.d/99-hardening.conf
systemctl reload sshd
# Verify it is parsed
sshd -T | grep -E 'permitroot|passwordauth'/var: Variable Data That Changes at Runtime
/var holds files whose size and content change continuously. The FHS splits it into predictable subdirectories, and understanding them prevents disk-full incidents.
- `/var/log` - log files. On systemd systems, journald writes binary logs to `/var/log/journal/`. Traditional text logs from rsyslog or syslog-ng still land here. - `/var/lib` - persistent application state. Databases, package manager state (`/var/lib/dpkg`, `/var/lib/rpm`), container runtime state. - `/var/cache` - cached data that can be regenerated. APT package cache lives at `/var/cache/apt/archives/`. Safe to delete. - `/var/spool` - queued data awaiting processing. Mail queues, print spools, cron job queues. - `/var/run` - on modern systems, this is a symlink to `/run`, a tmpfs. PID files, sockets, lock files. - `/var/tmp` - temporary files that survive reboots (unlike `/tmp` which is often cleared).
The most common production issue: `/var/log` fills the partition and the system stops accepting writes. Preventive measures:
On our test servers, we run logrotate with a weekly check via systemd timer and set a max journal size. The journal size limit is particularly important on systems that generate high log volume.
# Check journal disk usage
journalctl --disk-usage
# Cap journal size in /etc/systemd/journald.conf
[Journal]
SystemMaxUse=2G
SystemKeepFree=500M
MaxRetentionSec=30day
# Apply without reboot
systemctl restart systemd-journald
# Find what is consuming /var/log
du -sh /var/log/* | sort -rh | head -15/proc and /sys: Kernel Interfaces, Not Real Files
/proc and /sys are virtual filesystems. Nothing on /proc or /sys is stored on disk. The kernel generates the content on read and accepts configuration on write.
/proc exposes per-process information and kernel state. Every running process has a directory at `/proc/PID/` containing its file descriptors, memory maps, environment, command line, and namespace memberships. This is how tools like `ps`, `top`, `lsof`, and `strace` gather data.
/sys (sysfs) is the structured interface to kernel objects: devices, drivers, buses, power management. Udev rules read from /sys to make decisions about /dev node creation.
Kernel parameters exposed through /proc/sys can be read and written directly or managed with `sysctl`. The `sysctl` command is the correct interface for persistent changes:
Network tuning, memory management, and security settings all go through this interface. On high-throughput servers, the TCP parameters are the first thing we check:
Do not script writes directly to /proc/sys paths in production. Use sysctl.d drop-ins so changes survive reboots and are documented.
# Read a kernel parameter
cat /proc/sys/net/ipv4/tcp_rmem
# Write a persistent network tuning drop-in
cat > /etc/sysctl.d/99-network-tuning.conf << 'EOF'
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = bbr
EOF
# Apply immediately without reboot
sysctl -p /etc/sysctl.d/99-network-tuning.conf
# Inspect a process's open file descriptors
ls -la /proc/$(pgrep nginx | head -1)/fd/dev, /run, and /tmp: Transient Directories
/dev contains device nodes. On modern systems, devtmpfs is mounted here at boot and udev manages node creation dynamically. You should rarely need to create device nodes manually, but when you do, `mknod` is the tool.
/run is a tmpfs mounted early in the boot process. It replaced the old pattern of splitting runtime files between /var/run and /var/lock. PID files, Unix domain sockets, and lock files belong here. The key property: /run is cleared on every boot. Services write their PID files here and read them back - this is how `systemctl status` knows whether a service is running outside of cgroup tracking.
/tmp is also typically a tmpfs in 2026, though this is controlled by the systemd-tmpfiles configuration and the presence of `tmp.mount`. Verify:
The difference between /tmp and /var/tmp is survival across reboots. Use /tmp for files you need only during the current session. Use /var/tmp for files that need to persist across a reboot but are still not permanent data (build artifacts, large downloads in progress).
Systemd cleans both on a schedule defined in /usr/lib/tmpfiles.d/tmp.conf. Check the active cleanup rules:
# Verify /tmp is a tmpfs
findmnt /tmp
# Or
df -T /tmp
# Check tmpfiles cleanup configuration
cat /usr/lib/tmpfiles.d/tmp.conf
# List what is currently in /run
ls -la /run/
# Check which service owns a socket
stat /run/systemd/private/tmp
ls /run/*.pid 2>/dev/null/home, /root, and /srv: User and Service Data
/home contains user home directories. It is shareable and should live on its own partition or be NFS-mounted. The FHS does not specify subdirectory structure inside /home - that is up to the administrator.
/root is the home directory for the root account. It is explicitly not inside /home because /home may not be available during early boot or single-user mode. On our servers, /root contains only dot-files for shell configuration and a few scripts. We do not store sensitive key material here - that goes in a secrets manager.
/srv is the correct location for data served by the system. A web server's document root belongs at `/srv/http/` or `/srv/www/`. An FTP server's files belong at `/srv/ftp/`. An application's served files belong at `/srv/appname/`. In practice, many administrators ignore /srv and put everything in /var/www or /opt, which is why you see inconsistency across systems.
We recommend /srv for services you deploy yourself, because it is distinct from package-managed data in /var/lib and application code in /opt. When you are standing up a new project, choosing a clear, clean name for your service directory matters more than it seems - it affects every log path, every cron job, and every reference in your runbooks. If you are also registering a domain for the project, nicename.me is worth checking early since the project directory name and the domain name should ideally match.
For a web application:
The /srv hierarchy makes it immediately clear that this directory is externally-facing content, distinct from application state in /var/lib or binaries in /usr/local.
# Recommended /srv layout for a web application
mkdir -p /srv/myapp/{public,uploads,config}
chown -R www-data:www-data /srv/myapp/public
chmod 750 /srv/myapp/config
# Mount /home from NFS
cat >> /etc/fstab << 'EOF'
nfsserver:/export/home /home nfs4 defaults,_netdev,ro 0 0
EOF
# Check disk usage per home directory without entering them
du -sh /home/* | sort -rh | head -20/opt and /usr/local: Third-Party and Custom Software
/opt is for add-on application packages. The FHS specifies that each package installs into its own `/opt/packagename/` directory with its own bin, lib, and etc subdirectories. This self-contained model means uninstalling software is as simple as `rm -rf /opt/packagename` with no leftover files scattered across /usr.
Commercial software, proprietary tools, and large self-contained applications belong in /opt. Examples we see frequently: Oracle databases at /opt/oracle, JetBrains IDEs at /opt/jetbrains, custom monitoring agents. Kubernetes distributions like k3s install their own binaries under /opt/cni and /opt/local-path-provisioner.
The distinction from /usr/local: /usr/local is for software you compiled and installed to extend or override the base system. /opt is for self-contained third-party packages that do not integrate into the base system hierarchy.
For DevOps automation workflows, some teams use /opt to stage deployment artifacts before promoting them. If you are using an AI-assisted automation platform like taskbotshub.ai, the deployment scripts it generates will typically write to /opt for third-party tools and /usr/local for custom tooling.
Manage PATH for /opt binaries with a profile.d drop-in rather than editing /etc/profile directly:
# Add /opt/myapp/bin to PATH system-wide
cat > /etc/profile.d/myapp.sh << 'EOF'
export PATH="/opt/myapp/bin:$PATH"
export MYAPP_HOME="/opt/myapp"
EOF
chmod 644 /etc/profile.d/myapp.sh
# Verify after re-sourcing
source /etc/profile.d/myapp.sh
which myapp-binary
# Check what is installed in /opt
ls -la /opt/
du -sh /opt/* | sort -rhMount Points: /mnt, /media, and Planning Partition Layout
/mnt is for temporary manual mounts. When you need to mount a disk to recover data, mount it here. It has no subdirectory structure defined by the FHS.
/media is for removable media that the OS mounts automatically: USB drives, optical discs. On desktop systems, udev and udisks2 create subdirectories here automatically. On server systems, you can safely ignore /media.
Partition layout decisions have long-term consequences. On production servers, we use a standard layout that isolates variable data from static system files:
Separating /var prevents log floods from filling the root filesystem and crashing the system. Separating /tmp prevents runaway processes from consuming all disk space with temporary files. Separating /home prevents user data from impacting system operation.
For systems using LVM, you can resize partitions online with pvresize, lvextend, and resize2fs or xfs_growfs. XFS does not support shrinking. Plan for growth:
On our test server with a 100GB disk, we allocate: /boot 1GB (ext4), / 20GB (xfs), /var 40GB (xfs), /home 30GB (xfs), with 9GB left unallocated in the volume group for emergency expansion.
# Recommended LVM layout script for a new server
pvcreate /dev/sdb
vgcreate vg_system /dev/sdb
lvcreate -L 20G -n lv_root vg_system
lvcreate -L 40G -n lv_var vg_system
lvcreate -L 30G -n lv_home vg_system
# Leave remaining space unallocated for emergencies
mkfs.xfs /dev/vg_system/lv_root
mkfs.xfs /dev/vg_system/lv_var
mkfs.xfs /dev/vg_system/lv_home
# Emergency expansion when /var fills up
lvextend -L +10G /dev/vg_system/lv_var
xfs_growfs /varFinding Files: locate, find, and Understanding the Database
Two tools dominate file search on Unix systems. `find` traverses the live filesystem in real time. `locate` (from mlocate or plocate) queries a pre-built database updated nightly by `updatedb`.
For ad-hoc searches during incident response, `find` with `-mount` (do not cross filesystem boundaries) is the correct approach. For quick lookups in non-urgent situations, `plocate` on modern systems is significantly faster than the old mlocate implementation - on our test system, a search that takes 0.8 seconds with mlocate takes 0.04 seconds with plocate.
Understanding where to look for specific file types saves time during debugging:
- Config changed: `/etc/` and `/usr/lib/` - Binary missing or wrong version: `/usr/bin/`, `/usr/local/bin/`, `/opt/` - Library version conflict: `ldconfig -p | grep libname` - Service failing to start: `/var/log/` and `journalctl -u servicename` - Disk full: `du -sh /* 2>/dev/null | sort -rh | head -20`
The `lsof` command is underused. It shows every open file on the system including deleted files that are still held open (the classic cause of disk space not being reclaimed after log deletion):
# Find deleted files still held open (common cause of 'df shows full, du shows empty')
lsof +L1 | grep deleted
# Recover a deleted but open log file
lsof +L1 | grep deleted | awk '{print $2, $4, $9}'
# Then copy from /proc/PID/fd/FD before the process exits
cp /proc/12345/fd/3 /tmp/recovered-log.txt
# Find all setuid binaries (security audit)
find / -mount -perm -4000 -type f 2>/dev/null | sort
# Find files modified in the last 24 hours in /etc
find /etc -mount -mtime -1 -type f | sortImmutable and Read-Only Root Filesystems
In 2026, immutable root filesystems are standard practice for containers and increasingly common for servers running on Fedora Silverblue, Ubuntu Core, or custom embedded Linux. The FHS supports this model: only the variable directories need to be writable at runtime.
The minimum set of writable mounts for a functional read-only root system:
- `/etc` - or use a config management overlay - `/var` - all variable data - `/run` - tmpfs, always writable - `/tmp` - tmpfs, always writable - `/home` - user data
Some distributions handle /etc with an overlay filesystem: the base /etc is read-only from the OS image, and writes go to an upper layer. systemd-sysext handles OS image layering. overlayfs handles the merge.
For containers, the FHS compliance of your image directly affects correctness. A container that writes to /usr or /bin at runtime will fail on a read-only root. Audit your container images:
Our recommendation for new containerized services: start with a read-only root flag in your container runtime configuration. Fix every crash by identifying what needed to write somewhere it should not, and mount only the specific paths that legitimately need write access.
# Run a container with read-only root to find write violations
docker run --read-only \
--tmpfs /run \
--tmpfs /tmp \
-v myapp-data:/var/lib/myapp \
myapp:latest
# Find writes to non-variable paths during a test run
strace -f -e trace=openat,creat,rename,mkdir \
-o /tmp/strace.log \
myapp --test-mode 2>&1 | \
grep -v '/var\|/run\|/tmp\|/proc\|/sys'