How inotify Works Under the Hood
The kernel maintains a watch descriptor table per inotify instance. When you call inotify_add_watch, you hand it an inotify file descriptor plus a path and an event mask. The kernel attaches a watch to the inode - not the path. This distinction matters for symlinks, bind mounts, and files that get replaced atomically (which most editors and config management tools do).
When a watched event fires, the kernel writes a fixed-size inotify_event struct to the inotify file descriptor. That struct contains the watch descriptor, a bitmask of what happened, a cookie for correlating MOVED_FROM/MOVED_TO pairs, a name length, and then a variable-length filename. You read events from the fd in a loop. epoll works fine here, which is how inotifywait integrates with select-based event loops.
One practical trap: inotify watches inodes, not paths. If /etc/nginx/nginx.conf is replaced by nginx reload (which writes to a temp file then renames), your watch on the original inode gets an IN_DELETE_SELF event. The new file has a new inode with no watch. If you need to track a file that gets replaced atomically, watch the containing directory instead and filter on the filename in the event name field.
# Check kernel inotify headers are present
ls /usr/include/sys/inotify.h
# Confirm kernel support (should return empty - feature is compiled in)
grep -i inotify /boot/config-$(uname -r) | head -5
# CONFIG_INOTIFY_USER=yInstalling inotify-tools
inotify-tools is packaged in every major distribution. The current stable release is 4.23.9.0. On older servers you may find 3.x versions in repos; the API is compatible but 4.x adds --no-dereference and better symlink handling.
After install, verify you have both binaries. inotifywait blocks and reports events one at a time to stdout. inotifywatch accumulates counts over a period and prints a summary table. Most automation uses inotifywait.
# Debian / Ubuntu
apt install inotify-tools
# RHEL / AlmaLinux / Rocky - package is in EPEL
dnf install epel-release && dnf install inotify-tools
# Arch
pacman -S inotify-tools
# Verify
inotifywait --version
# inotifywait 4.23.9.0
inotifywatch --versioninotifywait: Basic Usage and Event Types
The minimal invocation blocks until one event fires on the target path, prints it, and exits. Useful for testing but not for production. Add -m (monitor mode) to run indefinitely.
Event types you will actually use in scripts: - IN_CLOSE_WRITE: file opened for writing has been closed. This is what you want for 'file finished writing' triggers. Do not use IN_MODIFY - it fires on every write syscall, which means hundreds of events during a large file transfer. - IN_CREATE: new file or directory created in watched directory. - IN_DELETE / IN_DELETE_SELF: file deleted inside directory, or the watched file itself was deleted. - IN_MOVED_FROM / IN_MOVED_TO: rename operations. The cookie field correlates the pair. - IN_ATTRIB: permissions, ownership, or extended attributes changed. - IN_CLOSE_NOWRITE: opened read-only and closed. Useful for audit logging.
The -e flag accepts comma-separated event names. Always be explicit. Watching all events in a busy directory produces enormous output and can flood your pipe.
# Watch a single file, block until one event
inotifywait /etc/nginx/nginx.conf
# Monitor mode, specific events, quiet output format
inotifywait -m -e close_write,moved_to,create /etc/nginx/
# Recursive watch on a directory tree
inotifywait -m -r -e close_write /var/www/html/
# CSV output format for easier parsing
inotifywait -m -e close_write --format '%T,%w,%f,%e' --timefmt '%s' /etc/
# Output example:
# 1753900800,/etc/,nginx.conf,CLOSE_WRITE,CLOSEWriting a Production-Grade Watch Script
The naive pattern - while read event; do run_command; done - has a race condition. If events arrive faster than your command completes, the queue backs up. On a busy filesystem this causes inotify to drop events once the kernel queue overflows (IN_Q_OVERFLOW). The kernel's default queue size is 16384 events.
A better pattern uses a debounce lock. When an event fires, check if a job is already running. If so, set a 'pending' flag and skip. When the job finishes, check the pending flag and rerun if set. This collapses bursts of events into a single action.
The script below watches /etc/nginx for config changes and runs nginx -t then reloads. It handles atomic file replacement (watches the directory, not the file), debounces with a lockfile, and logs with timestamps.
#!/bin/bash
WATCH_DIR="/etc/nginx"
LOCKFILE="/run/nginx-reload.lock"
LOG="/var/log/nginx-watch.log"
log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*" >> "$LOG"; }
do_reload() {
if [ -f "$LOCKFILE" ]; then
touch "${LOCKFILE}.pending"
return
fi
touch "$LOCKFILE"
log "Config change detected, testing nginx config"
if nginx -t 2>> "$LOG"; then
log "Config valid, reloading nginx"
systemctl reload nginx && log "Reload OK" || log "Reload FAILED"
else
log "Config test FAILED, skipping reload"
fi
rm -f "$LOCKFILE"
if [ -f "${LOCKFILE}.pending" ]; then
rm -f "${LOCKFILE}.pending"
do_reload
fi
}
log "Starting nginx config watcher on $WATCH_DIR"
inotifywait -m -r -e close_write,moved_to,create \
--format '%w%f' \
"$WATCH_DIR" | while read -r CHANGED_FILE; do
# Filter to only .conf files
[[ "$CHANGED_FILE" == *.conf ]] || continue
log "File changed: $CHANGED_FILE"
do_reload &
doneRunning the Watcher as a systemd Service
Running your watch script under systemd gives you automatic restart on failure, proper logging to journald, and resource limits. The Restart=on-failure directive handles the case where inotifywait exits unexpectedly - for example, if the watched directory is deleted and recreated.
Set RestartSec to something nonzero. If the watched directory does not exist on startup, inotifywait exits immediately and systemd will hammer the restart in a tight loop. Five seconds is enough to prevent throttling.
Use PrivateTmp and NoNewPrivileges where your script allows it. If the script only calls nginx and systemctl, you can also drop capabilities. On our test server running AlmaLinux 9, this unit has been running for 14 months without intervention.
[Unit]
Description=Nginx config file watcher
After=network.target nginx.service
Requires=nginx.service
[Service]
Type=simple
ExecStart=/usr/local/bin/nginx-watch.sh
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=nginx-watch
NoNewPrivileges=true
PrivateTmp=true
[Install]
WantedBy=multi-user.targetinotifywatch: Counting Events Over Time
inotifywatch is underused. It watches a path for a specified duration and outputs a frequency table of event types per file. This is useful for profiling which files in a directory are hottest, auditing unexpected write patterns, or quantifying filesystem activity before and after a software change.
On our test server we ran inotifywatch against /var/lib/postgresql/14/main/pg_wal during a pgbench run at scale factor 10. The output showed 3,847 close_write events and 2,211 create events in 60 seconds, confirming WAL throughput matched expectations before we scaled the instance.
The -z flag includes files with zero counts, giving you a complete picture of every file touched during the window.
# Watch /var/log for 60 seconds, show all event counts
inotifywatch -v -r -t 60 /var/log/
# Watch specific events only
inotifywatch -e close_write,create -t 120 /var/lib/postgresql/14/main/pg_wal/
# Example output:
# total close_write create filename
# 6058 3847 2211 /var/lib/postgresql/14/main/pg_wal/Kernel Limits: inotify_max_user_watches and Queue Size
The default inotify_max_user_watches limit is 8192 on most distributions. Each watch costs roughly 1 KB of kernel memory. If you run a recursive watch on a large codebase or a Docker overlay filesystem with thousands of directories, you will hit this limit fast. inotifywait prints 'Failed to watch [dir]: upper limit on inotify watches reached' and exits.
The fix is straightforward: raise the limit. On production systems watching large application trees, we typically set this to 524288. JetBrains IDEs document requiring at least 524288 for large projects, which gives you a useful reference point for application-level watchers too.
The queue size (inotify_max_queued_events) controls how many events the kernel buffers before dropping them and emitting IN_Q_OVERFLOW. The default 16384 is adequate for most uses. If your application consumes events slowly or processes them in batches, raise it. Both values survive reboot when set via sysctl.d.
# Check current limits
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
# Raise watches limit immediately (takes effect now)
sysctl fs.inotify.max_user_watches=524288
# Make persistent
cat > /etc/sysctl.d/90-inotify.conf << 'EOF'
fs.inotify.max_user_watches = 524288
fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 512
EOF
sysctl -p /etc/sysctl.d/90-inotify.conf
# Check how many watches are currently active across all processes
cat /proc/sys/fs/inotify/max_user_watches
find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | \
awk -F'/' '{print $3}' | xargs -I{} cat /proc/{}/fdinfo/* 2>/dev/null | \
grep -c '^wd:'What inotify Cannot Watch
inotify has hard limits on what it can observe. Knowing them prevents hours of debugging.
Network filesystems: inotify does not work over NFS, CIFS/Samba, or SSHFS. The kernel only sees local VFS operations. Remote writes bypass inotify entirely. If you need cross-host file event notification, fanotify with a userspace daemon, or application-level mechanisms like PostgreSQL LISTEN/NOTIFY, are the right tools.
Procfs and sysfs: /proc and /sys are virtual filesystems. inotify does not work on them. You cannot watch /proc/sys/kernel/hostname to detect hostname changes via inotify.
Replaced inodes: as noted earlier, tools that write to a temp file then rename (vim, most config management tools, systemd's atomic writes) generate events on the directory, not on the original file's watch descriptor. If you watch /etc/resolv.conf directly using inotifywait /etc/resolv.conf, you will see IN_DELETE_SELF when resolvconf replaces it, then nothing more. Watch /etc/ instead.
Docker and container overlay filesystems: inotify works inside containers on overlayfs. However, watches set from outside the container (on the host) against a container's merged overlay path are unreliable. If you need file event monitoring inside a container, run inotifywait inside the container.
Fifo throughput: if your event consumer is too slow, the kernel drops events and sets IN_Q_OVERFLOW in the event mask. You will see the string 'Q_OVERFLOW' in inotifywait output. If this happens, either increase max_queued_events or speed up your consumer.
# Detect Q_OVERFLOW events in inotifywait output
inotifywait -m -r /some/busy/dir 2>&1 | grep -i overflow
# Count active inotify instances per process
for pid in /proc/[0-9]*/; do
count=$(ls -l ${pid}fd 2>/dev/null | grep -c 'inotify')
[ "$count" -gt 0 ] && echo "PID ${pid##/proc/}: $count inotify fds"
donefanotify: When inotify Is Not Enough
fanotify (filesystem access notification) was added in kernel 2.6.36 and extended significantly through 5.x. It addresses two gaps that inotify cannot fill: system-wide filesystem monitoring without per-directory watches, and the ability to intercept and block operations before they complete.
fanotify can mark an entire mount point rather than individual directories. One fanotify instance can watch all writes to /var without enumerating subdirectories. This is how antivirus engines like ClamAV's clamonacc mode work on Linux - one fd covering a whole filesystem.
The blocking mode (FAN_ACCESS_PERM, FAN_OPEN_PERM) lets your process read an event, inspect it, then write FAN_ALLOW or FAN_DENY back to the kernel before the originating process's syscall returns. This is how mandatory access control hooks and some DLP agents are implemented at the filesystem layer.
fanotify requires CAP_SYS_ADMIN or CAP_DAC_READ_SEARCH. The userspace library libfanotify simplifies the API, but most production use involves direct syscalls in C or Go. If you are building a DevOps automation tool that needs system-wide file auditing, fanotify is worth the complexity. For tools that automate deployment pipelines or react to file changes across infrastructure, projects like those listed on taskbotshub.ai often layer on fanotify or inotify depending on scope requirements.
# Check if fanotify is supported (requires kernel >= 2.6.36)
grep FANOTIFY /boot/config-$(uname -r)
# CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
# Simple fanotify mark on a mount point (requires root)
# No direct CLI tool equivalent to inotifywait - must use C or a library
# Example using the fanotify Python binding:
pip install fanotify
python3 -c "
import fanotify
fd = fanotify.init(fanotify.FAN_CLASS_NOTIF, fanotify.O_RDONLY)
fanotify.mark(fd, fanotify.FAN_MARK_ADD | fanotify.FAN_MARK_MOUNT,
fanotify.FAN_CLOSE_WRITE, -1, '/')
print('Watching all close_write on / mount')
"Real-World Use Cases with Complete Examples
Three patterns come up repeatedly in our work. The first is cache invalidation: watch a template directory and rebuild static assets when source files change. The second is secret rotation: watch a mounted Vault secret path and SIGHUP the application when credentials are refreshed. The third is audit logging: record every write to /etc/ with the process and user responsible.
For secret rotation, the pattern is straightforward. Vault's Kubernetes secrets store agent writes renewed credentials to a mounted path. inotifywait detects the close_write event and sends SIGHUP to the application process. This eliminates polling and reduces the time between credential renewal and application pickup from up to 60 seconds (typical poll interval) to under one second.
For audit logging, inotify alone cannot tell you which process caused the event. For that you combine inotify (to detect the event) with inotify's filename information and then query the audit log or use ausearch. Alternatively, auditd with -w rules does the full job natively - inotify is not the right tool if you need who-did-it attribution.
When naming internal tools built around these patterns - a secret-watcher daemon, a cache-invalidator service - readable names matter for team adoption. Services like nicename.me can help when you are also registering a domain for an open-source release of such a tool.
#!/bin/bash
# Secret rotation watcher for Vault agent
# Watches /vault/secrets/ and sends SIGHUP to app when credentials change
SECRET_DIR="/vault/secrets"
APP_NAME="myapp"
log() { logger -t vault-watch "$*"; }
get_app_pid() {
pgrep -x "$APP_NAME" | head -1
}
log "Starting Vault secret watcher on $SECRET_DIR"
inotifywait -m -e close_write,moved_to \
--format '%f' \
"$SECRET_DIR" | while read -r SECRET_FILE; do
log "Secret file changed: $SECRET_FILE"
APP_PID=$(get_app_pid)
if [ -n "$APP_PID" ]; then
log "Sending SIGHUP to $APP_NAME (PID $APP_PID)"
kill -HUP "$APP_PID" && log "SIGHUP sent OK" || log "SIGHUP failed"
else
log "WARNING: $APP_NAME not running, skipping SIGHUP"
fi
doneDebugging inotify Watches
When watches are not firing as expected, start by confirming the watch is actually registered. The /proc filesystem exposes active inotify instances and their watch descriptors through fdinfo.
If events fire but not for the right file, check whether the tool writing the file does an atomic rename (which most do). Strace is your friend here: attach to the writing process and look for rename syscalls. If you see write -> fsync -> rename, watch the directory, not the file.
If you see no events at all on a directory you know is being written to, check whether it is a mount of a network filesystem or a special virtual filesystem. Mount type is visible in /proc/mounts.
For recursive watches that silently miss new subdirectories: inotifywait -r sets watches on all directories that exist at startup. Subdirectories created after startup get a watch added automatically via IN_CREATE handling, but there is a race window between directory creation and watch registration where events can be missed. If this matters - for example, watching a deployment target directory where a new release creates a new subdirectory structure - consider watching the parent and handling IN_CREATE events to set up new watches explicitly, or use a tool like watchman which handles this robustly.
# List all inotify file descriptors for a specific process
ls -la /proc/$(pgrep inotifywait)/fd | grep inotify
# Dump watch descriptors for an inotify fd
# Replace PID and FD_NUM with actual values
cat /proc/$(pgrep inotifywait)/fdinfo/3
# Output includes: pos, flags, and one 'wd:' line per watch
# Strace a writing process to see if it uses atomic rename
strace -e trace=openat,write,rename,renameat -p $(pgrep vim) 2>&1 | tail -20
# Check if the target path is on a network filesystem
mount | grep "$(stat -c %m /your/watched/path)"
# Count total active watches across the system
cat /proc/sys/fs/inotify/max_user_watches
find /proc/*/fdinfo -name '[0-9]*' -exec grep -l '^inotify' {} \; 2>/dev/null | wc -l