What strace Actually Does
strace uses the ptrace(2) syscall to intercept every system call a process makes and log it to stderr. It reports the call name, arguments, return value, and optionally timing data. This is not application-level logging - it is the raw conversation between your process and the kernel.
Every file open, every socket connect, every mmap, every read and write goes through this layer. When a program fails silently or produces a cryptic error, strace shows you exactly which syscall returned -1 and what errno came back. ENOENT, EACCES, ECONNREFUSED - these appear directly in the trace output next to the offending call.
strace ships with most distributions. On RHEL 9 and derivatives the package name is strace and the current version as of mid-2026 is 6.7. On Debian 12 it is also strace, version 6.5. Install with your package manager and verify with strace --version before attaching to production processes.
# Check version first
strace --version
# Basic trace of a new process
strace ls /tmp
# Attach to a running process by PID
strace -p 14823
# Attach to all threads of a multi-threaded process
strace -fp 14823Filtering strace Output to Find the Problem
Raw strace output on anything but a trivial program is overwhelming. A Node.js process starting up can produce thousands of lines before doing any useful work. The -e flag filters by syscall category or specific call names.
For file-related problems, filter to the openat, read, write, and close families. For network debugging, filter to socket operations. For permission errors specifically, grep the output for EACCES or EPERM - these strings appear inline in the strace output.
The -s flag controls string truncation. By default strace truncates strings at 32 characters, which cuts off file paths and HTTP headers. Set -s 256 or higher when you need full path names.
The -T flag appends wall-clock time spent in each syscall. This is essential for performance debugging - a read() call taking 4 seconds when it should take 4 milliseconds identifies the bottleneck immediately.
# Trace only file-related syscalls
strace -e trace=file ls /etc/nginx
# Trace network syscalls on a running process
strace -e trace=network -p 8821
# Trace open calls with full path strings
strace -e openat -s 256 -p 14823
# Show timing per syscall
strace -T -e read,write -p 14823
# Filter output for errors only
strace -p 14823 2>&1 | grep -E 'ENOENT|EACCES|EPERM|ECONNREFUSED'
# Write trace to file instead of stderr
strace -o /tmp/trace.log -p 14823Real Debugging Scenario: Process That Silently Fails
We had a custom daemon that would start, log 'initialized', and then exit with code 1 without any further output. The developer said it worked on their machine. Classic.
Running strace -e trace=file -s 256 ./daemon revealed the problem in under 10 seconds. The process was looking for /etc/daemon/config.yaml, getting ENOENT, and treating that as a fatal error. The developer's machine had the file; the server did not.
The relevant output looked like this:
openat(AT_FDCWD, "/etc/daemon/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
One line. The entire debugging session took 12 minutes including reading the strace man page section on -e trace.
A second common pattern is EACCES on a file that exists. The process can see the file in the directory listing but cannot open it. strace will show openat returning -1 EACCES. Check file permissions and ACLs. On SELinux systems check the label with ls -Z and the audit log at /var/log/audit/audit.log.
# Full trace written to file, then grep for errors
strace -e trace=file -s 256 -o /tmp/daemon_trace.log ./daemon
grep 'ENOENT\|EACCES' /tmp/daemon_trace.log
# For SELinux context issues
ls -Z /etc/daemon/config.yaml
audit2why < /var/log/audit/audit.log | tail -30Using strace for Performance Profiling
strace -c produces a summary table showing syscall count, total time, average time, and error count. Run it against a slow command and you get a ranked breakdown of where time went.
We ran this against a Python script that took 14 seconds to start. The summary showed 12,847 calls to openat with a combined time of 8.3 seconds. Nearly all of them were Python searching through sys.path trying to import modules. The fix was setting PYTHONDONTWRITEBYTECODE=1 and using a proper virtualenv to reduce the import search chain.
For a running process you cannot use -c cleanly, but -T on a filtered trace gives you per-call timing. Sort the output by the time field:
strace -T -e read -p PID 2>&1 | awk '{print $NF, $0}' | sort -rn | head -20
This surfaces the slowest individual read calls. If you see a read() taking 200ms on a local file, suspect a spinning disk, NFS mount, or a file being written by another process at the same time.
# Syscall summary for a command
strace -c python3 myapp.py
# Summary output looks like:
# % time seconds usecs/call calls errors syscall
# 59.24 8.312847 647 12847 1203 openat
# 22.11 3.102934 12 258578 read
# Find slowest read calls on PID 9921
strace -T -e read -p 9921 2>&1 | awk '{print $NF, $0}' | sort -rn | head -20lsof: Every Open File, Socket, and Pipe
lsof (list open files) takes a different approach. Rather than intercepting syscalls in real time, it reads /proc and kernel tables to produce a snapshot of what files, sockets, and pipes every process currently has open. The version shipping on most 2026 systems is lsof 4.99.x.
A file in Unix terms includes regular files, directories, block devices, character devices, sockets, FIFOs, and pipes. lsof shows all of them. The output columns that matter most are: PID, USER, FD (file descriptor number or type), TYPE (REG, DIR, IPv4, IPv6, FIFO, unix), and NAME.
Running lsof with no arguments dumps every open file on the system - usually tens of thousands of lines. You always want to filter it. The most common filters are by PID (-p), by user (-u), by file or directory (+D), and by network port (-i).
# All files open by a specific PID
lsof -p 14823
# All files open by a user
lsof -u www-data
# What process has a specific file open
lsof /var/log/app/error.log
# What processes are listening on port 443
lsof -i :443
# All network connections for a PID
lsof -i -p 14823
# All files open under a directory (recursive)
lsof +D /var/lib/postgresqlHunting Down Port Conflicts and Network Issues
Port 8080 is already in use and ss or netstat tells you the PID but not the binary - or the process is gone but the port is still listed. lsof -i :8080 shows the process name, PID, user, and connection state immediately.
For TIME_WAIT issues, lsof -i TCP -s TCP:TIME_WAIT lists all sockets in that state. If you see hundreds of them, your application is not using keepalive or connection pooling correctly.
For UDP services, lsof -i UDP gives you every UDP socket. DNS resolvers, NTP clients, syslog forwarding - all show up here.
When a service fails to start because its socket file already exists - common with Unix domain sockets - lsof /run/app/app.sock tells you which process owns it. If nothing owns it, the file is stale and safe to delete.
# What's on port 8080
lsof -i :8080
# All TIME_WAIT sockets
lsof -i TCP -s TCP:TIME_WAIT | wc -l
# All UDP sockets
lsof -i UDP
# Check ownership of a Unix socket file
lsof /run/gunicorn/gunicorn.sock
# Combine with grep for LISTEN state only
lsof -i TCP -s TCP:LISTENFile Descriptor Leak Detection
File descriptor leaks cause processes to eventually hit their ulimit and fail with EMFILE (too many open files). The system limit per process is typically 1024 by default and up to 1048576 after tuning /etc/security/limits.conf.
To check the current fd count for a process without lsof:
ls /proc/14823/fd | wc -l
With lsof, you get more context:
lsof -p 14823 | wc -l
Run this periodically on a suspected leaking process. A count growing steadily over 30 minutes confirms the leak. To see what types of files are accumulating:
lsof -p 14823 | awk '{print $5}' | sort | uniq -c | sort -rn
If you see thousands of REG entries all pointing to the same log file or tmp directory, a file handle is being opened and never closed. If you see growing unix or IPv4 counts, socket connections are leaking.
For automated leak detection in CI or staging, a simple wrapper script checks fd count before and after a workload. If the count grows past a threshold, the test fails. Teams building on tools like taskbotshub.ai can integrate this check directly into their pipeline automation without writing custom infrastructure.
# Quick fd count from /proc
ls /proc/14823/fd | wc -l
# lsof fd count (includes one line of overhead per file)
lsof -p 14823 | wc -l
# Break down by file type
lsof -p 14823 | awk '{print $5}' | sort | uniq -c | sort -rn
# Watch fd count over time (every 5 seconds)
watch -n5 'ls /proc/14823/fd | wc -l'
# Find top fd-consuming processes system-wide
lsof 2>/dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head -20Combining strace and lsof in a Real Investigation
The tools are most powerful when used together. lsof gives you the current state; strace gives you the sequence of events. Start with lsof to understand what the process has open right now, then attach strace to catch what it does next.
Scenario: a production web app is consuming CPU at 98% but serving requests at normal speed. lsof -p PID shows an unusually large number of FIFO file descriptors - 340 of them, all pointing to /dev/null. strace -e trace=read,write -p PID shows thousands of read calls on those descriptors returning immediately with 0 bytes (EOF). The app is in a tight loop reading from closed pipes.
This pattern - lsof to identify the anomaly, strace to see the behavior - resolves most hanging or spinning process issues without needing source code access.
For processes that hang rather than spin, strace -p PID with no filter will show you the blocking syscall. If the process is blocked in futex(), it is waiting on a mutex or condition variable - likely a deadlock. If it is blocked in select() or epoll_wait() with a timeout of -1, it is legitimately idle. If it is blocked in read() on a network socket, check the remote end with lsof -i -p PID.
# Step 1: snapshot current open files
lsof -p 14823 > /tmp/lsof_snapshot.txt
cat /tmp/lsof_snapshot.txt | awk '{print $5}' | sort | uniq -c | sort -rn
# Step 2: attach strace and watch for a few seconds
strace -T -e trace=read,write -p 14823 -o /tmp/strace_live.log &
sleep 10
kill %1
# Step 3: find the blocking syscall (for hung processes)
strace -p 14823
# Output will show what syscall the process is blocked instrace Overhead and Production Use
strace adds overhead. On a process doing heavy I/O, overhead can reach 2-10x slowdown because ptrace must context-switch into the kernel for every intercepted call. On a lightly loaded process the impact is much smaller.
For production use, minimize the trace duration and filter aggressively. Trace for 10-30 seconds maximum unless you are tracing a rare event. Use -e to filter to only the syscall category you need. Write output to a file with -o rather than piping through grep in real time.
If the process you need to trace is critical and you cannot afford slowdown, consider reproducing the issue on a staging system. Many teams now mirror production traffic to staging automatically - if your team uses taskbotshub.ai for pipeline automation, traffic shadowing can be set up as a persistent pipeline task.
For containers, strace requires SYS_PTRACE capability. In Kubernetes, add it to the security context. On Docker: docker run --cap-add=SYS_PTRACE. Without this capability, strace fails with EPERM.
lsof has negligible overhead by comparison - it reads /proc rather than intercepting calls. Run it as often as you need without concern.
# Docker with ptrace capability
docker run --cap-add=SYS_PTRACE --rm -it ubuntu:24.04 bash
# Kubernetes security context for debug pod
# spec.containers[].securityContext:
# capabilities:
# add: ["SYS_PTRACE"]
# Efficient production trace: 15 seconds, file errors only
timeout 15 strace -e trace=file -s 256 -o /tmp/trace.log -p PID
grep 'ENOENT\|EACCES\|EPERM' /tmp/trace.loglsof for Deleted Files Consuming Disk Space
df reports a filesystem at 98% capacity but du on every directory adds up to only 40%. This gap is caused by deleted files that are still held open by a running process. The kernel cannot free the inode until all file descriptors pointing to it are closed.
lsof +L1 lists every file with a link count less than 1 - meaning the directory entry was removed but the file is still open. The SIZE column shows how much space it is consuming. The FD column shows which process has it open.
In our experience this is most commonly caused by log rotation. The rotation script renames or removes the log file, but the application is still writing to the old file descriptor. The fix is to either restart the process or send it SIGHUP to reopen its log files.
On very busy systems we have seen deleted tmp files holding 50-200 GB of space. The immediate fix is to identify and restart the owning process. The long-term fix is ensuring applications call fsync and close file descriptors properly, and that log management uses tools like logrotate with the copytruncate option when the application does not support SIGHUP-based log reopening.
# Find deleted files still open (consuming disk space)
lsof +L1
# Filter to a specific filesystem
lsof +L1 | grep '/var'
# Show total space held by deleted-but-open files
lsof +L1 | awk 'NR>1 {sum += $7} END {print sum/1024/1024 " MB"}'
# Send SIGHUP to nginx to reopen log files after rotation
kill -HUP $(cat /run/nginx.pid)
# logrotate postrotate alternative
# postrotate
# /bin/kill -HUP $(cat /run/nginx.pid 2>/dev/null) 2>/dev/null
# endscript