How strace Works Under the Hood
strace uses the ptrace(2) system call to intercept and record every system call a process makes. When you attach strace to a process, the kernel sends SIGTRAP to the tracer on each syscall entry and exit, which is why strace adds overhead - typically 2x to 10x slowdown depending on syscall frequency. For a process making millions of stat() calls per second, that overhead matters. For a process stuck waiting on a network connection, it is nearly invisible.
Each line in strace output follows the format: syscall(args) = return_value. If there is an error, the errno symbol and string appear after the return value. For example, openat(AT_FDCWD, "/etc/missing.conf", O_RDONLY) = -1 ENOENT (No such file or directory). The error is explicit - no guessing, no log parsing.
strace also traces signals. When a process receives SIGTERM, strace records --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1234, si_uid=0} ---. This is useful for diagnosing processes that die unexpectedly without logging the cause.
# Verify strace version
strace --version
# strace -- version 6.8
# Basic attach to running process
strace -p 1234
# Attach to all threads of a multi-threaded process
strace -p 1234 -fStarting a Process Under strace vs Attaching
You have two entry points: launch a new process under strace, or attach to an already-running one with -p. Launching gives you the full picture from exec() onward. Attaching misses everything that happened before you connected, but it works on any running process you have permission to trace.
For setuid binaries, attaching fails unless you are root. strace -u username lets you drop privileges after the trace starts, useful when debugging a service that must run as a specific user from a root shell. On systems with Yama ptrace_scope set to 1, non-root users can only trace their own child processes. Check /proc/sys/kernel/yama/ptrace_scope and set it to 0 temporarily if you need to attach to an unrelated process, but revert it after.
When launching, strace forks the target and executes it as a child. The -f flag follows any child processes created by fork() or clone(). Without -f, you only see syscalls from the initial process, which means you miss everything a shell script spawns or everything a Python multiprocessing pool creates.
# Launch a new process under strace
strace ls /var/log
# Follow forks - essential for shell scripts and servers
strace -f bash deploy.sh
# Attach to pid 5678 with timestamps
strace -p 5678 -t
# Check ptrace scope
cat /proc/sys/kernel/yama/ptrace_scope
# Temporarily allow cross-process tracing (revert after debugging)
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scopeFiltering Syscalls: The -e Flag
Raw strace output on a busy process is overwhelming. A single gunicorn worker can produce thousands of lines per second. The -e trace= filter is what makes strace actually usable. You can filter by syscall name, by category, or by file descriptor type.
The built-in categories save time. -e trace=file covers all syscalls that take a filename argument: open, openat, stat, access, execve, and about 40 others. -e trace=network covers socket, connect, bind, sendto, recvfrom, and related calls. -e trace=process covers fork, exec, wait, and exit variants. -e trace=signal covers signal-related calls. You can combine them with commas: -e trace=file,network.
Negation with ! excludes specific calls. -e trace='!read,write' shows everything except read and write, which is useful when your process is stuck in a loop and those two calls are drowning out everything else.
For diagnosing permission errors specifically, -e trace=access,openat,stat is our standard starting point. For network timeouts, -e trace=network,signal shows both the socket operations and any alarm() or sigaction() calls managing the timeout logic.
# Trace only file-related syscalls
strace -e trace=file ls /etc
# Trace network calls on a running nginx worker
strace -p $(pgrep -n nginx) -e trace=network
# Exclude noisy read/write, focus on everything else
strace -p 4321 -e trace='!read,write'
# Combine file and process categories
strace -f -e trace=file,process python3 app.py
# Trace only open failures - pipe through grep
strace -e trace=openat ls /nonexistent 2>&1 | grep ENOENTReading and Writing Strings: -s and -x Flags
By default, strace truncates string arguments to 32 characters. That is almost always too short. A full file path, a DNS hostname, or an HTTP header gets cut off right where it becomes interesting. Set -s 512 or -s 1024 to see more. We use -s 512 as a default in most debugging sessions - it captures paths up to 512 bytes without making the output unreadable.
The -x flag prints all non-ASCII strings in hexadecimal. This is essential when debugging binary protocols or when you suspect character encoding issues. -xx prints all strings in hex, including ASCII. Rarely needed but useful when a process is reading or writing raw bytes that look like garbage in the default display.
For write() and read() calls specifically, you often want to see the actual data being transferred. strace -e trace=write -s 4096 -p PID will show the first 4096 bytes of every write call. Combine this with grep or head to isolate specific patterns. We have used this to capture the exact HTTP request a broken client was sending to a server with no access logging enabled.
# Increase string length to 512 bytes
strace -s 512 curl https://example.com
# Show hex for non-ASCII data
strace -x -s 512 ./binary-protocol-client
# Capture write data up to 4096 bytes per call
strace -e trace=write -s 4096 -p $(pgrep redis-server)
# Trace reads from file descriptors - useful for config parsing
strace -e trace=read -s 2048 python3 -c "import yaml; yaml.safe_load(open('/etc/app.yaml'))"Timestamps and Statistics: -t, -T, and -c
Three time-related flags serve different diagnostic purposes. -t adds a wall clock timestamp to each line (HH:MM:SS). -tt adds microsecond precision. -T appends the time spent inside each syscall in brackets at the end of each line. -c instead of any of these suppresses per-line output and prints a summary table when strace exits.
The -c summary is underused. It shows call count, total time, average time, and error count per syscall. Run strace -c -f python3 slowscript.py and the output will immediately show if 90% of wall time is in stat() calls hitting a cold filesystem cache, or if the process is blocking on futex() waiting on a lock, or if it is spending all its time in read() calls on a slow disk.
On our test server running a Python web application with a cold cache, strace -c showed that stat() was being called 14,000 times during a single request and consuming 340ms of the total 400ms response time. The application was walking the entire Python path for each import on every request. That single strace -c run pointed directly at the problem.
Combine -T with -e to find the slowest individual syscall invocations. strace -T -e trace=read -p PID 2>&1 | sort -t'<' -k2 -rn | head -20 will rank the slowest read() calls by latency.
# Per-syscall timing in each line
strace -T -e trace=file ls /var
# Microsecond timestamps
strace -tt -p 9999
# Summary table - best for identifying bottlenecks
strace -c -f python3 manage.py migrate
# Find slowest read() calls, sorted by latency
strace -T -e trace=read -p $(pgrep postgres) 2>&1 | \
grep '<' | sort -t'<' -k2 -rn | head -20Writing Output to a File: -o and -ff
For long-running traces or multi-process traces, writing to stdout is impractical. Use -o filename to write strace output to a file. The file will contain the trace interleaved with any child process output if you also use -f. To separate each process into its own file, use -ff -o prefix. This creates files named prefix.PID for every process in the trace tree.
-ff is the right default for tracing forking servers. A prefork Apache or a gunicorn master with workers will each get their own trace file, making it trivial to isolate the worker handling the broken request from the others. We name the prefix with the service and timestamp: strace -ff -o /tmp/gunicorn-trace-$(date +%s) -p $(pgrep -n gunicorn).
For production use, set a size limit or use log rotation. strace itself has no built-in rotation, but writing to a named pipe or using tee with a size limit via head -c works. Alternatively, run the trace for a fixed duration with timeout: timeout 30 strace -o /tmp/trace.log -p PID. After 30 seconds, strace detaches cleanly.
# Write all output to a single file
strace -o /tmp/app-trace.log -f ./application
# Separate file per process/thread
strace -ff -o /tmp/worker-trace -p $(pgrep -n gunicorn)
# Creates: /tmp/worker-trace.1234, /tmp/worker-trace.1235, etc.
# Time-limited trace - 60 seconds then detach
timeout 60 strace -o /tmp/limited-trace.log -f -p $(pgrep nginx)
# Trace with microseconds, separate files, network only
strace -tt -ff -e trace=network -o /tmp/net-debug -p $(pgrep haproxy)Diagnosing Real Problems: Five Practical Scenarios
Scenario 1 - process failing to start with no output: Run strace ./binary 2>&1 | grep -E 'ENOENT|EACCES|EPERM'. This catches missing files, wrong permissions, and denied operations. We have found missing shared libraries this way when ldd showed everything as satisfied but an LD_PRELOAD was overriding the loader.
Scenario 2 - process consuming 100% CPU doing nothing useful: strace -c -p PID for 10 seconds tells you which syscall is burning the cycles. If futex() dominates, you have lock contention. If getpid() or clock_gettime() dominates, you likely have a tight loop calling these repeatedly, often a configuration or logging issue.
Scenario 3 - network connection timing out: strace -e trace=network,signal -T -p PID shows each connect() attempt and how long it blocks. A connect() returning after exactly 30.0 seconds confirms a kernel-level TCP timeout, not an application timeout. A connect() returning immediately with ECONNREFUSED confirms the port is closed on the remote end.
Scenario 4 - process reading wrong config file: strace -e trace=openat -s 512 ./service 2>&1 | grep 'O_RDONLY' shows every file the process actually opens for reading. We have traced services that ignored a config file passed on the command line and kept reading a hardcoded path from a previous installation.
Scenario 5 - intermittent database errors in production: strace -ff -o /tmp/db-trace -e trace=network -p $(pgrep -n myapp) captures all network activity. Then grep for 'ECONNRESET\|ETIMEDOUT\|EPIPE' across all the trace files. The timestamp from -tt tells you exactly when the error occurred, which you can correlate with database logs.
For teams running automated deployments where these trace patterns are part of runbooks, integrating strace-based health checks into a CI/CD pipeline is worth considering. Platforms like taskbotshub.ai can orchestrate these diagnostic scripts as part of automated post-deployment verification, running strace checks against canary instances before promoting a release.
# Scenario 1: catch permission and missing file errors
strace ./binary 2>&1 | grep -E 'ENOENT|EACCES|EPERM'
# Scenario 2: CPU hotspot identification
strace -c -p $(pgrep myservice) & sleep 10; kill %1
# Scenario 3: network timeout diagnosis with per-call timing
strace -e trace=network,signal -T -tt -p $(pgrep myapp) 2>&1 | grep connect
# Scenario 4: which config files does this binary actually read?
strace -e trace=openat -s 512 ./service 2>&1 | grep O_RDONLY
# Scenario 5: scan all trace files for network errors
grep -h 'ECONNRESET\|ETIMEDOUT\|EPIPE' /tmp/db-trace.*strace with seccomp and Containers
Containers complicate strace in two ways. First, the container runtime may apply a seccomp profile that blocks ptrace. Second, if you are outside the container trying to trace a process inside it, namespaces mean the PID visible from the host is different from the PID visible inside the container.
To trace a process in a Docker container from the host, find the host PID with docker top CONTAINER or ps aux | grep the process name, then use that PID with strace -p. The trace output will be identical to what you would see inside. To trace from inside a container, the container needs SYS_PTRACE capability. Add it with --cap-add=SYS_PTRACE at container run time, or add it to the security context in Kubernetes with securityContext.capabilities.add: [SYS_PTRACE].
For seccomp profile debugging specifically, strace 6.x added -e trace=%seccomp which shows seccomp_unotify and related calls. Combined with --output-separately in 6.x, you can get per-thread seccomp traces in separate files, which is useful when debugging why one thread in a multi-threaded app is hitting a seccomp violation while others are not.
Namespaces also affect the file paths strace reports. If a process in a mount namespace has a different view of the filesystem, strace will show the path as seen by the process, not the host path. Keep this in mind when reading openat() paths from containerized processes.
# Find host PID of process running inside Docker container
docker top mycontainer | grep myprocess
# Or:
ps aux | grep myprocess
# Trace from outside the container using host PID
strace -p HOST_PID -e trace=file
# Run container with ptrace capability for internal tracing
docker run --cap-add=SYS_PTRACE myimage
# Kubernetes pod with ptrace capability
# In pod spec:
# securityContext:
# capabilities:
# add: ["SYS_PTRACE"]
# seccomp-related syscalls in strace 6.x
strace -e trace=%seccomp -p PIDJSON Output and Scripting in strace 6.x
strace 6.x added --output-format=json which produces machine-readable output suitable for automated analysis. Each syscall entry becomes a JSON object with fields for pid, timestamp, syscall name, arguments, return value, and duration. This makes strace output parseable without fragile regex against human-readable text.
A typical JSON line looks like: {"pid":1234,"syscall":"openat","args":["AT_FDCWD","/etc/hosts","O_RDONLY"],"ret":3,"duration":0.000124,"timestamp":"2026-08-22T14:32:01.123456Z"}. You can pipe this to jq for filtering: strace --output-format=json -e trace=file ./app 2>&1 | jq 'select(.ret < 0)' to show only failed syscalls.
For scripting, the -z flag shows only successful syscall lines, and -Z shows only failed ones. These are text-mode filters available in older strace versions too, without needing JSON. strace -Z -e trace=openat ./app immediately shows every file open that failed, without any grep postprocessing.
The combination of -Z and -c is particularly useful: strace -Z -c ./app gives you a summary of only failed syscalls, which tells you at a glance how many times the application hit errors in each category.
# JSON output (strace 6.x)
strace --output-format=json -e trace=file ./app 2>&1 | jq .
# Show only failed syscalls in JSON, pipe to jq
strace --output-format=json ./app 2>&1 | jq 'select(.ret < 0)'
# Text mode: show only failed syscalls (-Z flag)
strace -Z -e trace=openat,stat ./configure
# Summary of only failed syscalls
strace -Z -c ./app
# Failed file opens with full paths
strace -Z -e trace=openat -s 512 ./app 2>&1Performance Impact and When to Use ltrace Instead
strace overhead on a production server is real. A process making 100,000 syscalls per second will slow to somewhere between 10,000 and 50,000 syscalls per second under strace, depending on the system and the specific calls. For most interactive debugging sessions on a server handling a few hundred requests per minute, this is acceptable. For a high-frequency trading system or a database under heavy write load, it is not.
For production systems where you cannot afford strace overhead, consider perf trace instead. perf trace uses kernel ring buffers and eBPF in newer kernels, producing similar output with significantly lower overhead. The syntax is similar: perf trace -p PID -e openat works the same as strace -p PID -e trace=openat but with overhead closer to 5-10% instead of 100-500%.
ltrace traces library calls rather than syscalls. If your problem is at the library level - wrong arguments to a libc function, a library returning unexpected values - ltrace -p PID is faster and more targeted than strace. Use ltrace when you know the issue is above the syscall boundary. Use strace when you need to see what actually hits the kernel.
For Golang binaries, strace works but output volume is high because Go's runtime makes many syscalls managing goroutines. Use -e trace=file,network and ignore the futex and nanosleep noise. For Rust binaries compiled with the musl target, strace output is unusually clean because musl makes far fewer syscalls than glibc.
In our experience, strace is the right first tool 80% of the time when a process misbehaves without explanation. The other 20% splits between ltrace for library-level bugs, gdb for logic errors you can reproduce, and perf for performance profiling where strace overhead invalidates the measurement.
# perf trace - lower overhead alternative
perf trace -p $(pgrep myapp) -e openat
# ltrace for library call tracing
ltrace -p $(pgrep myapp) -e 'malloc+free+fopen+fclose'
# Reduce strace overhead by filtering aggressively
strace -e trace=network -Z -p $(pgrep haproxy)
# For Go binaries, filter out runtime noise
strace -e trace=file,network -e '!futex,nanosleep,sched_yield' -p $(pgrep mygoapp)