What a Signal Actually Is
A signal is an asynchronous notification sent to a process or process group by the kernel, another process, or the process itself. The kernel interrupts the target process's normal execution, saves its state, and either runs a registered handler, takes a default action, or ignores the signal entirely. The signal number is a small integer - on Linux x86-64 there are 64 signals (1 through 64), split into standard signals (1-31) and real-time signals (32-64).
Signals are not queued for standard signals. If SIGTERM arrives twice before the process handles the first one, the process may only see it once. Real-time signals (SIGRTMIN through SIGRTMAX) are queued and carry ordering guarantees, which makes them useful for custom application protocols but uncommon in day-to-day administration.
Every signal has three possible dispositions: the default action (which varies by signal - terminate, core dump, stop, or ignore), a custom handler function installed by the process, or explicit ignore. The exceptions are SIGKILL and SIGSTOP, which cannot be caught, ignored, or blocked - ever. The kernel enforces this absolutely.
# List all signals on Linux
kill -l
# Same on BSD
kill -l
# See signal numbers and names together
kill -L # Linux only, shows table formatSignal Delivery: From kill() to Handler Execution
When you run `kill -TERM 1234`, the shell calls the kill(2) syscall, which asks the kernel to set a pending signal bit in the target process's task_struct. The signal is not delivered immediately. It becomes pending until the target process transitions from kernel space back to user space - typically at the next system call boundary or after returning from an interrupt handler.
This has a practical consequence: a process stuck in an uninterruptible sleep (state D in `ps` output) cannot receive any signal, including SIGKILL. The D state means the process is waiting for I/O inside a kernel path that has explicitly disabled signal delivery. You cannot kill a D-state process with any signal. You wait for the I/O to complete, fix the underlying resource (usually NFS or a dead block device), or reboot.
Signal masking is the mechanism processes use to defer delivery. A process calls sigprocmask(2) to add signals to its signal mask. Masked signals remain pending but are not delivered until unmasked. This is how multi-threaded programs protect critical sections from being interrupted mid-operation. Each thread has its own signal mask, but signal handlers are per-process.
# Check a process's current signal masks via /proc
cat /proc/$(pgrep nginx | head -1)/status | grep -E '^Sig'
# Output fields:
# SigPnd: pending signals (bitmask, hex)
# SigBlk: blocked signals
# SigIgn: ignored signals
# SigCgt: caught (handled) signals
# Decode the hex bitmask on Linux
python3 -c "mask=0x0000000000004000; [print(i+1) for i in range(64) if mask>>i&1]"SIGTERM: The Right Way to Stop a Process
SIGTERM (signal 15) is the standard termination request. Its default action is to terminate the process, but crucially, the process can catch it and run cleanup code before exiting. This is what you want 95% of the time. Properly written daemons catch SIGTERM, finish in-flight requests, flush write buffers, close database connections, release locks, and then exit cleanly.
Systemd sends SIGTERM as the first step in service shutdown. After the configurable TimeoutStopSec (default 90 seconds on most distros), it sends SIGKILL. Docker does the same: `docker stop` sends SIGTERM and waits 10 seconds before SIGKILL. Both of these workflows assume your application handles SIGTERM correctly. If it does not, you get dirty shutdowns, corrupted state, and orphaned temp files.
In our experience, the most common reason engineers reach for SIGKILL is that the target process has a broken or absent SIGTERM handler. Fix the application, not the signal choice. For processes you do not control, check whether the application documents a preferred shutdown signal - PostgreSQL, for example, uses SIGTERM for smart shutdown (wait for clients), SIGINT for fast shutdown (drop clients), and SIGQUIT for immediate shutdown with core dump.
# Graceful shutdown - always try this first
kill -TERM
# or equivalently
kill -15
kill # SIGTERM is the default
# PostgreSQL-specific shutdown signals
kill -TERM $(head -1 /var/lib/postgresql/data/postmaster.pid) # smart
kill -INT $(head -1 /var/lib/postgresql/data/postmaster.pid) # fast
kill -QUIT $(head -1 /var/lib/postgresql/data/postmaster.pid) # immediate + core
# Confirm a process exited cleanly
wait ; echo "exit: $?" SIGKILL: What It Does and When It Is Actually Appropriate
SIGKILL (signal 9) is handled entirely by the kernel. The kernel removes the process from the scheduler and releases its resources. The process itself never executes any code in response - there is no handler, no cleanup, no flush. File descriptors are closed by the kernel, but write buffers that the application manages in userspace (not kernel page cache) may not be flushed. Databases with write-ahead logs will recover; applications that write to files without proper fsync() may leave corruption.
SIGKILL is appropriate in three cases: the process has a broken SIGTERM handler and you have confirmed the data risk is acceptable, the process is consuming runaway resources and must stop immediately, or you are scripting a teardown that already accounted for dirty state. On our test server running a containerized workload, we use SIGKILL only after a 30-second SIGTERM window in shutdown scripts.
Note that `kill -9` does not work on zombie processes. A zombie (state Z) is already dead - it has exited but its parent has not called wait(2) to collect the exit status. The zombie holds no resources except a PID and a slot in the process table. To clear it, fix or signal the parent process.
# Only after SIGTERM fails or is inappropriate
kill -KILL
kill -9
# Kill an entire process group (note the negative PID)
kill -KILL -
# Find zombies
ps aux | awk '$8 == "Z"'
# Find the zombie's parent and inspect it
awk '/PPid/{print $2}' /proc//status
ps -p -o pid,stat,cmd SIGHUP: Reload, Not Hangup
SIGHUP (signal 1) originally meant the controlling terminal disconnected - a modem hangup in 1970s Unix. Its default action is process termination. But almost every long-running daemon redefines it as a configuration reload signal, because it was the first signal that felt semantically appropriate for "something changed, re-read your config."
nginx reloads its configuration on SIGHUP without dropping connections. Apache httpd does the same (graceful restart). syslogd and rsyslogd reopen log files on SIGHUP, which is how logrotate works - it renames the log file, sends SIGHUP, and the daemon creates a new file at the original path. If you forget the SIGHUP after a logrotate postrotate script, the daemon keeps writing to the renamed file via its open file descriptor.
nohup(1) works by setting SIGHUP disposition to ignore before exec-ing the child process. When you disconnect from a terminal, the shell sends SIGHUP to its process group. Processes ignoring SIGHUP continue running. This is also why tmux and screen sessions survive SSH disconnection - the shell is a child of the multiplexer, not the SSH session.
# Reload nginx config without dropping connections
kill -HUP $(cat /var/run/nginx.pid)
# or
nginx -s reload # sends SIGHUP internally
# Force rsyslogd to reopen log files after logrotate
kill -HUP $(cat /var/run/rsyslogd.pid)
# Check what SIGHUP does to a specific process (Linux)
grep -A5 'sighup\|SIGHUP' /proc//status # rarely useful
# Better: read the application's source or documentation
# Run a command immune to SIGHUP
nohup ./long-running-job.sh > /tmp/job.log 2>&1 & SIGINT and SIGQUIT: Terminal Control Signals
SIGINT (signal 2) is sent by the terminal driver when you press Ctrl+C. Its default action is termination. Almost every interactive program catches it to provide a clean exit - Python's REPL prints a newline and exits, bash loops abort the current command, and long-running CLI tools cancel their operation. The key difference from SIGTERM is that SIGINT originates from the terminal and goes to the foreground process group, hitting every process in the group simultaneously.
SIGQUIT (signal 3) is Ctrl+\. It terminates the process and generates a core dump by default. When debugging a hung process interactively, SIGQUIT is more useful than SIGKILL because the core dump gives you a stack trace. Java programs dump thread state to stderr on SIGQUIT rather than terminating - this is explicitly overridden in the JVM. On a Java service that appears frozen, `kill -QUIT
SIGTSTP (signal 20 on Linux) is Ctrl+Z, which stops the process and moves it to the background. SIGCONT (signal 18) resumes a stopped process. These four signals - SIGINT, SIGQUIT, SIGTSTP, SIGCONT - are the basis of Unix job control.
# Send SIGQUIT to get a Java thread dump without killing the process
kill -QUIT $(pgrep -f 'java.*MyApp')
# Thread dump appears on the process's stderr/stdout
# Suspend and resume a process
kill -TSTP # equivalent to Ctrl+Z
kill -CONT # resume it
# List jobs and their signals in bash
jobs -l
# Demonstrate SIGINT going to the whole foreground group
# Run: sleep 300 | cat
# Then Ctrl+C kills both sleep and cat SIGUSR1 and SIGUSR2: Application-Defined Signals
SIGUSR1 (signal 10) and SIGUSR2 (signal 12) are explicitly reserved for application use. The kernel has no built-in meaning for them. Their default action is termination, but any application that uses them catches them first and defines its own semantics.
Apache httpd uses SIGUSR1 for graceful restart. HAProxy uses SIGUSR1 to initiate a soft-stop (finish existing connections, accept no new ones). nginx uses SIGUSR1 to reopen log files (equivalent to what SIGHUP does in rsyslogd). Puma (the Ruby application server) uses SIGUSR2 to trigger a hot restart that replaces the process binary in place. The takeaway is that SIGUSR1 and SIGUSR2 behavior is entirely application-specific - always check documentation before sending them to an unfamiliar process.
In DevOps automation pipelines, SIGUSR signals are useful for triggering non-standard behaviors in custom daemons without adding a full management API. If you are building automation tooling around signal handling, platforms like taskbotshub.ai provide pipeline scaffolding that integrates signal-based process management with broader orchestration workflows, which reduces the boilerplate of building signal-aware shutdown sequences into CI/CD systems.
# HAProxy soft-stop: drain connections, then exit
kill -USR1 $(cat /var/run/haproxy.pid)
# nginx: reopen log files (for logrotate integration)
kill -USR1 $(cat /var/run/nginx.pid)
# Puma hot restart
kill -USR2 $(cat /var/run/puma.pid)
# In a custom bash daemon, trap SIGUSR1 for a status dump
trap 'echo "Processed: $count" >> /tmp/daemon-status.log' USR1
# Then trigger it from another terminal
kill -USR1 $(pgrep -f daemon.sh)Writing Signal Handlers That Do Not Break Things
Signal handlers execute asynchronously and can interrupt any point in the program's execution, including code that is not reentrant. The only functions that are safe to call from a signal handler are async-signal-safe functions, documented in signal-safety(7). This list includes write(2), _exit(2), kill(2), and a handful of others. It explicitly excludes printf(), malloc(), and most of the C standard library.
The correct pattern for a SIGTERM handler in C is to set a volatile sig_atomic_t flag and return. The main loop checks the flag and exits cleanly. This is not just academic - we have debugged production deadlocks caused by malloc() being called inside a signal handler while the main thread held malloc's internal lock. The handler blocked on the same lock, and the process froze.
In shell scripts, signal traps are simpler because bash's trap mechanism defers handler execution to between commands, avoiding reentrancy issues. Use `trap 'cleanup; exit 1' TERM INT` at the top of any script that creates temp files, starts child processes, or holds external resources. Without this, Ctrl+C leaves debris.
Python's signal module runs handlers in the main thread between bytecode instructions. The GIL makes this safe from most reentrancy issues, but you still cannot safely call certain C extensions from handlers. The standard pattern is identical to C: set a flag in the handler, check it in the main loop.
# Shell script with proper signal handling
#!/bin/bash
TMPDIR=$(mktemp -d)
cleanup() {
rm -rf "$TMPDIR"
echo "Cleaned up, exiting" >&2
}
trap 'cleanup; exit 1' TERM INT HUP
trap 'cleanup; exit 0' EXIT
# ... script body ...
# C: async-signal-safe SIGTERM handler pattern
# volatile sig_atomic_t got_sigterm = 0;
# void handle_sigterm(int sig) { got_sigterm = 1; }
# // In main loop:
# if (got_sigterm) { cleanup(); exit(0); }
# Python equivalent
import signal, sys
shutdown = False
def handle_term(signum, frame): global shutdown; shutdown = True
signal.signal(signal.SIGTERM, handle_term)
while not shutdown:
do_work()Sending Signals: kill, pkill, killall, and sigqueue
kill(1) sends a signal to a PID or process group. pkill(1) sends to processes matched by name, user, or other attributes. killall(1) on Linux is similar to pkill; on BSD, killall sends to all processes owned by a user. The behavior difference between Linux and BSD killall is a classic gotcha - `killall apache2` on Linux targets processes named apache2; `killall` on macOS/BSD without a name argument kills all processes you own.
Precision matters. pkill -x matches only exact process name matches, avoiding accidents when 'python' matches 'python3.12' and 'python3.12-config'. The -n flag sends to the newest matching process, -o to the oldest. When scripting signal delivery in automation, always verify the target with pgrep before pkill.
For real-time signals, sigqueue(3) allows passing an integer value alongside the signal. The receiving process accesses it via the si_value field in the siginfo_t structure. This is useful for custom daemons that need lightweight one-way messaging without a socket - you can encode a command code in the signal value. Standard signals delivered via kill() carry no payload beyond the signal number and sender PID.
# pkill with exact name match, dry run first
pgrep -xa 'nginx'
pkill -x nginx # only if pgrep output looks right
# Send to process group (all processes in the group)
kill -TERM -$(ps -o pgid= -p | tr -d ' ')
# Linux: send to all processes in a cgroup
cat /sys/fs/cgroup/system.slice/myapp.service/cgroup.procs | \
xargs -I{} kill -TERM {}
# Safer pkill: signal by PID file, not name
kill -HUP $(cat /var/run/myapp.pid)
# sigqueue from C (no shell equivalent)
# sigqueue(pid, SIGUSR1, (union sigval){ .sival_int = 42 }); Signals in Containers and init Systems
PID 1 inside a container is special. The kernel does not apply default signal dispositions to PID 1 - a signal with a default action of termination will be silently discarded if PID 1 has not installed a handler for it. This is why `docker stop` sometimes appears to hang: the container's PID 1 (often a bare application binary) ignores SIGTERM because the kernel is protecting the init process from accidental termination.
The solution is a proper init for containers. tini (used by Docker as --init) is 150 lines of C that registers handlers for SIGTERM and SIGCHLD, reaps zombie child processes, and forwards signals to child process groups. Using `docker run --init` or specifying tini as ENTRYPOINT fixes both the signal problem and the zombie reaping problem in one step.
Systemd handles signals at the unit level. The KillSignal directive sets which signal systemd sends first (default SIGTERM), and KillMode controls whether it signals only the main process, the entire control group, or the process group. Setting KillMode=cgroup ensures all child processes spawned by the service are signaled - critical for services that fork workers.
# Run container with proper init
docker run --init myimage
# Or use tini explicitly in Dockerfile
# ENTRYPOINT ["/tini", "--", "/app/server"]
# Check what PID 1 ignores inside a container
docker exec cat /proc/1/status | grep Sig
# Systemd unit: configure signal behavior
# /etc/systemd/system/myapp.service
# [Service]
# KillSignal=SIGTERM
# KillMode=cgroup
# TimeoutStopSec=30
# Send a signal to a systemd service without systemctl
systemctl kill --kill-who=main --signal=SIGHUP myapp.service
systemctl kill --kill-who=all --signal=SIGTERM myapp.service Debugging Signal Problems
strace(1) is the fastest way to observe signal delivery. `strace -e signal -p
For a process that is not responding to signals, check the SigBlk field in /proc/
gdb can inject signals into a running process: `signal SIGTERM` inside gdb will deliver the signal and let you observe handler execution. Combined with `info signal` to see how gdb is configured to handle each signal, this is the most precise debugging tool available. On production systems where gdb is not appropriate, perf and eBPF programs can trace signal delivery with near-zero overhead.
# Trace signals on a running process
strace -e trace=signal -p $(pgrep myapp)
# Decode SigCgt bitmask to see which signals are handled
python3 -c "
mask = int('$(awk '/SigCgt/{print $2}' /proc/$(pgrep myapp)/status)', 16)
names = {1:'HUP',2:'INT',3:'QUIT',9:'KILL',10:'USR1',12:'USR2',15:'TERM'}
for bit,name in names.items():
if mask >> (bit-1) & 1: print(f'SIG{name} is caught')
"
# Check if a process is in uninterruptible sleep (D state)
ps -o pid,stat,wchan -p
# wchan shows the kernel function where the process is sleeping
# Inject a signal via gdb (use with caution on production)
gdb -p -ex 'signal SIGUSR1' -ex detach -ex quit Signal Reference: The Ones You Will Actually Use
Here is the practical reference without the full table of 64 signals. SIGHUP (1): reload config or reopen files in daemons. SIGINT (2): interactive cancel, same as Ctrl+C. SIGQUIT (3): terminate with core dump, useful for thread dumps in Java. SIGKILL (9): unconditional kernel-enforced termination, no cleanup. SIGUSR1 (10): application-defined, check docs. SIGSEGV (11): segmentation fault, sent by kernel on invalid memory access. SIGUSR2 (12): application-defined, check docs. SIGPIPE (13): write to a broken pipe, default action is termination - frequently caught and ignored by network servers. SIGALRM (14): timer expiry from alarm(2), used for timeouts. SIGTERM (15): graceful shutdown request. SIGCHLD (17): child process changed state, used by init processes and shells. SIGCONT (18): resume a stopped process. SIGSTOP (19): unconditional stop, cannot be caught. SIGTSTP (20): terminal stop (Ctrl+Z), can be caught.
SIGPIPE deserves a mention because it causes unexpected crashes. If your server writes to a client that has closed the connection, the write() call triggers SIGPIPE. Default action is termination. Network servers should either set SO_NOSIGPIPE on the socket (BSD) or use MSG_NOSIGNAL in send() calls (Linux), or globally ignore SIGPIPE and handle the EPIPE errno from write() instead.
When naming long-running services and daemons that will eventually need to document their signal behavior to operators, consistent and recognizable naming matters from day one. Services with clear names are easier to document, script, and grep for in process lists. Tools like nicename.me can help when you are at the naming stage of a project and want to validate whether a service or domain name is distinctive and available before you commit it to deployment scripts and runbooks.
# Ignore SIGPIPE globally in a shell script that pipes to unreliable consumers
trap '' PIPE
# In C, ignore SIGPIPE before any network I/O
signal(SIGPIPE, SIG_IGN);
# Then check errno == EPIPE on write() failures
# SIGALRM for a shell timeout
(
sleep 5 && kill -ALRM $$
) &
trap 'echo "Timeout!"; exit 1' ALRM
wait # or the slow operation
# SIGCHLD: reap children in a daemon loop (C pseudocode)
# signal(SIGCHLD, SIG_IGN); // auto-reap on Linux
# or handle explicitly with waitpid()