Installing Valgrind and Understanding the Toolchain
Install Valgrind from your distro package or build from source. The distro package on Debian/Ubuntu is usually one or two minor versions behind, which rarely matters for leak detection but does matter for newer ARM or RISC-V targets.
On Debian 12 or Ubuntu 24.04:
``` apt install valgrind valgrind-dbg ```
On RHEL 9 / AlmaLinux 9:
``` dnf install valgrind valgrind-devel ```
Valgrind is not a single tool. It is a framework. The tools you care about for memory problems are:
- **Memcheck** - the default, finds leaks, invalid reads/writes, use of uninitialised memory - **Massif** - heap profiler, tells you which call site is allocating the most memory over time - **DHAT** - dynamic heap analysis tool, finds memory that is allocated but rarely or never accessed
For leak detection in the classic sense, Memcheck is what you run first. Massif is what you run when Memcheck tells you there is a leak but the call stack is not enough to understand why the allocation keeps growing.
Verify your installation and check which tools are available:
``` valgrind --version ls /usr/lib/valgrind/ ```
You should see files like `memcheck-amd64-linux`, `massif-amd64-linux`, and so on. If the `valgrind-dbg` package is missing, the tool still works but stack traces for glibc internals will be incomplete.
apt install valgrind valgrind-dbg
valgrind --versionCompiling Your Binary for Useful Valgrind Output
Valgrind works on any ELF binary, but without debug symbols the output is nearly useless. You get addresses instead of function names, and you cannot pin the leak to a line number.
Compile with `-g` and disable optimisations that reorder or eliminate variables:
``` gcc -g -O1 -fno-omit-frame-pointer -o myapp myapp.c ```
Use `-O1` rather than `-O0` because `-O0` sometimes produces false positives due to the way uninitialised stack slots behave. `-O1` is close enough to your production build to be representative while keeping symbols intact.
If you are linking against third-party shared libraries that ship without debug symbols, install the debug symbol packages. On Debian:
``` apt install libc6-dbg libssl3-dbg ```
On RHEL-based systems the package convention is `*-debuginfo`:
``` dnf debuginfo-install glibc openssl ```
For CMake projects, set the build type:
``` cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS="-fno-omit-frame-pointer" .. ```
In our experience, `RelWithDebInfo` is the most practical choice for profiling production-like code: it keeps `-g` and uses `-O2` with debug info, so stack traces are accurate even with some inlining.
gcc -g -O1 -fno-omit-frame-pointer -o myapp myapp.cRunning Memcheck: The Right Flags
The default `valgrind ./myapp` invocation runs Memcheck but with settings that miss a category of leaks called "still reachable" - memory that was allocated, never freed, but still has a live pointer at exit. For security-sensitive daemons this matters because such memory can hold credentials or session tokens.
The flags we use on every initial run:
``` valgrind \ --tool=memcheck \ --leak-check=full \ --show-leak-kinds=all \ --track-origins=yes \ --verbose \ --log-file=valgrind-$(date +%Y%m%d-%H%M%S).log \ ./myapp --your-app-args ```
What each flag does:
- `--leak-check=full` reports every individual leaked block, not just a summary - `--show-leak-kinds=all` includes definitely lost, indirectly lost, possibly lost, and still reachable - `--track-origins=yes` traces uninitialised values back to their source allocation - adds ~50% overhead but is essential for debugging - `--log-file` writes to a timestamped file so you can compare runs
Valgrind will slow your binary by 10x to 50x. For short-lived test binaries this is irrelevant. For a daemon that forks workers, use `--trace-children=yes` and `--trace-children-skip=/usr/bin/sh,/bin/sh` to avoid polluting output with shell subprocess noise.
If your application reads from a config file or socket and terminates cleanly, the log will contain a LEAK SUMMARY at the end. A clean run looks like:
``` ==12345== LEAK SUMMARY: ==12345== definitely lost: 0 bytes in 0 blocks ==12345== indirectly lost: 0 bytes in 0 blocks ==12345== possibly lost: 0 bytes in 0 blocks ==12345== still reachable: 1,024 bytes in 3 blocks ==12345== suppressed: 0 bytes in 0 blocks ```
"Still reachable" from glibc's own allocator internals is normal. Everything under "definitely lost" or "indirectly lost" is a bug you own.
valgrind \
--tool=memcheck \
--leak-check=full \
--show-leak-kinds=all \
--track-origins=yes \
--verbose \
--log-file=valgrind-$(date +%Y%m%d-%H%M%S).log \
./myapp --your-app-argsReading Valgrind Output: What the Stack Traces Mean
A typical Memcheck leak report looks like this:
``` ==12345== 48 bytes in 1 blocks are definitely lost in loss record 3 of 12 ==12345== at 0x4848899: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==12345== by 0x10922F: create_session (session.c:87) ==12345== by 0x109A41: handle_request (server.c:214) ==12345== by 0x109C10: worker_loop (server.c:301) ==12345== by 0x109E02: thread_entry (server.c:412) ```
The number after `==` is the PID. The stack reads bottom-up: `thread_entry` called `worker_loop`, which called `handle_request`, which called `create_session` at line 87 of `session.c`, which called `malloc`. That allocation was never freed.
Open `session.c:87` and look at what `create_session` allocates. In our test server, the pattern was a `struct session *` allocated inside every request handler but only freed on the success path - the error path returned early without freeing. Classic and common.
For "indirectly lost" reports: these are allocations made by a struct that itself was directly lost. Fix the parent allocation and the indirect leak disappears. Do not chase indirect leaks in isolation.
For "possibly lost": Valgrind found a pointer to the middle of an allocated block, not the start. This often means a pointer was incremented past the base, which is common in hand-rolled string builders. It can be a real bug or it can be intentional. Check the specific allocation site.
To cross-reference source lines quickly, use `addr2line`:
``` addr2line -e ./myapp -f 0x10922F ```
addr2line -e ./myapp -f 0x10922FWriting Suppression Files to Reduce Noise
Third-party libraries, glibc internals, OpenSSL, and Python embedding all produce Valgrind reports that are not your bugs. Without suppressions, a single OpenSSL `SSL_CTX_new` call generates a page of "still reachable" output that drowns out your actual leaks.
Generate a suppression file from an existing run:
``` valgrind \ --leak-check=full \ --show-leak-kinds=all \ --gen-suppressions=all \ --log-file=suppressions-raw.log \ ./myapp ```
Extract the suppression blocks from the log:
``` grep -A 20 '{' suppressions-raw.log | grep -v '^--' > myapp.supp ```
Edit `myapp.supp` manually. Each block looks like:
``` { openssl_ctx_leak Memcheck:Leak match-leak-kinds: reachable fun:malloc fun:CRYPTO_malloc fun:SSL_CTX_new } ```
Give each suppression a meaningful name. Then run with suppressions applied:
``` valgrind \ --leak-check=full \ --show-leak-kinds=all \ --suppressions=myapp.supp \ --log-file=clean-run.log \ ./myapp ```
Valgrind ships with pre-built suppression files for glibc and pthreads. On Debian they are in `/usr/lib/valgrind/`. Check what is available:
``` ls /usr/lib/valgrind/*.supp ```
Reference them directly:
``` valgrind --suppressions=/usr/lib/valgrind/debian.supp --suppressions=myapp.supp ./myapp ```
Keep your suppression file in version control alongside your source. It is documentation of known third-party behaviour, not a way to hide your own bugs.
ls /usr/lib/valgrind/*.supp
valgrind --suppressions=/usr/lib/valgrind/debian.supp --suppressions=myapp.supp ./myappUsing Massif for Heap Growth Profiling
When Memcheck confirms a leak but the call site points to a generic allocator wrapper, Massif gives you the heap usage over time as a series of snapshots. This is the right tool when memory grows by 50MB per hour but there is no "definitely lost" block at exit because the pointers are still reachable through a growing cache or list.
Run Massif:
``` valgrind \ --tool=massif \ --pages-as-heap=yes \ --massif-out-file=massif.out.%p \ ./myapp ```
`--pages-as-heap=yes` includes memory allocated via `mmap` directly, not just `malloc`. Without it, you miss anonymous mappings used by some allocators and by JVM-based tools.
After the run, visualise with `ms_print`:
``` ms_print massif.out.12345 | head -100 ```
The output is a text-based chart with peak heap usage and a ranked list of call stacks responsible for each snapshot. The top entry is almost always your leak site.
For a more interactive view, install `massif-visualizer` if you have a desktop available:
``` apt install massif-visualizer massif-visualizer massif.out.12345 ```
On a headless server, forward X11 with `ssh -X` or export the SVG:
``` ms_print massif.out.12345 > massif-report.txt ```
In our experience on test server runs lasting 30 minutes each, Massif identified the specific cache implementation that was never evicting entries - a `g_hash_table` in a GLib-based daemon that grew without bound because the eviction callback had a logic error and silently returned without removing entries.
valgrind \
--tool=massif \
--pages-as-heap=yes \
--massif-out-file=massif.out.%p \
./myapp
ms_print massif.out.12345 | head -100Integrating Valgrind into CI Pipelines
Running Valgrind manually is useful for investigation. Running it automatically on every merge request is what prevents regressions. The key is making Valgrind exit with a non-zero code when leaks are found, so your CI system treats it as a test failure.
Use `--error-exitcode`:
``` valgrind \ --tool=memcheck \ --leak-check=full \ --show-leak-kinds=definite,indirect \ --error-exitcode=1 \ --suppressions=myapp.supp \ ./myapp --ci-mode ```
Note we narrowed `--show-leak-kinds` to `definite,indirect` for CI. Checking `possibly lost` and `still reachable` in CI generates too many false positives from third-party libraries that vary across distro versions. Tighten this as your suppression file matures.
In a GitLab CI YAML:
```yaml memcheck: stage: test image: debian:12 script: - apt-get update -qq && apt-get install -y valgrind build-essential - make CFLAGS="-g -O1 -fno-omit-frame-pointer" - valgrind --tool=memcheck --leak-check=full --show-leak-kinds=definite,indirect --error-exitcode=1 --suppressions=myapp.supp ./myapp --ci-mode artifacts: when: on_failure paths: - valgrind-*.log ```
For teams using DevOps automation platforms, tools like taskbotshub.ai can schedule nightly Valgrind runs against staging builds and post results directly to your incident channel, which is more practical than running the full Massif suite on every commit.
For long-running daemon tests, the `--ci-mode` flag in the example above is an application-level argument you implement to make the daemon process a fixed number of requests and exit cleanly. Valgrind needs a clean exit to produce a complete leak summary. A daemon that must be killed with SIGKILL will produce incomplete results.
valgrind \
--tool=memcheck \
--leak-check=full \
--show-leak-kinds=definite,indirect \
--error-exitcode=1 \
--suppressions=myapp.supp \
./myapp --ci-modeValgrind with Multithreaded Applications
Valgrind's Memcheck is thread-safe and detects races that result in memory corruption, but it does not replace Helgrind for pure data race detection. For multithreaded leak detection, the main difference is that you need to ensure all threads join cleanly before the process exits, or Valgrind will report allocations from running threads as leaks.
Use `--fair-sched=yes` to reduce scheduling non-determinism during testing:
``` valgrind \ --tool=memcheck \ --fair-sched=yes \ --leak-check=full \ --show-leak-kinds=all \ ./myapp-threaded ```
For thread-specific leaks, the PID in the Valgrind output is always the process PID, not the thread ID. To correlate a leak to a specific thread, add `--num-callers=30` to get a deeper stack trace that includes the thread entry point:
``` valgrind --tool=memcheck --num-callers=30 --leak-check=full ./myapp-threaded ```
The default call depth is 12, which is often not enough to distinguish between two similar thread pool implementations calling the same allocation wrapper.
For OpenMP applications, set the thread count explicitly to make runs reproducible:
``` OMP_NUM_THREADS=4 valgrind --tool=memcheck --leak-check=full ./myapp-omp ```
We tested this against a network proxy written in C with 16 worker threads on our test server. Without `--fair-sched=yes`, the leak report changed between runs because different threads were still alive at exit. With `--fair-sched=yes` and a clean shutdown sequence in the test harness, results were consistent across 10 consecutive runs.
valgrind \
--tool=memcheck \
--fair-sched=yes \
--num-callers=30 \
--leak-check=full \
--show-leak-kinds=all \
./myapp-threadedUsing DHAT to Find Allocated-but-Unused Memory
DHAT (Dynamic Heap Analysis Tool) is underused. It does not find classic leaks - it finds memory that is allocated, held for the entire program lifetime, but read zero or one time. This is the category of waste that causes the "memory keeps growing but Memcheck finds nothing" complaint.
Run DHAT:
``` valgrind \ --tool=dhat \ --dhat-out-file=dhat.out.%p \ ./myapp ```
Open the output in the DHAT viewer. As of Valgrind 3.21+, the viewer is a standalone HTML file:
``` firefox /usr/lib/valgrind/dh_view.html ```
Load the `dhat.out.*` file through the browser interface. The report ranks allocation sites by total bytes allocated, bytes at peak, bytes read, and bytes written. An allocation site with 50MB allocated and 100 bytes read is almost certainly a bug.
For headless analysis, use the `dhat-to-text` script if available, or parse the JSON output directly:
``` python3 /usr/share/doc/valgrind/dhat-to-text.py dhat.out.12345 ```
DHAT is particularly effective at finding overprovisioned read buffers - situations where a network server allocates a 64KB buffer per connection expecting large payloads, but 95% of actual payloads are under 1KB. The memory is technically "used" so Memcheck does not flag it, but DHAT shows the read ratio and you can size the buffer appropriately.
valgrind \
--tool=dhat \
--dhat-out-file=dhat.out.%p \
./myapp
firefox /usr/lib/valgrind/dh_view.htmlCombining Valgrind with AddressSanitizer and Other Tools
Valgrind and AddressSanitizer (ASan) are not interchangeable. ASan runs at roughly 2x overhead versus Valgrind's 10-50x. ASan is compiled into the binary; Valgrind wraps any binary without recompilation. Use ASan in your regular test suite for fast feedback, use Valgrind for deep investigation.
You cannot run ASan and Valgrind simultaneously on the same binary - ASan intercepts the same allocator hooks Valgrind instruments, and the combination crashes. Build two separate binaries:
``` # ASan build for fast CI gcc -g -O1 -fsanitize=address,leak -fno-omit-frame-pointer -o myapp-asan myapp.c
# Debug build for Valgrind gcc -g -O1 -fno-omit-frame-pointer -o myapp-valgrind myapp.c ```
For Rust projects, Valgrind works on the compiled binary but you need to disable the Rust allocator's custom hooks. Add to `Cargo.toml`:
```toml [profile.dev] debug = true opt-level = 1 ```
And run:
``` CALLGRIND_OPTS="" valgrind --tool=memcheck --leak-check=full target/debug/myapp ```
For Python C extensions, run the interpreter under Valgrind with the Python suppression file:
``` valgrind \ --tool=memcheck \ --leak-check=full \ --suppressions=/usr/share/doc/python3.12/valgrind-python.supp \ python3 test_myextension.py ```
The Python suppression file is installed with the `python3.12-dev` package on Debian. Without it, CPython's own object allocator generates hundreds of false leak reports.
# ASan build
gcc -g -O1 -fsanitize=address,leak -fno-omit-frame-pointer -o myapp-asan myapp.c
# Valgrind build
gcc -g -O1 -fno-omit-frame-pointer -o myapp-valgrind myapp.c
# Python extension
valgrind \
--tool=memcheck \
--leak-check=full \
--suppressions=/usr/share/doc/python3.12/valgrind-python.supp \
python3 test_myextension.pyPractical Checklist Before Filing a Bug Report
When you find a leak and need to report it upstream or assign it internally, collect this information before writing the ticket:
1. Valgrind version: `valgrind --version` 2. Compiler and flags used to build the binary 3. Distro and kernel version: `uname -a && cat /etc/os-release` 4. The full Valgrind log file with `--leak-check=full --show-leak-kinds=all --track-origins=yes` 5. The suppression file used, so the reviewer can reproduce without your local third-party noise 6. The minimum reproducer - the smallest input or request sequence that triggers the leak
For the reproducer, use Valgrind's `--vgdb=yes` and `--vgdb-error=1` flags to pause execution at the first error and attach `gdb` for live inspection:
``` valgrind --tool=memcheck --vgdb=yes --vgdb-error=1 ./myapp ```
In a second terminal:
``` gdb ./myapp (gdb) target remote | vgdb (gdb) monitor leak_check full reachable any ```
This lets you inspect the live heap at the exact moment the first leak is detected, including viewing the contents of the leaked allocation with `(gdb) x/32xb addr`.
Version control your suppression files alongside your test suite. If your project has a public name or you are releasing an open source tool built around these diagnostics, a clean, memorable project name matters for discoverability - services like nicename.me can help you find an available domain that matches your tool name before you publish.
valgrind --tool=memcheck --vgdb=yes --vgdb-error=1 ./myapp