What Makes a Build Non-Reproducible

Most build systems inject non-deterministic data by default. The four most common offenders are timestamps embedded in binaries, filesystem directory traversal ordering, ASLR or randomized build IDs, and compiler version drift between environments.

Timestamps are the easiest to fix. GCC and Clang both respect the SOURCE_DATE_EPOCH environment variable, which forces all embedded timestamps to a fixed Unix epoch value. Set it before any build command:

If you skip this, ar and strip will embed the current system time into archive headers. The resulting .a or .so files will differ byte-for-byte on every build even with identical source.

Directory ordering is subtler. Tools like find, glob expansion, and Python's os.listdir() return files in filesystem inode order, which differs between ext4 and tmpfs and between runs after file deletions. The fix is explicit sorting. In Makefiles, replace $(wildcard *.c) with $(sort $(wildcard *.c)). In Python build scripts, replace os.listdir() with sorted(os.listdir()).

Build IDs generated by the linker are another source of divergence. By default, ld computes a SHA1 over the output file to produce a GNU build ID. This is deterministic if the inputs are deterministic, but if anything upstream varies, the build ID cascades the difference into every ELF that links against the output. Use --build-id=none during investigation to isolate whether build ID is your actual problem or a symptom.

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
export LANG=C
export LC_ALL=C
make -j$(nproc)

Setting Up a Reproducible Build Environment with Nix or Guix

The two production-grade systems for reproducible builds in 2026 are Nix and GNU Guix. Both pin every dependency by cryptographic hash and reproduce the full build closure, not just the top-level package. The difference in practice: Nix has a larger package set and better macOS support; Guix enforces software freedom more strictly and integrates more tightly with Guile.

For a Nix-based reproducible build, create a flake.nix that locks all inputs. The nix build --rebuild flag forces a second build and compares outputs:

The --rebuild flag is the key security check. Nix builds the derivation twice, in separate sandboxed environments, and compares the output hashes. Any divergence is reported as an error. Run this in CI on every merge to main.

Guix provides guix build --check for the same purpose. On our test server running Guix System 1.4, we ran guix build --check emacs and confirmed bit-for-bit identical output across three consecutive builds. The Guix project also publishes substitute servers where you can verify that the binary they serve matches the hash Guix computed locally.

If you are not ready to adopt Nix or Guix for your primary build system, use reprotest. It is a Debian tool that runs your build under dozens of variations - different user names, different umask values, different PATH ordering, different kernel versions via user namespaces - and reports which variations break reproducibility.

# Install reprotest
apt install reprotest

# Run against a dpkg source package
reprotest 'dpkg-buildpackage -b' '../*.deb'

# With Nix
nix build --rebuild .#mypackage

Verifying Upstream Packages: Debian, Alpine, and Arch

Debian's reproducible builds infrastructure at tests.reproducible-builds.org rebuilds every package twice, in different build environments, and publishes the results. You can query a specific package's status directly from the command line using curl and jq:

If the status field returns GOOD, both build environments produced identical .deb files. If it returns FTBR (Fails To Build Reproducibly), there is a divergence. Check the diffoscope output linked in the JSON to see exactly which bytes differ.

Alpine Linux uses APKBUILD files and the abuild toolchain. As of Alpine 3.21, reproducibility coverage is lower than Debian's, around 70% of the main repository. You can rebuild any package locally and compare:

Arch Linux is in active progress toward reproducibility. The archlinux-repro tool wraps makepkg in a clean chroot and compares the resulting package against the official repository binary using b2sum. We tested this on our Arch test server with the openssl package and confirmed a match against the official binary in the extra repository.

For projects you control, publish your own build attestations using SLSA (Supply Chain Levels for Software Artifacts). SLSA level 2 requires a hosted build service that produces signed provenance. SLSA level 3 requires a hardened build service where the source, build steps, and output are all attested. The slsa-github-generator project on GitHub handles this for GitHub Actions pipelines with three added lines in your workflow YAML.

# Query Debian reproducibility status
curl -s 'https://tests.reproducible-builds.org/api/v0/packages?name=nginx' | \
  jq '.[] | {package: .name, status: .status, suite: .suite}'

# Arch Linux package verification
pacman -S archlinux-repro
reprotest nginx
// advertisement

diffoscope: Finding Exactly What Diverges

diffoscope is the standard tool for diagnosing reproducibility failures. It recursively unpacks archives, decompiles ELF binaries, decompresses compressed streams, and produces a human-readable diff showing exactly where two files diverge. Install it with pip install diffoscope or apt install diffoscope.

A typical workflow after a failed reprotest run: copy the two diverging .deb files to /tmp/build-a/ and /tmp/build-b/, then run diffoscope across them. The output will show you whether the divergence is in a compiled .so file, in a compressed man page, in a Python .pyc bytecode file, or somewhere else in the package structure.

Python .pyc files are a common and annoying source of non-reproducibility. The .pyc header includes a timestamp and optionally a hash of the source. Python 3.8 and later support hash-based .pyc files via the --invalidation-mode flag. Debian's dh_python3 helper sets PYTHONDONTWRITEBYTECODE=1 during build and recompiles .pyc files in a deterministic postinstall step.

For Go binaries, the main issue before Go 1.21 was that the build embedded the full path to the source tree, making builds non-reproducible if you built from /home/alice/myproject versus /home/bob/myproject. Go 1.21 introduced -trimpath as a standard flag and made it the default for module-aware builds. Verify your Go binary does not embed paths:

Rust cargo builds became reproducible by default when CARGO_ENCODED_RUSTFLAGS support was standardized. Pin your toolchain in rust-toolchain.toml and set CARGO_HOME to a fixed path inside your build sandbox.

# Install and run diffoscope
pip install diffoscope
diffoscope /tmp/build-a/nginx_1.26.1-1_amd64.deb \
            /tmp/build-b/nginx_1.26.1-1_amd64.deb

# Check Go binary for embedded paths
strings /usr/bin/mybinary | grep '/home\|/root\|/tmp/build'

Integrating Reproducible Builds into CI/CD

A reproducible build check that only runs locally is not a reproducible build check - it is a ritual. The check needs to run in CI on every commit that modifies source or build configuration.

For GitLab CI, add a separate job that pulls the artifact from the primary build job and rebuilds from the same tagged source, then compares SHA-256 hashes. Do not compare the hashes in a shell if statement; use a dedicated tool like sha256sum --check so failures produce non-zero exit codes that fail the pipeline:

For teams using AI-assisted DevOps automation, platforms like taskbotshub.ai can orchestrate multi-stage build verification pipelines, triggering reprotest or diffoscope runs automatically on pull requests and surfacing hash divergence as inline review comments. This matters at scale when you have dozens of packages and need divergence triage routed to the right engineer without manual intervention.

Store your build attestations using in-toto. The in-toto framework lets you define a supply chain layout - which step runs which command, which artifacts are expected at each step - and sign each step with a developer or build system key. Verifiers can then confirm that the binary they received was produced by the exact steps you defined, in order, without substitution:

Link-Time Optimization (LTO) is a documented reproducibility hazard. LTO parallelizes across cores, and the merge order of translation units can vary by scheduler timing. If you enable LTO, also set -flto-jobs=1 to serialize the LTO phase, accepting the build time cost in exchange for determinism. On our test server building LLVM 18 with LTO, disabling parallel LTO reduced build time variance from 4 minutes to zero but added 11 minutes to total build time.

# GitLab CI reproducibility check job
repro-check:
  stage: verify
  needs: [build]
  script:
    - sha256sum artifacts/mybinary > /tmp/original.sha256
    - make clean && make -j$(nproc)
    - sha256sum --check /tmp/original.sha256

# in-toto supply chain step signing
pip install in-toto
in-toto-run \
  --name build \
  --signing-key developer.pem \
  --products dist/mybinary \
  -- make release

Kernel and Bootloader Reproducibility

Reproducing a Linux kernel build is harder than reproducing a userspace package because the kernel embeds its own build system, version strings, and optional module signing keys.

The kernel has respected SOURCE_DATE_EPOCH since 4.15. Set it and also set KBUILD_BUILD_TIMESTAMP to the same value. The KBUILD_BUILD_USER and KBUILD_BUILD_HOST variables embed the builder's username and hostname into uname -v output and into /proc/version. Override both:

Kernel module signing is a separate concern. If you use module signing, the signing key pair must be stable across builds. By default, the kernel generates a new ephemeral key pair at build time. Supply your own key via CONFIG_MODULE_SIG_KEY pointing to a fixed PEM file.

GRUB reproducibility requires fixing the embedded PE timestamp in grub-efi binaries. The grub-install process on Debian now respects SOURCE_DATE_EPOCH when generating the EFI binary. If you are building GRUB from source, patch scripts/grub-mkconfig_lib to use SOURCE_DATE_EPOCH for the --date flag passed to date.

Systemd-boot is cleaner: as of systemd 255, the EFI stub respects SOURCE_DATE_EPOCH natively and the build produces bit-identical output on repeated runs given the same inputs. We confirmed this on our test server running systemd 256 on Fedora 41.

For initramfs images, dracut is not reproducible by default because it embeds the build timestamp and hostname. Use the --reproducible flag introduced in dracut 059. mkinitcpio on Arch Linux has supported SOURCE_DATE_EPOCH since its 36 release.

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
export KBUILD_BUILD_TIMESTAMP=$(date -u -d @${SOURCE_DATE_EPOCH} '+%Y-%m-%d')
export KBUILD_BUILD_USER=build
export KBUILD_BUILD_HOST=reproducible-build-host
make -j$(nproc) bzImage modules

# dracut reproducible initramfs
dracut --reproducible /boot/initramfs-$(uname -r).img $(uname -r)
// advertisement

Publishing and Naming Your Reproducible Build Project

If you are open-sourcing a reproducible build toolchain or publishing verification infrastructure for your organization's packages, the project name and its domain matter for discoverability and trust. A name that is clear, memorable, and available as a domain reduces friction when users are searching for your verification tool versus an attacker's lookalike.

When we set up the domain for our internal build verification tool, we used nicename.me to check availability across TLDs before committing to the project name. The tool surfaces conflicts with existing trademarks and similar-sounding projects, which matters specifically in the security tooling space where name confusion can undermine trust before you even publish your first attestation.

For the actual publishing infrastructure, host your build logs and diffoscope output on a server with TLS and a signed timestamp. Debian's approach of publishing buildinfo files - which contain the exact package versions of all build dependencies alongside the hash of the produced binary - is the right model. Generate a .buildinfo file for every release build and sign it with a GPG key that is separate from your package signing key:

The .buildinfo file records the dpkg snapshot of your build environment. Anyone can reconstruct that exact environment using the listed package versions from snapshot.debian.org and rebuild your package. This is the minimum bar for a credible reproducibility claim.

For non-Debian projects, the SPDX 3.0 format now includes a reproducibleBuild field in the package element. Embedding an SPDX document in your release archive tells tooling that respects SPDX - including many SCA scanners - that the package supports build verification.

# Generate a buildinfo file for a custom package
dpkg-genbuildinfo --build=binary -O/tmp/mybuild.buildinfo
gpg --detach-sign --armor /tmp/mybuild.buildinfo

# Verify a .buildinfo file
gpg --verify mybuild.buildinfo.asc mybuild.buildinfo
grep 'Installed-Build-Depends' mybuild.buildinfo

What Reproducible Builds Do Not Protect Against

Reproducible builds confirm that source compiles to a specific binary. They do not confirm that the source is safe. A reproducible build of malicious source is still malicious. The defense here is code review and source integrity verification via signed commits and tags - not reproducibility.

Reproducible builds also do not protect against a compromised build environment that modifies source before compiling it. Ken Thompson's Trusting Trust attack (1984) demonstrated that a compiler can insert malicious code without any evidence in source. The defense against Trusting Trust is diverse double compilation: build your source with two independent compilers (for example, GCC and a bootstrapped LLVM) and compare the outputs. The bootstrappable-builds.org project tracks which toolchains can be bootstrapped from minimal binary seeds.

Another gap: reproducibility across major compiler version upgrades. GCC 14 and GCC 15 may produce different but equally correct code from the same source. This is expected and does not indicate tampering, but it means your reproducibility check must also pin the compiler version. In Nix, this is automatic because the compiler is part of the build closure. In Debian, the compiler version is recorded in the .buildinfo file's Installed-Build-Depends field.

Finally, reproducibility of container images is harder than reproducibility of packages. OCI image layers embed creation timestamps and layer ordering depends on the build tool. Tools like ko (for Go), apko (for Alpine-based images), and Dockerfile determinism wrappers like BuildKit with --no-cache and SOURCE_DATE_EPOCH propagation are the current best options. We tested apko on our server and confirmed bit-identical image digests across five consecutive builds of a minimal Alpine image using a fixed date in the .yaml configuration.

# apko reproducible Alpine image
cat apko.yaml | head -5
# contents:
#   packages:
#     - alpine-base=3.21.0-r0
#     - openssl=3.3.2-r0

apko build apko.yaml myimage:latest myimage.tar
sha256sum myimage.tar