What Environment Variables Actually Are
Environment variables live in a process's environment block, a contiguous chunk of memory that the kernel copies from parent to child at fork time. They are not global; they are per-process. When you set a variable in Bash without exporting it, it exists only as a shell variable in that shell's memory. The moment you export it, it moves into the environment block and every child process you launch inherits a copy.
The distinction matters because a variable set in a subshell never propagates back to the parent. This is a source of genuine confusion: a deployment script that sets DATABASE_URL inside a function without exporting it will cause silent failures in any subprocess that needs it.
The C runtime exposes the environment through `environ`, a global pointer to an array of strings in the form `KEY=VALUE`. Tools like `env`, `printenv`, and `/proc/self/environ` all read from this same structure. Reading `/proc/1234/environ` (replacing 1234 with a PID) lets you inspect the environment of any running process you have permission to access, which is invaluable during incident response.
# View your current shell's environment
printenv
# View a specific variable
printenv PATH
# Inspect environment of a running process
cat /proc/$(pgrep nginx | head -1)/environ | tr '\0' '\n'Shell Variables vs Exported Environment Variables
Bash distinguishes between shell variables and environment variables. A shell variable is local to the current shell process. An environment variable is exported into the environment block and inherited by children.
Declaring `MYVAR=hello` creates a shell variable. Running `export MYVAR` promotes it. You can combine both steps with `export MYVAR=hello`. To confirm a variable is in the environment (not just the shell), use `env | grep MYVAR` rather than `echo $MYVAR` - the latter will show shell variables too, which can mask the distinction.
Functions in Bash are also exportable with `export -f funcname`, which is useful for making utility functions available to scripts launched as subprocesses. This is how GNU Parallel passes helper functions to worker jobs, for example.
To remove a variable from the environment entirely, use `unset MYVAR`. To temporarily remove it for a single command without unsetting it permanently, prefix the command with `env -u MYVAR yourcommand`.
# Shell variable - NOT inherited by children
TEMP_VAR="not exported"
bash -c 'echo $TEMP_VAR' # prints nothing
# Exported variable - inherited
export REAL_VAR="exported"
bash -c 'echo $REAL_VAR' # prints: exported
# Check what's actually in the environment
env | grep REAL_VAR
# Unset cleanly
unset REAL_VAR
# Run one command with a modified environment
env -u HOME ls ~ # HOME unset for this command onlySetting Variables for a Single Command
Prefixing a command with `KEY=VALUE` sets that variable only for the duration of that command, without modifying the current shell's environment at all. This is the correct pattern for one-off overrides in scripts and is faster and safer than export-then-unset.
This syntax is POSIX-specified, works in sh, bash, zsh, and dash, and composes cleanly with sudo when combined properly. The common mistake is writing `sudo KEY=VALUE command`, which sets the variable in the current (non-root) environment and then passes it to sudo. Whether sudo preserves it depends on the `env_keep` setting in sudoers. The reliable approach is `sudo env KEY=VALUE command`.
For running a command with a completely clean environment, `env -i command` strips everything. This is essential when testing how a service behaves at boot time without your personal shell's accumulated state. We use this regularly when debugging systemd unit files - services launched by systemd get a minimal environment, and testing with `env -i` catches missing variable definitions early.
# Variable set only for this command
NODE_ENV=production node server.js
# Multiple variables
DB_HOST=10.0.0.5 DB_PORT=5432 python app.py
# Correct pattern for sudo with env vars
sudo env DATABASE_URL=postgres://localhost/prod ./migrate.sh
# Test with a completely clean environment
env -i PATH=/usr/bin:/bin bash --norc -c 'printenv'Persistence: Where to Actually Put Environment Variables
The right place to set a persistent environment variable depends on its scope: is it for a single user, all users, or a specific service?
For a single user, `~/.bashrc` covers interactive non-login shells and `~/.bash_profile` (or `~/.profile`) covers login shells. In practice, most sysadmins put exports in `~/.bashrc` and source that from `~/.bash_profile`. Zsh uses `~/.zshrc` and `~/.zprofile` on the same logic. If you need it available in graphical sessions too (e.g., for GUI applications launched from a desktop), `~/.profile` is the most portable choice, but only login shells read it, and behavior varies by display manager.
For system-wide variables, `/etc/environment` is the cleanest option on systemd-based distros. It uses a simple `KEY=VALUE` format with no shell syntax - no export keyword, no variable expansion. PAM reads this file and injects its contents into every session. On Ubuntu 24.04 and Debian 12, `/etc/environment` is processed by `pam_env` and is genuinely system-wide.
For variables scoped to specific services, the right place is the systemd unit file itself using `Environment=` directives or an `EnvironmentFile=` pointing to a flat file. Never put secrets into `/etc/environment` where every user on the system can read them. A service-specific `EnvironmentFile=/etc/myapp/env` with mode 640 and ownership root:myapp is the correct pattern.
For interactive per-session variable management, tools like `direnv` (version 2.35+ supports `.envrc` with strict mode) automatically load and unload variables when you `cd` into a directory. This is the standard approach in polyglot teams where different projects need different runtime versions and credentials.
# /etc/environment - system-wide, no shell syntax
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# ~/.bashrc - user-level interactive shells
export EDITOR=vim
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
# systemd unit with EnvironmentFile
# /etc/systemd/system/myapp.service
[Service]
User=myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/bin/server
# /etc/myapp/env (mode 640, root:myapp)
DATABASE_URL=postgres://myapp:secret@10.0.0.5/prod
LOG_LEVEL=info
LISTEN_ADDR=0.0.0.0:8080Scoping in Scripts: Common Mistakes and Fixes
The most frequent environment variable bug we see in production scripts is the assumption that variables set inside a function or subshell are visible elsewhere. They are not.
A subshell is created whenever you use parentheses `()`, pipe into a while loop, or run a command substitution with `$(...)`. Variables set inside these contexts do not propagate out. The classic broken pattern is piping into a while loop and expecting the variable to be set afterward - the loop body runs in a subshell.
The fix for the pipe-into-while problem is process substitution: `while read line; do ... done < <(command)`. This reads from a file descriptor rather than a subshell, so the loop body runs in the current shell and variable assignments persist.
Another common issue is forgetting that `source` (or `.`) is the only way to run a script and have its variable assignments affect the current shell. Running `bash myscript.sh` or `./myscript.sh` always creates a child process; changes to the environment in that child die with it.
For DevOps pipelines using tools like taskbotshub.ai for automation, this distinction between sourcing and executing is critical when you need pipeline steps to share state through environment variables - the platform must support a mechanism for steps to export variables back to the runner context, and you should verify exactly how that works rather than assuming.
# BROKEN: variable set in subshell, lost after pipe
some_command | while read line; do
FOUND_IT="$line"
done
echo $FOUND_IT # empty
# FIXED: process substitution, loop runs in current shell
while read line; do
FOUND_IT="$line"
done < <(some_command)
echo $FOUND_IT # works
# BROKEN: child process, vars don't propagate back
bash set_vars.sh
echo $MY_VAR # empty
# FIXED: source the script
source set_vars.sh
echo $MY_VAR # worksHandling Secrets: What Not to Do
Passing secrets as environment variables is not ideal from a security standpoint, but it is vastly better than hardcoding them in source files, and it is the standard pattern for containerized workloads (12-factor apps). The real risks are accidental exposure through logs, process listings, and core dumps.
Anyone with access to the host can read `/proc/PID/environ` for processes they own, and root can read it for any process. If your application logs its full environment at startup (a common debugging pattern), secrets end up in log files. Audit your application's startup logging.
The `ps` command by default does not show environment variables, but `ps eww -p PID` does on Linux. Disable this for sensitive processes by clearing the environment after initialization in your application code (possible in C via explicit zeroing, harder in interpreted languages).
For production systems, the better pattern is environment variables that hold paths or identifiers to secrets, not the secrets themselves. `DATABASE_CREDENTIALS_FILE=/run/secrets/db` combined with a secrets manager like HashiCorp Vault, AWS Secrets Manager, or systemd-creds (available since systemd 250) is more secure. Systemd-creds encrypts secrets at rest and injects them at service startup, removing them from disk after the process reads them.
Never commit `.env` files to version control. Use `.gitignore` patterns and enforce them with pre-commit hooks. A leaked AWS key in a public repo gets found by automated scanners within minutes - we have seen this happen on test servers spun up for articles on this site.
# systemd-creds: encrypt a secret at rest (systemd 250+)
systemd-creds encrypt --name=db-password - -
# paste your secret, Ctrl+D
# Reference in unit file
[Service]
LoadCredential=db-password:/etc/credstore/db-password.cred
# Available to the service at $CREDENTIALS_DIRECTORY/db-password
# Check if your env is leaking in ps
ps eww -p $(pgrep myapp)
# Safer: read secret from file, not env
export DB_PASS=$(cat $CREDENTIALS_DIRECTORY/db-password)PATH Management at Scale
PATH is the most frequently modified environment variable and the one most often corrupted by careless scripts. The standard pattern `export PATH=$PATH:/new/dir` appends to PATH, but order matters - directories earlier in PATH shadow later ones.
When managing multiple tool versions (Go, Python, Node, Ruby), use version managers that prepend to PATH: `export PATH=$HOME/.local/go/bin:$PATH`. The version manager handles the logic of which binary to expose. Tools like asdf-vm manage this with shims and a `.tool-versions` file per project.
On systems with many users or automated deployment pipelines, PATH inconsistency between the interactive shell and the cron or systemd context causes real failures. A binary available interactively may not be in PATH when the job runs as root via cron. The fix is to either use absolute paths in cron jobs and scripts, or to set PATH explicitly at the top of every script with `export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/yourtool/bin`.
To audit PATH for duplicates and non-existent directories, which slow down command lookup:
For team environments where multiple engineers work on the same infrastructure code, standardizing PATH in a shared `.envrc` (committed to the repo, loaded by direnv) is cleaner than managing it in each person's shell profile. If you are also managing project naming conventions and domain assets alongside your infrastructure repos, tools like nicename.me can help check name availability and set consistent naming across project directories, which matters when those names feed into PATH entries or service hostnames.
# Audit PATH: show each directory on its own line
echo $PATH | tr ':' '\n'
# Remove duplicates from PATH
export PATH=$(echo $PATH | tr ':' '\n' | awk '!seen[$0]++' | tr '\n' ':' | sed 's/:$//')
# Check for non-existent directories in PATH
echo $PATH | tr ':' '\n' | while read dir; do
[ -d "$dir" ] || echo "MISSING: $dir"
done
# Set absolute PATH in scripts for reliability
#!/usr/bin/env bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binEnvironment Variables in Containers and CI/CD
In Docker, environment variables are injected at three points: the Dockerfile `ENV` instruction (baked into the image), `docker run -e KEY=VALUE` (runtime), and `--env-file ./myfile` (runtime from a file). Variables set with `ENV` in a Dockerfile appear in `docker inspect` output and image metadata, which means they are effectively public if the image is pushed to a registry. Never use `ENV` for secrets.
Docker Compose uses the `environment:` key in the service definition or an `env_file:` directive pointing to a flat file. Variables in the host shell are interpolated into the compose file when you reference them as `${VARIABLE}`, which is useful for injecting CI-provided secrets without storing them in the compose file itself.
In Kubernetes, environment variables come from `env:` (literal values), `envFrom:` (entire ConfigMap or Secret), or the Downward API (pod metadata). For secrets, use `envFrom` referencing a Secret object, not a ConfigMap. Since Kubernetes 1.25, the `immutable: true` field on Secrets prevents accidental modification and reduces API server load in large clusters.
In GitHub Actions, environment variables set with `echo "KEY=VALUE" >> $GITHUB_ENV` persist across subsequent steps in the same job. Variables scoped to a single step are set with `env:` in the step definition. Secrets are injected as environment variables via the `secrets` context and are masked in logs automatically - but only exact matches are masked, so splitting a secret across multiple concatenations can expose it.
For complex multi-step pipeline orchestration where you need deterministic variable passing across jobs and services, purpose-built automation platforms handle the state management that raw CI environment variables cannot. Evaluating platforms like taskbotshub.ai is worth the time if your pipelines regularly fail because of environment variable state not propagating correctly across stages.
# Docker: runtime env vars (preferred for secrets)
docker run -e DB_HOST=10.0.0.5 -e DB_PORT=5432 myapp:latest
# Docker: env file
docker run --env-file ./prod.env myapp:latest
# Kubernetes: Secret as env vars
apiVersion: v1
kind: Pod
spec:
containers:
- name: myapp
image: myapp:latest
envFrom:
- secretRef:
name: myapp-secrets
# GitHub Actions: persist variable across steps
- name: Set version
run: echo "APP_VERSION=$(git describe --tags)" >> $GITHUB_ENV
- name: Use version
run: echo "Building $APP_VERSION"Debugging Environment Variable Problems
When a script or service behaves differently in production than in development, environment variables are the first thing to check. The diagnostic approach should be systematic.
First, capture the full environment of a working instance: `printenv | sort > working.env`. Then capture the failing instance's environment the same way and run `diff working.env failing.env`. This surfaces missing or differing variables immediately.
For systemd services, `systemctl show myapp.service | grep Env` shows what variables are set in the unit. To see the complete effective environment of a running service, use `cat /proc/$(systemctl show -p MainPID --value myapp.service)/environ | tr '\0' '\n' | sort`.
For debugging variable expansion in scripts, run the script with `bash -x` to trace every command and see variables expanded to their values: `bash -x myscript.sh`. For a specific section, wrap it with `set -x` and `set +x`.
Stale environment variables in long-running processes are another class of problem. A daemon reads its environment at startup and does not pick up subsequent changes to `/etc/environment` or its unit file without a restart. Always `systemctl daemon-reload && systemctl restart myservice` after modifying environment configuration, never just `systemctl reload`.
# Full environment diff between two contexts
printenv | sort > /tmp/current.env
ssh prod-server 'printenv | sort' > /tmp/prod.env
diff /tmp/current.env /tmp/prod.env
# Inspect systemd service environment
systemctl show myapp.service | grep -i env
# Full env of running service by PID
SVC_PID=$(systemctl show -p MainPID --value myapp.service)
cat /proc/$SVC_PID/environ | tr '\0' '\n' | sort
# Trace script execution with variable expansion
bash -x /opt/deploy/release.sh 2>&1 | head -100
# Reload systemd and restart after env changes
systemctl daemon-reload && systemctl restart myapp.service