Where Service Files Live and Why It Matters
systemd resolves unit files from multiple directories in a strict priority order. Files in /etc/systemd/system/ override those in /usr/lib/systemd/system/, which is where package managers write their units. Drop-in fragments live under /etc/systemd/system/myapp.service.d/ and merge with the base unit without replacing it. Understanding this hierarchy prevents the classic mistake of editing a file under /usr/lib/ and having a package update silently overwrite it.
For any service you own, write to /etc/systemd/system/. For overrides to a vendor unit, use drop-ins. Never edit files under /usr/lib/systemd/system/ directly. Run `systemctl cat nginx.service` to see the effective merged unit including all drop-ins currently applied to a unit - this is the ground truth, not the raw file.
# Show resolved unit file + all applied drop-ins
systemctl cat nginx.service
# Show which file is providing the unit
systemctl show nginx.service -p FragmentPath
# List all drop-in directories systemd will check
systemd-analyze unit-pathsThe Three Sections: [Unit], [Service], [Install]
Every .service file is an INI-style text file with three sections. [Unit] is metadata and dependency ordering. [Service] is the process configuration. [Install] tells systemctl enable where to hook the unit into the boot sequence. Only [Service] is strictly required for a functional unit, but omitting [Install] means you cannot enable the service to start at boot.
A minimal working service file looks like the example below. It runs a Go binary as a non-root user, restarts on failure, and logs to the journal. We use this as the base and build up from here.
[Unit]
Description=MyApp HTTP API server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.targetChoosing the Right Type= Directive
Type= controls how systemd determines that your service has successfully started. Picking the wrong type causes dependency chains to break or services to appear ready before they are.
Type=simple is the default when ExecStart is set and no Type is given. systemd considers the service started the moment the main process forks. Use this for any daemon that does not daemonize itself and stays in the foreground - most modern Go, Rust, and Node.js services work this way.
Type=exec is stricter than simple. systemd waits until the exec() call inside the main process completes before marking the service active. This avoids a race condition where a wrapper script forks the real process after systemd already considers the unit started. On systemd 240+, prefer exec over simple for anything that uses a launcher script.
Type=forking is for traditional daemons that double-fork and write a PID file. You must also set PIDFile= to the full path of the PID file so systemd can track the actual daemon process. Without PIDFile=, systemd tracks the launcher and the unit appears to die when the launcher exits.
Type=notify requires the service to call sd_notify(3) with READY=1 when it is fully initialized. This is the correct type for services that need to complete initialization - database connection pools, TLS certificate loading - before accepting traffic. nginx supports this natively. For your own services, link against libsystemd or use the sd_notify socket protocol directly.
Type=oneshot is for tasks that run to completion and exit. Scripts, batch jobs, and setup tasks use this. Pair it with RemainAfterExit=yes if you want systemctl to report the service as active after the process exits.
# Check if a service supports sd_notify
strings /usr/sbin/nginx | grep -i READY
# Test sd_notify manually
systemd-notify --ready --status="Initialization complete"
# For a oneshot that stays "active" after exit
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/setup-network-routes.shDependency Ordering: After=, Requires=, Wants=, BindsTo=
Ordering and dependency directives are separate concerns in systemd, and conflating them is the source of most unit file bugs we see in code review.
After= and Before= control ordering only. They do not create a dependency. If you list After=postgresql.service, systemd will start your service after PostgreSQL if both are being started in the same transaction. But if PostgreSQL is not enabled or not being started, systemd will not pull it in.
Wants= creates a soft dependency. If the wanted unit fails to start, your service starts anyway. Use this for optional integrations.
Requires= creates a hard dependency. If the required unit fails or stops, your service stops too. Use this when your service genuinely cannot run without another unit. Be careful: Requires= without After= means both units start simultaneously, which is rarely what you want. Almost always write both together.
BindsTo= is stronger than Requires=. If the bound unit stops for any reason, your service is immediately stopped. Use this for units that are logically part of another unit - a sidecar proxy that must die when the main container exits, for example.
For database-backed web services, the correct pattern is After=postgresql.service Requires=postgresql.service. For a service that should start after the network is ready, use After=network-online.target Wants=network-online.target. The network-online.target waits for at least one interface to have an address, unlike network.target which fires immediately after interfaces are configured.
# Verify your dependency graph before production
systemd-analyze verify /etc/systemd/system/myapp.service
# See the full dependency tree for a unit
systemctl list-dependencies myapp.service
# Show reverse dependencies (who depends on this unit)
systemctl list-dependencies --reverse myapp.serviceEnvironment Variables and EnvironmentFile=
Never hardcode secrets in a service file. The file is world-readable by default and ends up in version control. Use EnvironmentFile= pointing to a file owned by root with mode 0600, or use systemd credentials introduced in systemd 250.
EnvironmentFile= takes a path. If the path is prefixed with a hyphen, systemd will silently ignore a missing file instead of failing. Variables defined in the file are available to ExecStart and all other Exec* directives as standard environment variables. Format is KEY=value, one per line, with shell-style quoting supported.
For structured secrets management in 2026, the LoadCredential= directive is the correct approach on systemd 250+. It loads a file into a credentials directory accessible via the CREDENTIALS_DIRECTORY environment variable. The credential file is never exposed in the unit file itself and can be sourced from a TPM or systemd-creds encrypted store.
For teams using AI-assisted DevOps tooling to generate and validate unit files at scale, platforms like taskbotshub.ai can automate unit file generation with secrets hygiene checks baked into the pipeline, reducing the chance of credentials leaking into service file templates.
# EnvironmentFile approach (systemd 219+)
[Service]
EnvironmentFile=-/etc/myapp/myapp.env
ExecStart=/opt/myapp/bin/myapp
# /etc/myapp/myapp.env (chmod 600, chown root:root)
DB_PASSWORD=hunter2
API_KEY=sk-prod-abc123
# LoadCredential approach (systemd 250+)
[Service]
LoadCredential=db-password:/etc/credentials/myapp-db-password
ExecStart=/opt/myapp/bin/myapp
# Access in the process:
# cat $CREDENTIALS_DIRECTORY/db-passwordRestart Policies and Failure Handling
Restart= controls when systemd restarts the service. The options and their behaviors are precise and often misunderstood.
Restart=no means never restart. Restart=always restarts regardless of exit code or signal. Restart=on-failure restarts when the process exits with a non-zero code, is killed by a signal not listed in SuccessExitStatus=, or times out. Restart=on-abnormal restarts on signal or timeout but not on clean non-zero exits. For most daemons, Restart=on-failure with RestartSec=5s is the right choice.
StartLimitIntervalSec= and StartLimitBurst= control the restart rate limiter. By default on systemd 240+, a service that fails 5 times in 10 seconds is put into a failed state and will not be restarted again until you run systemctl reset-failed myapp. To disable the rate limiter entirely, set StartLimitIntervalSec=0. We do not recommend disabling it in production - a service in a tight crash loop will generate enormous journal output and can destabilize the host. Instead, tune the burst window to something reasonable for your MTTR.
RestartSec= accepts time units: 5s, 1min, 500ms. Setting it too low on a service that fails at startup floods the journal. For services with expensive initialization (JVM startups, large model loads), use RestartSec=15s or higher.
TimeoutStartSec= controls how long systemd waits for the service to report ready (for Type=notify) or for ExecStart to return (for Type=forking). The default is 90 seconds on most distributions. For services with long initialization, increase this. For fast-starting microservices, decrease it to catch hangs early.
[Service]
Restart=on-failure
RestartSec=10s
StartLimitIntervalSec=60s
StartLimitBurst=3
TimeoutStartSec=30s
TimeoutStopSec=30s
# Reset a failed/rate-limited service
systemctl reset-failed myapp.service
systemctl start myapp.service
# See current restart count and state
systemctl show myapp.service -p NRestarts -p ActiveState -p SubStateProcess Hardening with Security Directives
systemd's sandboxing directives are the most direct way to apply the principle of least privilege without writing SELinux policy or seccomp profiles from scratch. These directives are available without any kernel module and work on any system with a reasonably modern kernel (4.14+).
NoNewPrivileges=yes prevents the process and all its children from gaining additional privileges via setuid binaries or file capabilities. This is a no-cost security win for any service not deliberately using privilege escalation. Set it on every service.
ProtectSystem=strict mounts /usr, /boot, and /etc read-only inside the service's mount namespace. Use ReadWritePaths= to add back specific directories the service needs to write to. ProtectHome=yes makes /home, /root, and /run/user invisible to the process.
PrivateTmp=yes gives the service its own /tmp and /var/tmp. Files written there are invisible to other services and cleaned up when the service stops. Always enable this for services that use temporary files.
PrivateNetwork=yes gives the service a loopback-only network namespace. Use this for services that only communicate via Unix sockets or that should have no network access at all.
SystemCallFilter= restricts which syscalls the service can make. The @system-service set covers the syscalls needed by most server daemons. Pair this with SystemCallErrorNumber=EPERM so disallowed syscalls return EPERM instead of causing SIGSYS, which is easier to debug.
Running systemd-analyze security myapp.service gives each service an exposure score from 0.0 (fully sandboxed) to 10.0 (no restrictions). On our test server running a default nginx install, the score was 9.2 before adding hardening directives and dropped to 2.1 after applying the full set below.
[Service]
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
CapabilityBoundingSet=
AmbientCapabilities=
# Allow writes only to the data directory
ReadWritePaths=/var/lib/myapp /run/myapp
# Check exposure score
systemd-analyze security myapp.serviceUsing Drop-ins to Override Without Forking
When you need to customize a vendor-provided unit - adding an environment variable to a PostgreSQL unit, increasing the open file limit for nginx - use drop-in files rather than modifying the unit in /usr/lib/. The correct workflow is `systemctl edit nginx.service`, which opens an editor on a new file at /etc/systemd/system/nginx.service.d/override.conf and reloads the daemon afterward.
To override an ExecStart directive, you must first clear it with an empty ExecStart= assignment, then set the new value. If you skip the empty assignment, systemd appends the new ExecStart to the list and runs both, which is almost never what you want.
Drop-ins are also the correct mechanism for adding service-level resource limits without touching the base unit. LimitNOFILE= for open file descriptors, LimitNPROC= for processes, and the unified cgroup v2 directives MemoryMax=, CPUQuota=, and TasksMax= can all go in a drop-in.
# Edit with systemctl (preferred - handles reload automatically)
systemctl edit nginx.service
# Manual drop-in path
mkdir -p /etc/systemd/system/nginx.service.d/
cat > /etc/systemd/system/nginx.service.d/limits.conf << 'EOF'
[Service]
LimitNOFILE=65536
MemoryMax=2G
CPUQuota=150%
EOF
systemctl daemon-reload
systemctl restart nginx.service
# Verify the drop-in is applied
systemctl cat nginx.service | grep -A2 'drop-in'
systemctl show nginx.service -p LimitNOFILETemplated Units for Multiple Instances
If you run the same service binary for multiple tenants, environments, or shards, templated units eliminate duplication. A template unit file has an @ in the name: myapp@.service. The instance identifier is passed on the command line and becomes available inside the unit as %i (the literal value) and %I (the value with escaped characters decoded).
Templates are how you run multiple OpenVPN tunnels, multiple Gunicorn workers for different apps, or multiple PostgreSQL clusters from a single unit definition. The instance name is whatever you pass to systemctl - there is no registration step.
When naming your services and instances, keep identifiers lowercase, hyphen-separated, and DNS-safe. This matters both for systemd's escaping rules and for operational clarity in the journal. If you are also picking a domain or project name for the service you are packaging, services like nicename.me make it straightforward to find clean, memorable names that match the identifier you have already chosen for the unit file.
For the example below, `systemctl start myapp@prod.service` and `systemctl start myapp@staging.service` start two independent instances, each reading from /etc/myapp/prod.yaml and /etc/myapp/staging.yaml respectively.
# /etc/systemd/system/myapp@.service
[Unit]
Description=MyApp instance %i
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/%i.yaml --instance %i
EnvironmentFile=-/etc/myapp/%i.env
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
# Usage
systemctl enable --now myapp@prod.service
systemctl enable --now myapp@staging.service
systemctl status 'myapp@*.service'Verifying and Debugging Your Unit File
Before enabling a new unit in production, run it through systemd-analyze verify. This catches syntax errors, missing required directives, and some common logic mistakes without starting any processes.
For runtime debugging, journalctl is the primary tool. The service's stdout and stderr both go to the journal when StandardOutput and StandardError are unset (the defaults route to the journal). Use -u to filter by unit, -f to follow, and --since to bound the time range. Adding -x shows explanatory text for systemd event messages, which helps decode obscure failure codes.
If a service fails to start, check the journal first, then check systemctl status for the exit code. Exit code 203 (EXEC) means the binary was not found or not executable. Exit code 217 (USER) means the specified User= does not exist. Exit code 226 (NAMESPACE) means a namespace directive failed - usually PrivateNetwork=yes on a system without network namespace support.
For performance analysis, systemd-analyze critical-chain myapp.service shows the dependency chain that contributed to the service's start time, pinpointing what is delaying boot. On our test server, a Django application was taking 42 seconds to reach active state because it listed After=time-sync.target without the corresponding Wants=, making it wait for a target that was never activated.
# Pre-flight check
systemd-analyze verify /etc/systemd/system/myapp.service
# Reload and test
systemctl daemon-reload
systemctl start myapp.service
systemctl status myapp.service
# Follow logs in real time
journalctl -u myapp.service -f
# Show logs from the last start attempt
journalctl -u myapp.service -e --since "10 minutes ago"
# Decode the exit code from a failed start
systemctl show myapp.service -p ExecMainStatus -p Result
# Boot timing analysis
systemd-analyze critical-chain myapp.service
systemd-analyze blame | head -20