systemd Architecture: What You Actually Need to Know
systemd replaces both the traditional init process (PID 1) and a pile of separate daemons: crond, syslogd, inetd, and ntpd can all be replaced by native systemd components. The core objects are units. A unit is a configuration file describing a resource: a service, a socket, a mount point, a timer, or a target (the replacement for runlevels).
Unit files live in three locations, loaded in this priority order: /etc/systemd/system/ (admin-controlled, highest priority), /run/systemd/system/ (runtime, ephemeral), and /usr/lib/systemd/system/ (vendor-supplied, lowest priority). Drop a file in /etc/systemd/system/ and it overrides the vendor default. This is important: never edit files under /usr/lib/systemd/system/ directly, because package upgrades will overwrite your changes.
The dependency model uses Wants=, Requires=, After=, and Before= directives. Wants= is a soft dependency - if the dependency fails, the unit still starts. Requires= is hard - failure cascades. After= and Before= control ordering only, not dependency. Most custom services need After=network-online.target to avoid racing the network stack on boot.
# Show the full dependency tree for a unit
systemctl list-dependencies nginx.service
# Show reverse dependencies: what depends on this unit
systemctl list-dependencies --reverse nginx.service
# Show all unit file load paths systemd searches
systemd-analyze unit-pathsWriting a Production-Grade Service Unit File
The minimal unit file that actually works in production looks nothing like the tutorials that show three lines and call it done. Real services need sandboxing, restart policies, resource limits, and proper logging. Here is a unit file for a Go HTTP service that we run on our test server at myunix.org.
The [Unit] section sets metadata and dependencies. The [Service] section is where most of the work happens. Type=simple is correct when your binary stays in the foreground. Type=exec is the better choice on systemd 240+ because systemd waits until exec() succeeds before considering the service started, which prevents race conditions with socket activation. Type=forking is legacy behavior for daemons that background themselves - avoid it for new services.
Restart=on-failure with RestartSec=5 and a StartLimitIntervalSec=60 / StartLimitBurst=3 combination gives you three restart attempts in 60 seconds before systemd gives up and marks the service as failed. Without StartLimitBurst, a crashing service will restart forever and hammer your logs.
The sandboxing directives are the part most guides skip. PrivateTmp=true gives the service its own /tmp. NoNewPrivileges=true prevents privilege escalation via setuid binaries. ProtectSystem=strict makes /usr and /boot read-only. These cost nothing in performance and eliminate entire classes of compromise.
[Unit]
Description=MyApp HTTP Service
Documentation=https://internal.wiki/myapp
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
[Service]
Type=exec
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3
# Hardening
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
CapabilityBoundingSet=
SystemCallFilter=@system-service
LockPersonality=true
RestrictRealtime=true
# Resource limits
LimitNOFILE=65536
MemoryMax=2G
CPUQuota=80%
[Install]
WantedBy=multi-user.targetsystemctl: The Commands That Matter
Most engineers use start, stop, restart, and status. The rest of systemctl is where the real power is. Here is what we use daily.
systemctl daemon-reload is mandatory after editing any unit file. Without it, systemd keeps the old version in memory. This is the most common reason why 'I edited the service file but nothing changed.'
systemctl edit unit creates an override file at /etc/systemd/system/unit.d/override.conf without touching the vendor file. Use this instead of copying the whole file when you only need to change one or two directives. systemctl cat unit shows the final merged configuration including all overrides, which is what systemd actually uses.
systemctl show outputs all properties of a unit as key=value pairs, machine-readable. Pipe it through grep to extract specific values in scripts. systemctl is-active and systemctl is-enabled return exit codes 0 for true, usable directly in shell conditionals.
systemctl list-units --type=service --state=failed gives you all failed services in one shot. Add --no-pager to use it in scripts without it blocking on a pager.
# Reload unit definitions after editing
systemctl daemon-reload
# Edit only the override, not the vendor file
systemctl edit nginx.service
# Show effective merged config
systemctl cat nginx.service
# Show all properties (machine-readable)
systemctl show nginx.service --property=ActiveState,MainPID,MemoryCurrent
# Use in a shell conditional
if systemctl is-active --quiet postgresql.service; then
echo "postgres is running"
fi
# All failed services
systemctl list-units --type=service --state=failed --no-pager
# Mask a service so nothing can start it, not even dependencies
systemctl mask snapd.serviceSocket Activation: On-Demand Service Startup
Socket activation is one of the most useful features in systemd and almost nobody uses it. The idea: systemd holds the socket open and hands it to the service only when a connection arrives. The service does not need to run at all until needed, which reduces boot time and idle memory usage. SSH, DBus, and systemd itself use this pattern.
You need two unit files: a .socket unit and a matching .service unit. systemd connects them by name - nginx.socket activates nginx.service. The socket unit defines the listening address. The service unit gets the socket file descriptor passed in through file descriptor 3 (the standard systemd socket activation protocol, compatible with inetd).
In our testing on a 4-core VM, moving three low-traffic internal tools to socket activation cut the idle service count from 47 to 44 and saved about 180MB of RSS. Not dramatic, but it adds up on servers with dozens of micro-services.
For services that support socket activation natively (NGINX does with --with-compat on recent builds, PostgreSQL does not), you get zero-downtime restarts for free: systemd holds the socket, the old service process drains, the new one starts and picks up the socket descriptor.
# /etc/systemd/system/myapp.socket
[Unit]
Description=MyApp Socket
[Socket]
ListenStream=8080
Accept=no
[Install]
WantedBy=sockets.target
# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp Service (socket-activated)
[Service]
Type=simple
ExecStart=/opt/myapp/bin/server
StandardInput=socket
# Enable and start only the socket - service starts on demand
systemctl enable --now myapp.socketReplacing Cron with systemd Timers
systemd timers are strictly more capable than cron for system-level tasks. They log to journald (cron does not), they can be monotonic (run X seconds after boot, not at a wall-clock time), they support randomized delay to prevent thundering herd, and they can be inspected with systemctl list-timers to see the next scheduled run and the last run time.
A timer unit requires a matching service unit. The timer triggers the service. The service does the work. Keep them in the same directory with the same base name.
Calendar expressions in systemd use a different syntax from cron. Daily backup at 2:30 AM is OnCalendar=*-*-* 02:30:00. Every 15 minutes is OnCalendar=*:0/15. The first Monday of every month is OnCalendar=Mon *-*-1..7 00:00:00. Run systemd-analyze calendar 'your-expression' to validate and see the next five trigger times before deploying.
AccuracySec= controls how precisely systemd honors the schedule. The default is 1 minute, meaning a 2:30 timer might fire at 2:30:47. Set AccuracySec=1s if you need precision. RandomizedDelaySec= adds a random delay up to the specified duration, useful when you have 50 servers running the same timer and do not want all of them hammering a database simultaneously.
If you are running DevOps automation pipelines that need smarter scheduling than timers provide, tools like taskbotshub.ai handle event-driven job orchestration with dependencies between tasks, which systemd timers cannot express natively.
# /etc/systemd/system/db-backup.service
[Unit]
Description=Database Backup
[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup-db.sh
StandardOutput=journal
StandardError=journal
# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run DB backup daily at 02:30
[Timer]
OnCalendar=*-*-* 02:30:00
AccuracySec=1s
RandomizedDelaySec=120
Persistent=true
[Install]
WantedBy=timers.target
# Check the schedule before enabling
systemd-analyze calendar '*-*-* 02:30:00'
# Enable and check
systemctl enable --now db-backup.timer
systemctl list-timers db-backup.timerjournald: Querying Logs Like a Pro
journald stores structured binary logs that journalctl renders. The structured part matters: every log entry has metadata fields like _SYSTEMD_UNIT, _PID, _UID, PRIORITY, and SYSLOG_IDENTIFIER that you can filter on directly, without grep.
The most useful journalctl flags that most engineers do not use: -u filters by unit, --since and --until accept natural time expressions like '1 hour ago' or '2026-08-15 14:00:00'. -o json outputs the full structured record. -o json-pretty is readable. -o short-monotonic shows time since boot, useful for correlating with kernel messages.
Priority filtering with -p err shows only errors and above. -p warning..err gives a range. PRIORITY values follow syslog: 0=emerg, 3=err, 4=warning, 6=info, 7=debug.
For persistent logs across reboots, set Storage=persistent in /etc/systemd/journald.conf and create /var/log/journal/. By default on many distros, logs are in /run/log/journal/ which is tmpfs and disappears on reboot. Run journalctl --list-boots to see available boot logs once you have persistence configured.
journalctl --disk-usage tells you how much space the journal is consuming. Use --vacuum-size=500M or --vacuum-time=30d to trim it. Set SystemMaxUse=2G in journald.conf to cap it permanently.
# Logs for nginx since yesterday
journalctl -u nginx.service --since yesterday --no-pager
# Only errors from the last hour
journalctl -u myapp.service -p err --since '1 hour ago'
# Structured JSON output for log shipping
journalctl -u myapp.service -o json --since '5 minutes ago' | \
jq -c '{time: .SYSLOG_TIMESTAMP, msg: .MESSAGE, pid: ._PID}'
# Follow live (like tail -f)
journalctl -u myapp.service -f
# Show logs from the previous boot
journalctl -u nginx.service -b -1
# All kernel messages from current boot
journalctl -k -b 0
# Trim journal to 500MB
journalctl --vacuum-size=500M
# Check journal disk usage
journalctl --disk-usageTargets: Managing System States
Targets replace SysV runlevels. multi-user.target is runlevel 3 (multi-user, no GUI). graphical.target is runlevel 5. rescue.target is single-user mode. emergency.target drops you to a minimal shell with only the root filesystem mounted.
To change the default target (what the system boots into), use systemctl set-default. To switch targets on a running system without rebooting, use systemctl isolate. Not all targets support isolation - only those with AllowIsolate=yes in their unit file. rescue.target and graphical.target do. network.target does not.
You can create custom targets to group services. A common pattern: create a maintenance.target that stops all application services and starts only monitoring, then switch to it with systemctl isolate maintenance.target during planned maintenance. This is cleaner than stopping services one by one and forgetting to restart them.
systemd-analyze gives you boot time broken down by service. systemd-analyze blame lists services sorted by startup time, slowest first. systemd-analyze plot > boot.svg generates an SVG timeline of the entire boot sequence. On our test server running Ubuntu 24.04 with a clean service set, total boot time from BIOS handoff to multi-user.target is 4.2 seconds, with NetworkManager taking 1.1 seconds as the largest contributor.
# Check current default target
systemctl get-default
# Set default to multi-user (no GUI) permanently
systemctl set-default multi-user.target
# Switch to rescue mode on running system
systemctl isolate rescue.target
# Analyze boot performance
systemd-analyze
systemd-analyze blame | head -20
systemd-analyze plot > /tmp/boot-$(hostname)-$(date +%Y%m%d).svg
# Check critical-chain: the serialized path that determined total boot time
systemd-analyze critical-chain multi-user.targetTransient Units and systemd-run
systemd-run creates a transient unit that exists only until the command finishes. This is the correct way to run one-off tasks with systemd resource control and logging, instead of running them bare in a shell where they inherit no limits and logs go nowhere permanent.
The --scope flag runs the command in a scope unit (a group of externally started processes, not a full service). The --unit flag sets a name. --property passes any service property. --uid and --gid set the user. --wait blocks until the command finishes and returns the exit code.
In our experience, systemd-run is the right tool when you need to run a database migration, a one-time data import, or a maintenance script under controlled resource limits without writing a permanent unit file. The journal captures all output with the unit name as the identifier, so you can retrieve it later with journalctl -u your-unit-name.
For teams building automated deployment pipelines, wrapping migration scripts in systemd-run gives you logging and resource control without the overhead of managing unit files per environment.
# Run a script with 4GB memory limit, log to journal
systemd-run --unit=db-migration --uid=postgres \
--property=MemoryMax=4G \
--property=CPUQuota=50% \
--wait \
/usr/local/bin/run-migrations.sh
# Check output after it finishes
journalctl -u db-migration
# Run an interactive shell in a controlled cgroup
systemd-run --scope --uid=1000 --property=MemoryMax=1G \
/bin/bash
# Run in the background, get the unit name
systemd-run --unit=import-job --uid=www-data \
/opt/app/import-data.py --source /data/new
systemctl status import-job.serviceTemplate Units for Multi-Instance Services
Template units let you run multiple instances of the same service with a single unit file. The file name contains @ - for example, worker@.service. Each instance is identified by a specifier passed after the @: worker@1.service, worker@2.service, and so on.
Inside the unit file, %i is the instance name. %p is the prefix (everything before the @). You can use these in ExecStart, Environment, WorkingDirectory, or anywhere else. This pattern is how OpenSSH sshd handles multiple connections, how container runtimes manage instances, and how we manage multiple PHP-FPM pool workers at different memory limits.
To start all instances in an array, use a loop or a target that Wants= all of them. Enabling worker@1.service and worker@2.service separately is the common pattern. If you are naming your instances after something meaningful - project names, tenant IDs, environment names - keeping those identifiers consistent across your stack matters. Tools like nicename.me can help when you are trying to generate clean, slug-safe identifiers for use in unit names and configuration files where spaces and special characters cause problems.
Instance-specific configuration can be passed through EnvironmentFile with %i in the path, so each instance reads /etc/myapp/worker-%i.conf.
# /etc/systemd/system/worker@.service
[Unit]
Description=Worker Instance %i
After=network-online.target redis.service
[Service]
Type=exec
User=worker
EnvironmentFile=/etc/myapp/worker-%i.conf
ExecStart=/opt/myapp/bin/worker --instance %i --config /etc/myapp/worker-%i.conf
Restart=on-failure
RestartSec=5
PrivateTmp=true
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
# Enable three instances
systemctl enable --now worker@1.service worker@2.service worker@3.service
# Check all instances
systemctl status 'worker@*.service'
# Logs from instance 2 only
journalctl -u worker@2.service -fDebugging Failing Services
A service that fails to start gives you a one-line error from systemctl status. The actual reason is almost always in journalctl. Run journalctl -u servicename.service -n 50 immediately after a failure to see the last 50 log lines including the error.
For services that fail before producing any output, systemd-analyze verify /etc/systemd/system/myunit.service checks the unit file syntax and reports common mistakes: missing After= for known dependencies, typos in directive names (systemd silently ignores unknown directives in older versions, which is how subtle bugs hide), and permission problems.
If the unit file looks correct but the service still fails, run the ExecStart command manually as the target user with sudo -u serviceuser /path/to/binary --args. This isolates whether the problem is in systemd configuration or in the binary itself. Check that the User= account exists, that the WorkingDirectory= exists and is readable, and that any EnvironmentFile= paths are present.
For crashes that are intermittent, set StandardOutput=journal and StandardError=journal explicitly and add SyslogIdentifier=myapp so you can filter by that identifier across reboots. Combine with Persistent=true in journald.conf to retain logs across reboots.
CPU and memory limit issues show up as OOMKilled in journalctl (look for OOM in kernel messages: journalctl -k | grep -i oom) or as the service being SIGKILL'd. systemctl show myapp.service --property=MemoryCurrent shows live memory use. systemd-cgtop gives you a real-time cgroup resource view similar to top.
# Get the real error after a failure
journalctl -u myapp.service -n 50 --no-pager
# Validate unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service
# Run as the service user manually to isolate the problem
sudo -u myapp /opt/myapp/bin/server --config /etc/myapp/config.yaml
# Live resource usage by cgroup
systemd-cgtop
# Check if OOM killer hit your service
journalctl -k | grep -i 'out of memory'
journalctl -k | grep -i oom | tail -20
# Show current memory usage of a service
systemctl show myapp.service --property=MemoryCurrent,CPUUsageNSec
# Enable coredumps for a service
# Add to [Service] section:
# LimitCORE=infinity
# Then check with:
coredumpctl list
coredumpctl info