Where Linux Logs Actually Live

The answer changed significantly with systemd's adoption. On any modern distro running systemd (RHEL 7+, Debian 8+, Ubuntu 15.04+), logs split into two parallel streams: the binary journal managed by systemd-journald, and traditional flat text files under /var/log/ written by rsyslog or syslog-ng forwarding from the journal.

The journal lives at /var/run/journal/ (volatile, lost on reboot) or /var/log/journal/ (persistent). Check which mode you're in:

If /var/log/journal/ exists and is a directory owned by systemd-journal, you have persistent storage. If it doesn't exist, journald is writing to the volatile path and you lose logs on reboot. Fix that immediately on any server you care about:

The flat-file hierarchy under /var/log/ follows a loose standard. /var/log/syslog or /var/log/messages catches general kernel and daemon output. /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS/Fedora) captures authentication events including SSH logins, sudo usage, and PAM activity. /var/log/kern.log isolates kernel messages. Application-specific logs go to subdirectories: /var/log/nginx/, /var/log/apache2/, /var/log/postgresql/, and so on.

Knowing this layout matters when you're responding to a security event at 2am without your usual tooling. `find /var/log -name '*.log' -newer /var/log/syslog -ls` will surface any log file modified more recently than syslog itself, which is useful for spotting applications writing to unexpected locations.

# Check journal storage mode
ls -la /var/log/journal/

# Enable persistent journal storage
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

# Verify journal disk usage
journalctl --disk-usage

Reading the Journal with journalctl

journalctl is the primary interface to systemd-journald and has filters most sysadmins underuse. The default output with no arguments dumps everything from the current boot, oldest first. That's rarely useful. Most real work involves time bounds, unit filters, and priority filters combined.

Priority levels follow syslog convention: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug. `-p err` shows err and above (0-3). To see only the last boot's errors from a specific unit:

The `-b` flag accepts negative integers: `-b -1` is the previous boot, `-b -2` is two boots ago. `journalctl --list-boots` shows all available boots with their timestamps and IDs. On a server that rebooted unexpectedly, start with `journalctl -b -1 -p err` to see what was erroring before the crash.

Time-bounded queries use `--since` and `--until` with flexible formats. `--since '2026-08-17 14:00:00'` and `--since '1 hour ago'` both work. For incident response, `--since` and `--until` bracketing a known event window is faster than scrolling.

Cursor-based following is more reliable than `tail -f` for journal output because it handles log rotation and system restarts: `journalctl -f -u nginx.service` follows nginx logs live. Add `--output=json` to pipe structured output to jq or a log aggregator.

JSON output from journalctl exposes fields that the default formatter hides. `_SYSTEMD_UNIT`, `_PID`, `_COMM`, `_EXE`, `_HOSTNAME`, and `_BOOT_ID` are all queryable. You can filter on any journal field directly: `journalctl _COMM=sshd _PID=1234` ands those filters together. Prefix a field with `+` to or them.

# Last boot errors from sshd
journalctl -b -0 -p err -u sshd.service

# All authentication failures in the last 4 hours
journalctl --since '4 hours ago' -u sshd.service | grep 'Failed password'

# JSON output for parsing
journalctl -u nginx.service --output=json --since today | jq '.MESSAGE'

# Filter by executable path
journalctl _EXE=/usr/sbin/sshd --since '2026-08-17 00:00:00'

# Count kernel OOM events this boot
journalctl -b -k | grep -c 'Out of memory'

# Show logs between two timestamps
journalctl --since '2026-08-17 13:45:00' --until '2026-08-17 14:15:00'

Parsing Flat Log Files: grep, awk, and sed Patterns That Actually Work

Flat log files remain the reality for most application logs and for infrastructure running rsyslog without full journal forwarding. The standard syslog format puts timestamp, hostname, process name, PID, and message on each line. nginx and Apache use their own formats. PostgreSQL uses CSV or the default single-line format.

For sshd authentication failures, the pattern is consistent across distros:

The IP extraction here uses grep's `-oP` (Perl regex, output only matching) with a lookahead. On systems without GNU grep, use `awk '{print $NF}' | tr -d '()'` as a fallback. Sorting by frequency (`sort | uniq -c | sort -rn`) gives you your top brute-force sources in seconds.

For nginx access logs in the default combined format, extracting 5xx errors with their request details:

awk is significantly faster than grep+cut pipelines on large files because it processes the line once. On a 2GB nginx access log we tested on a 4-core VM, the awk one-liner finished in 11 seconds versus 34 seconds for an equivalent grep+cut chain.

For structured application logs in JSON format (common with Node.js, Go services, and anything using logrus or zap), jq is the right tool. `jq -r 'select(.level == "error") | [.timestamp, .msg, .error] | @tsv'` converts filtered JSON log entries to tab-separated output you can pipe further.

When log files are rotated and compressed (gzip by default with logrotate), use `zcat` or `zgrep` instead of `cat` or `grep`. `zgrep 'Failed password' /var/log/auth.log.*.gz` searches all archived auth logs without decompressing them to disk. `zcat /var/log/syslog.2.gz | grep -c 'ERROR'` counts errors in a specific archived file.

# Top IPs with failed SSH logins
grep 'Failed password' /var/log/auth.log \
  | grep -oP 'from \K[0-9.]+' \
  | sort | uniq -c | sort -rn | head -20

# nginx 5xx errors with timestamps and paths
awk '$9 >= 500 {print $4, $7, $9}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -30

# PostgreSQL slow queries over 1000ms from today's log
grep 'duration:' /var/log/postgresql/postgresql-2026-08-17_0000.log \
  | awk -F'duration: ' '{if ($2+0 > 1000) print}' \
  | tail -50

# Search all rotated auth logs including gzip archives
zgrep 'sudo' /var/log/auth.log /var/log/auth.log.*.gz 2>/dev/null

# Parse JSON app logs, extract errors as TSV
jq -r 'select(.level == "error") | [.time, .msg, .err] | @tsv' \
  /var/log/myapp/app.log
// advertisement

Kernel Ring Buffer and dmesg

The kernel ring buffer is separate from syslog and journald. `dmesg` reads it directly from /dev/kmsg. On kernels 3.5+ (anything you're running in 2026), `dmesg` supports human-readable timestamps with `-T`, level filtering with `-l`, and facility filtering with `-f`.

For hardware issues, OOM events, and filesystem errors, the kernel ring buffer is the first place to check. Filesystem errors look like `EXT4-fs error` or `XFS: Internal error`. NIC errors produce `eth0: NETDEV WATCHDOG` or `tg3: eth0: transmit timed out`. Storage issues produce `sd 2:0:0:0: [sdb] timing out command`.

The ring buffer has a fixed size (typically 512KB but configurable via `CONFIG_LOG_BUF_SHIFT` at compile time). On busy systems it wraps and you lose old entries. For anything production, make sure your rsyslog or journald is capturing kernel messages to persistent storage. Check with `journalctl -k --since boot | wc -l` - if the count is suspiciously round, you may have lost entries.

For OOM (out of memory) killer events specifically, the kernel logs extensive detail: which process was killed, the OOM score, and a snapshot of memory usage per process. `dmesg -T | grep -A 20 'Out of memory'` captures the full context including the kill decision.

# Human-readable dmesg with timestamps, errors only
dmesg -T -l err,crit,alert,emerg

# Watch kernel messages live
dmesg -w

# OOM events with context
dmesg -T | grep -A 20 'Out of memory'

# Storage errors
dmesg -T | grep -iE '(error|fail|timeout)' | grep -iE '(sd[a-z]|nvme|dm-|md[0-9])'

# Network errors
dmesg -T | grep -iE '(eth|ens|eno|bond|tun|br)[0-9]' | grep -iE '(error|timeout|reset)'

Configuring rsyslog for Centralized Collection

rsyslog 8.x (the version shipping on RHEL 9 and Ubuntu 22.04+) uses a module-based configuration with RainerScript syntax. The old `facility.priority action` format still works but is deprecated for complex configs. A production rsyslog setup should forward to a central log server while keeping local copies.

The key rsyslog performance settings are often left at defaults and cause problems at scale. `$WorkDirectory` should point to fast local storage. `queue.size` controls how many messages buffer in memory before writing to disk or dropping. `queue.highwatermark` triggers the spooling to disk queue. For a server generating 5000 messages/second, the defaults will cause loss.

On the receiving side (your central log server), rsyslog needs to listen on TCP 514 (or 6514 for TLS) and write to per-host log files. The `%HOSTNAME%` template variable creates the directory structure. With this config, logs from each host land in /var/log/remote//syslog.log and rotate independently.

For DevOps teams using automated log pipeline deployment, tools like those at taskbotshub.ai can template rsyslog configs across fleets and validate the resulting pipeline end-to-end, catching dropped-message scenarios before they hit production.

TLS encryption for log forwarding requires the rsyslog-gnutls module and matching certificates. Without TLS, log data including authentication events travels in plaintext. On any network segment you don't fully control, plaintext syslog is a compliance and security problem.

# /etc/rsyslog.conf snippet - high-throughput forwarding with disk queue
module(load="imuxsock")
module(load="imklog")
module(load="omfwd")

# Disk-assisted queue for forward action
*.* action(
  type="omfwd"
  target="logs.internal.example.com"
  port="514"
  protocol="tcp"
  queue.type="LinkedList"
  queue.filename="fwdqueue"
  queue.size="50000"
  queue.highWatermark="40000"
  queue.lowWatermark="2000"
  queue.maxDiskSpace="1g"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1"
)

logrotate: Preventing Disk Exhaustion

logrotate runs daily via cron or systemd timer (/etc/cron.daily/logrotate or logrotate.timer). It reads /etc/logrotate.conf and all files under /etc/logrotate.d/. The most common production failure mode is a log growing unbounded because the application's logrotate config uses `notifempty` and the log never becomes empty, or because the postrotate script fails to signal the daemon.

To test a logrotate config without actually rotating: `logrotate --debug /etc/logrotate.d/nginx`. This shows what would happen without side effects. The debug output lists each file, whether it qualifies for rotation, and what actions would run.

The `copytruncate` directive handles applications that keep a file descriptor open and won't reopen on SIGHUP. Instead of moving the file and creating a new one, it copies the content then truncates the original. The downside is a brief window where log entries between the copy and truncate are lost. For nginx, the correct approach is `postrotate` with `nginx -s reopen`, not `copytruncate`.

For high-volume logs, `compress` and `delaycompress` together mean: compress the n-1 rotation (not the immediately rotated file). This avoids compressing a file that the application may still briefly write to. `compresscmd /usr/bin/zstd` with `compressext .zst` gives significantly better compression ratios than gzip for log data, typically 4-5x versus gzip's 2-3x, at comparable or faster speeds.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
  daily
  rotate 30
  size 100M
  compress
  delaycompress
  compresscmd /usr/bin/zstd
  compressoptions -19
  compressext .zst
  missingok
  notifempty
  sharedscripts
  postrotate
    systemctl kill --signal=USR1 myapp.service 2>/dev/null || true
  endscript
}

# Test config without rotating
logrotate --debug /etc/logrotate.d/myapp

# Force rotation immediately (ignores size/time constraints)
logrotate --force /etc/logrotate.d/myapp

# Check logrotate state file
cat /var/lib/logrotate/status | grep myapp
// advertisement

Security-Focused Log Analysis Patterns

From a security standpoint, the most valuable log sources are: auth.log/secure (who authenticated and from where), audit.log (syscall-level activity if auditd is running), and application access logs. Correlating across these three sources catches attacks that look clean in any single source.

For SSH brute force detection, raw failure counts are less useful than tracking the progression from failed attempts to successful login from the same IP. An attacker who fails 200 times then succeeds is more dangerous than a scanner that fails 10,000 times and never succeeds:

The auditd log (/var/log/audit/audit.log) requires `ausearch` and `aureport` for practical querying. `ausearch` is faster than grep for audit logs because it understands the binary-ish format. `aureport --auth --summary` gives a weekly authentication summary. `ausearch -k privileged` shows all executions of setuid binaries if you've set up the standard privileged command watch rules.

For file integrity monitoring via audit, watch rules on /etc/passwd, /etc/shadow, /etc/sudoers, and /root/ are standard:

When analyzing logs for lateral movement or privilege escalation, look at sudo events in auth.log paired with new process spawns in audit.log. An attacker who gets sudo access will often immediately run `id`, `whoami`, `cat /etc/shadow`, or `crontab -e`. These show up in audit execve records within seconds of the sudo event.

Log retention for security purposes should be a minimum of 90 days locally and 1 year in cold storage to satisfy most compliance frameworks (PCI-DSS requires 12 months, 3 months immediately available). If you're managing log infrastructure naming and organizing log buckets by project or service, clean naming conventions matter at scale - resources like nicename.me can help when you're setting up domain-based log endpoints or naming internal log collection services consistently.

# Find IPs with failed SSH attempts that later succeeded
awk '/Failed password/{fail[$NF]++} /Accepted password/{ok[$NF]++} \
  END{for(ip in ok) if(ip in fail) print fail[ip], "fails then success:", ip}' \
  /var/log/auth.log | sort -rn

# All sudo commands run today
grep 'sudo:' /var/log/auth.log | grep "$(date '+%b %e')" | grep 'COMMAND'

# Audit report: authentication summary
ausearch --start today --end now -m USER_AUTH | \
  awk -F'acct=' '{print $2}' | awk '{print $1}' | \
  sort | uniq -c | sort -rn

# Watch critical files for modification
auditctl -w /etc/passwd -p wa -k passwd_changes
auditctl -w /etc/sudoers -p wa -k sudoers_changes
auditctl -w /root/.ssh/authorized_keys -p wa -k root_ssh

# Query those watch events
ausearch -k passwd_changes --start today

Structured Logging and Log Aggregation in 2026

Flat text logs are increasingly the minority in modern infrastructure. Containerized workloads write to stdout/stderr captured by the container runtime (Docker's json-file driver, containerd's logging plugins), Kubernetes routes to node-level log agents, and microservices emit JSON. The log pipeline has more stages.

For Kubernetes, `kubectl logs` is the surface interface but it only accesses logs from running pods. For evicted or crashed pods: `kubectl logs --previous `. For aggregation across a deployment: `kubectl logs -l app=myapp --all-containers=true --since=1h`.

On the aggregation backend, the current practical stack for self-hosted infrastructure in 2026 is Loki + Grafana (for teams already running Prometheus) or OpenSearch (for teams needing full-text search and already familiar with Elasticsearch). Both accept logs from Vector or Fluent Bit as the collection agent. Fluent Bit has a smaller memory footprint (typically 10-15MB versus Vector's 30-50MB) and handles high-volume container log collection efficiently.

For parsing rsyslog output into structured fields before forwarding to Loki, the `mmnormalize` module with liblognorm rule files is faster than regex-based parsing. A liblognorm rule for sshd auth failures:

Vector.dev (version 0.39+ as of mid-2026) handles transformation, enrichment, and routing in a single binary. Its VRL (Vector Remap Language) is worth learning if you're building any serious log pipeline. `vector tap` lets you inspect data at any pipeline stage in real time, which is invaluable for debugging parsing rules before they hit production.

For teams building automated log analysis pipelines - anomaly detection, alert routing, or pattern extraction - the AI-assisted DevOps tools available through platforms like taskbotshub.ai can reduce the time spent writing and tuning log parsing rules significantly compared to manual regex development.

# Vector config: tail syslog, parse, forward to Loki
# /etc/vector/vector.toml
[sources.syslog_files]
type = "file"
include = ["/var/log/syslog", "/var/log/auth.log"]
read_from = "beginning"

[transforms.parse_syslog]
type = "remap"
inputs = ["syslog_files"]
source = '''
  . = parse_syslog!(.message)
'''

[sinks.loki]
type = "loki"
inputs = ["parse_syslog"]
endpoint = "http://loki:3100"
labels.job = "syslog"
labels.host = "{{ host }}"

# Test Vector config
vector validate /etc/vector/vector.toml

# Inspect live data at transform stage
vector tap parse_syslog