syslog Fundamentals: Facilities, Severities, and the RFC

The syslog protocol is defined in RFC 5424 (2009), which superseded RFC 3164. Every log message carries a priority value (PRI) encoded as an integer: PRI = (facility * 8) + severity. There are 24 facilities (0-23) and 8 severity levels (0-7). Knowing the math matters when you write filters.

Facilities you will encounter daily: kern (0), user (1), mail (2), daemon (3), auth (4), syslog (5), lpr (6), news (7), uucp (8), cron (9), authpriv (10), ftp (11), and local0 through local7 (16-23). The local0-local7 range is reserved for custom application use - if you run an application that generates its own logs, assign it a local facility to make filtering trivial.

Severity levels from most to least critical: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). In rsyslog filter rules, specifying a severity selects that level and everything more critical. To match only a single severity, prefix with an equals sign.

RFC 5424 added structured data fields (SD-ID key-value pairs) and a MSGID field. rsyslog can parse and emit RFC 5424 format natively via the omfwd module with the RSYSLOG_SyslogProtocol23Format template. RFC 3164 is still common on older network appliances and is the default format for most /var/log writes, so you need to handle both in a real environment.

# Decode a PRI value manually
# PRI 190 = facility 23 (local7), severity 6 (info)
python3 -c "pri=190; print(f'facility={pri//8}, severity={pri%8}'}"

# List all facilities rsyslog recognizes
rsyslogd -N1 2>&1 | grep -i facility

rsyslog Architecture: Inputs, Parsers, Filters, Actions, Queues

rsyslog processes messages through a pipeline: input modules (im*) receive messages, parser modules (pm*) decode them, ruleset filters evaluate each message, and output/action modules (om*) deliver them. Understanding this pipeline is prerequisite to any non-trivial configuration.

Input modules you need to know: imuxsock reads from /dev/log (local syslog socket), imklog reads kernel ring buffer messages (dmesg), imjournal reads from systemd journal (replaces imuxsock on systemd hosts), imtcp and imudp receive remote syslog over TCP and UDP respectively, imptcp is a high-performance TCP input using epoll, imrelp uses the RELP protocol for guaranteed delivery.

Output modules: omfwd sends via UDP or TCP, omrelp sends via RELP, omfile writes to files, ommysql/ompgsql write to databases, omelasticsearch writes to Elasticsearch, omhttp sends to HTTP endpoints.

Queues are where most performance problems and data loss incidents originate. Every action in rsyslog has its own queue. The main queue (between input and rulesets) and action queues can be tuned independently. Queue types: direct (no queue, synchronous), linkedlist (in-memory, low overhead for moderate load), fixedarray (faster but pre-allocates memory), disk (persists to disk), disk-assisted (in-memory with disk overflow).

For a centralized log server receiving 50,000 events per second, we tested fixedarray queues with 250,000 slots and saw message loss drop to zero compared to the default direct queue under burst load. The key parameters are $MainMsgQueueType, $MainMsgQueueSize, and $MainMsgQueueDequeueBatchSize.

# Check rsyslog version and compiled-in modules
rsyslogd -v

# Check which input modules are active
rsyslogd -N1 -f /etc/rsyslog.conf 2>&1 | grep -E 'module|input'

# Verify rsyslog is reading from journal vs socket
grep -rE 'imuxsock|imjournal' /etc/rsyslog.conf /etc/rsyslog.d/

rsyslog Configuration Syntax: RainerScript vs Legacy

rsyslog supports two configuration syntaxes. The legacy syntax uses dollar-prefixed global directives and tab-separated facility.severity action lines - you will see this in every default config file shipped by distributions. RainerScript is the modern syntax, used since rsyslog 6.x, and supports if-then-else logic, variables, functions, and complex object definitions. Use RainerScript for anything new.

Legacy syntax example that you should recognize but not write:

*.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure

This sends info-and-above from all facilities except mail, authpriv, and cron to /var/log/messages, and all authpriv messages to /var/log/secure. The semicolon separates selectors; .none means exclude that facility.

RainerScript equivalents are explicit and testable. The module() and input() directives replace $ModLoad and legacy input directives. The action() directive replaces bare file paths and @/@@forwarding syntax. Always use action() because it exposes queue parameters.

A complete minimal /etc/rsyslog.conf for a standalone host using RainerScript:

Global() directives set process limits and template defaults. module() loads input modules. input() binds them to ports or sockets. Rules then filter and route. The include() directive pulls in /etc/rsyslog.d/*.conf snippets, which is where you put application-specific configs.

Test your config before reloading: rsyslogd -N1 returns exit code 0 on success and prints errors to stderr. Always run this before systemctl reload rsyslog - a syntax error will cause rsyslog to exit and stop collecting logs entirely.

# Validate config without restarting
rsyslogd -N1 -f /etc/rsyslog.conf
echo "Exit: $?"

# Minimal RainerScript standalone config
cat > /etc/rsyslog.d/00-base.conf << 'EOF'
# Load modules
module(load="imuxsock")
module(load="imklog" permitnonkernelfacility="on")

# Set default template
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Auth logs
if $syslogfacility-text == "authpriv" then {
    action(type="omfile" file="/var/log/secure")
    stop
}

# Cron
if $syslogfacility-text == "cron" then {
    action(type="omfile" file="/var/log/cron")
    stop
}

# Everything else at info and above
if $syslogseverity <= 6 then {
    action(type="omfile" file="/var/log/messages")
}
EOF

rsyslogd -N1 -f /etc/rsyslog.conf && systemctl reload rsyslog
// advertisement

Centralized Log Server: Receiver and Sender Configuration

A centralized rsyslog receiver collects logs from multiple hosts over TCP or UDP. UDP is connectionless and loses messages under load - use TCP or RELP for production. imptcp outperforms imtcp on Linux because it uses epoll instead of select.

On the receiver (log server), open TCP port 514 and create per-host log directories using the $HOSTNAME property. The dynafile feature creates files on demand based on template-resolved paths.

On the sender hosts, configure a forwarding action in /etc/rsyslog.d/50-remote.conf. Point it at the log server with TCP using @@hostname:port (two @ signs = TCP, one @ = UDP). Add a disk-assisted queue so messages persist locally if the log server is unreachable.

Message volume test: on our test server (4-core VM, 8GB RAM), imptcp with a fixedarray main queue of 100,000 messages sustained 85,000 events per second before latency climbed above 1ms. With imtcp under the same conditions, throughput peaked at 31,000 events per second. The difference is the epoll vs select bottleneck.

For firewall configuration on the receiver: open TCP 514 (syslog), TCP 2514 (RELP), and if using TLS, TCP 6514 (syslog-tls). Lock these down to only the IP ranges of your senders - syslog receivers are a common scanning target because many deployments leave UDP 514 open to 0.0.0.0/0.

# --- RECEIVER /etc/rsyslog.d/10-server.conf ---
module(load="imptcp")
input(type="imptcp" port="514")

# Template: store per-host in /var/log/remote/HOSTNAME/
template(name="RemoteHostLog" type="string"
  string="/var/log/remote/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%.log")

# Write all incoming messages to dynamic file
ruleset(name="remote") {
    action(type="omfile"
           dynaFile="RemoteHostLog"
           dirCreateMode="0750"
           fileCreateMode="0640"
           queue.type="fixedArray"
           queue.size="50000")
}

input(type="imptcp" port="514" ruleset="remote")

# --- SENDER /etc/rsyslog.d/50-remote.conf ---
action(type="omfwd"
       target="192.168.1.10"
       port="514"
       protocol="tcp"
       queue.type="LinkedList"
       queue.filename="fwdq"
       queue.maxdiskspace="1g"
       queue.saveonshutdown="on"
       action.resumeRetryCount="-1")

# Validate and reload on both hosts
rsyslogd -N1 && systemctl reload rsyslog

TLS Encryption for syslog Transport

Sending syslog over plaintext TCP means log data - including authentication events, sudo commands, and SSH sessions - transits your network unencrypted. On any network you do not fully control, this is unacceptable. rsyslog supports TLS via the gtls driver (GnuTLS) and the ossl driver (OpenSSL, added in rsyslog 8.2102.0).

Prerequisites: rsyslog must be compiled with TLS support. Check with rsyslogd -v | grep -i tls. On RHEL 9, the rsyslog-gnutls package provides the gtls driver. On Debian 12, install rsyslog-gnutls.

You need a CA certificate, a server certificate, and client certificates if you want mutual TLS. For internal use, create a simple CA with openssl. The rsyslog documentation recommends 4096-bit RSA or EC P-384 keys for log transport.

Receiver TLS config loads the gtls netstream driver and sets permittedPeer to enforce client authentication. Sender TLS config mirrors this. The permittedPeer directive accepts exact hostnames, wildcard prefixes, or fingerprints - use fingerprints in high-security environments because they survive DNS manipulation.

After enabling TLS, verify the connection is encrypted with openssl s_client before trusting that logs are flowing:

openssl s_client -connect logserver:6514 -CAfile /etc/pki/rsyslog/ca.pem

A TLS handshake failure will leave rsyslog silently queuing messages to disk (if you configured a disk queue) or dropping them (if you did not). Always configure disk-assisted queues on senders before enabling TLS so you have a recovery path during certificate rotation.

# Install TLS support
# RHEL/Rocky 9:
dnf install rsyslog-gnutls
# Debian 12:
apt install rsyslog-gnutls

# Generate CA + server cert (internal use)
mkdir -p /etc/pki/rsyslog
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-384 \
  -keyout /etc/pki/rsyslog/ca-key.pem \
  -out /etc/pki/rsyslog/ca.pem \
  -days 3650 -nodes -subj "/CN=rsyslog-CA"

openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-384 \
  -keyout /etc/pki/rsyslog/server-key.pem \
  -out /etc/pki/rsyslog/server-req.pem \
  -nodes -subj "/CN=logserver.internal"

openssl x509 -req -in /etc/pki/rsyslog/server-req.pem \
  -CA /etc/pki/rsyslog/ca.pem \
  -CAkey /etc/pki/rsyslog/ca-key.pem \
  -CAcreateserial \
  -out /etc/pki/rsyslog/server-cert.pem -days 365

# --- RECEIVER TLS config snippet ---
# Add to /etc/rsyslog.d/10-server-tls.conf
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/pki/rsyslog/server-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/pki/rsyslog/server-key.pem"
)

module(load="imtcp" StreamDriver.Name="gtls"
                   StreamDriver.Mode="1"
                   StreamDriver.Authmode="anon")
input(type="imtcp" port="6514")

# Verify TLS handshake
openssl s_client -connect 127.0.0.1:6514 \
  -CAfile /etc/pki/rsyslog/ca.pem

Filtering, Routing, and Structured Log Output

RainerScript filters give you boolean logic, string functions, and property access. The three property classes are: message properties ($msg, $rawmsg, $hostname, $fromhost-ip, $syslogtag, $programname), timestamp properties ($timereported, $timegenerated), and internal properties ($syslogfacility, $syslogseverity).

Common filtering patterns: route all SSH authentication events to a dedicated file; drop debug messages from a chatty daemon; forward only auth and authpriv to a SIEM.

The stop action halts processing for a matched message - essential for preventing messages from hitting multiple actions unintentionally. Without stop, a message matching authpriv.* will continue through all subsequent rules.

For SIEM forwarding, use JSON output instead of the traditional syslog format. rsyslog's built-in template RSYSLOG_FileFormat is RFC 3339 timestamp plus message. For structured output, define a custom JSON template:

template(name="json-log" type="list") { ... }

Elasticsearch users should use the omelasticsearch module directly rather than piping JSON through a file. omelasticsearch supports bulk indexing, index name templating by date, and retry on HTTP 429.

For teams running automated log analysis or alert pipelines, connecting rsyslog output to an AI-assisted operations platform can significantly reduce mean time to detect. taskbotshub.ai offers log-driven automation workflows that consume syslog streams and trigger remediation runbooks - useful when you want action on specific error patterns without writing custom alert scripts.

# Route SSH events, filter noise, forward auth to SIEM
# /etc/rsyslog.d/20-routing.conf

# Drop debug from a noisy daemon
if $programname == "haproxy" and $syslogseverity == 7 then stop

# SSH auth to dedicated file
if $programname == "sshd" then {
    action(type="omfile" file="/var/log/ssh-auth.log")
}

# JSON template for SIEM
template(name="SIEMJson" type="list") {
    constant(value="{")
    property(name="timereported" outname="timestamp"
             dateFormat="rfc3339" format="jsonf")
    constant(value=",")
    property(name="hostname" format="jsonf")
    constant(value=",")
    property(name="syslogtag" format="jsonf")
    constant(value=",")
    property(name="syslogseverity-text" outname="severity" format="jsonf")
    constant(value=",")
    property(name="msg" format="jsonf")
    constant(value="}\n")
}

# Forward auth events to SIEM over TCP with JSON
if $syslogfacility-text == "authpriv" or
   $syslogfacility-text == "auth" then {
    action(type="omfwd"
           target="siem.internal"
           port="10514"
           protocol="tcp"
           template="SIEMJson"
           queue.type="LinkedList"
           queue.size="10000")
}
// advertisement

Log Rotation with logrotate

rsyslog writes log files but does not rotate them. logrotate handles rotation, compression, and retention. On RHEL, /etc/logrotate.d/syslog is installed by the rsyslog package. On Debian, /etc/logrotate.d/rsyslog is provided.

The default RHEL rotation rotates weekly, keeps 4 weeks, and sends SIGHUP to rsyslog after rotation. SIGHUP causes rsyslog to close and reopen all file handles - this is the correct signal, not SIGTERM.

For a centralized log server receiving per-host files under /var/log/remote/, the default logrotate config does not cover these dynamic paths. You need a wildcard glob or a separate logrotate stanza. Use the missingok flag so logrotate does not error when a host-specific file is absent.

Rotation frequency should match your disk capacity and retention policy. A server receiving 1 GB/day of raw logs needs either daily rotation with compression or a minimum of 30 days of disk allocated per retained day. gzip compression typically achieves 10:1 to 20:1 on syslog text - a 1 GB daily log compresses to 50-100 MB.

After changing logrotate config, test it without actually rotating: logrotate -d /etc/logrotate.d/rsyslog runs a dry run and prints what it would do. Force an actual rotation with logrotate -f to verify rsyslog reacquires the file handle correctly.

Postrotate script should use systemctl kill -s HUP rsyslog rather than killall or pkill, because systemctl correctly resolves the PID and handles cgroup membership on systemd hosts.

# Custom logrotate for centralized remote logs
cat > /etc/logrotate.d/rsyslog-remote << 'EOF'
/var/log/remote/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl kill -s HUP rsyslog.service 2>/dev/null || true
    endscript
}
EOF

# Dry run
logrotate -d /etc/logrotate.d/rsyslog-remote

# Force rotation to test postrotate
logrotate -f /etc/logrotate.d/rsyslog-remote

# Verify rsyslog re-opened the file (inode change)
ls -i /var/log/remote/webserver01/$(date +%Y-%m-%d).log
lsof -p $(systemctl show rsyslog --property MainPID | cut -d= -f2) | grep remote

Auditing and Security Hardening for rsyslog

rsyslog itself is a security-relevant process. A compromised rsyslog can suppress audit trails. Harden the deployment on three levels: process isolation, file permissions, and immutability.

File permissions: rsyslog runs as root by default on most distributions. The PrivilegeDrop feature in rsyslog 8.x allows dropping to a dedicated user after binding to privileged ports. Set $PrivDropToUser rsyslog and $PrivDropToGroup rsyslog in the global config. Log files under /var/log/ should be owned by root:adm with mode 640 - readable by administrators, not by application users.

Systemd hardening: override the rsyslog service unit with systemd-edit to add ProtectSystem=strict, PrivateTmp=yes, NoNewPrivileges=yes, and RestrictSUIDSGID=yes. This prevents rsyslog from writing outside explicitly configured paths and blocks privilege escalation via SUID binaries.

Log integrity: if you need tamper-evident logs, rsyslog supports log signing via the lmsig_ksi module (Guardtime KSI integration) or you can pipe logs through a hash chain using a custom action. A simpler approach for most environments: ship logs to a write-once remote target (WORM storage, or a log receiver that the sending hosts cannot SSH into) within seconds of generation.

Monitoring rsyslog itself: rsyslog exposes internal statistics via the impstats module. Enable it to track queue depth, dropped messages, and action failures. Write impstats output to a separate file and alert on queue depth exceeding 80% of configured size.

For /etc/rsyslog.conf ownership: the file should be root:root 640. Any world-readable configuration is a minor information leak (reveals log routing, SIEM addresses). On hosts where multiple teams have sudo access, this matters.

# Systemd hardening override
systemctl edit rsyslog
# Add:
[Service]
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
RestrictSUIDSGID=yes
ReadWritePaths=/var/log /var/spool/rsyslog

# Enable internal stats monitoring
cat >> /etc/rsyslog.d/99-stats.conf << 'EOF'
module(load="impstats"
       interval="60"
       severity="6"
       log.syslog="off"
       log.file="/var/log/rsyslog-stats.log")
EOF

# Check queue depth from stats log
tail -f /var/log/rsyslog-stats.log | grep -i queue

# Verify privilege drop is active
ps aux | grep rsyslog
# Should show rsyslog user, not root, after drop

# Check file permissions
ls -la /etc/rsyslog.conf /var/log/messages /var/log/secure

Troubleshooting Common rsyslog Problems

rsyslog problems fall into four categories: messages not appearing, messages duplicated, high CPU or memory, and remote forwarding failures.

Messages not appearing: first check if rsyslog is running (systemctl status rsyslog), then check if the relevant input module is loaded (grep imuxsock /etc/rsyslog.conf), then send a test message and watch the log file in real time:

logger -p auth.info "test message from $HOSTNAME" && tail -f /var/log/secure

If the message does not appear, check if systemd journal is capturing it instead: journalctl -f. On systemd hosts with imjournal configured, rsyslog reads from the journal rather than /dev/log. If the journal is backed up, rsyslog stalls. Check journal cursor lag: journalctl --disk-usage and systemctl status systemd-journald.

Messages duplicated: usually caused by both imuxsock and imjournal being loaded simultaneously, or by a log file being matched by two separate rules without a stop action. Audit your ruleset with rsyslogd -N1 and grep for duplicate action targets.

High CPU: most often caused by debug logging at high volume. Find the source: journalctl -f --priority=debug | head -100 to identify the noisy program, then add a drop rule. Also check if rsyslog is doing disk I/O on every message - sync after write is enabled by default on some distributions. Prefix the output file path with a hyphen to disable sync: action(type="omfile" file="-/var/log/messages").

Remote forwarding failures: check the action queue depth with impstats, verify TCP connectivity with nc -zv logserver 514, and check TLS certificate expiry with openssl s_client -connect logserver:6514. Certificate expiry is the most common cause of silent TLS failures - rsyslog queues messages without alerting when TLS handshake fails repeatedly.

# Test local logging pipeline end to end
logger -p auth.warning "rsyslog test $(date +%s)"
grep 'rsyslog test' /var/log/secure

# Check if journal is stalling rsyslog
journalctl -u rsyslog --since "5 minutes ago" | grep -i error

# Disable sync-on-write for performance (hyphen prefix)
# In /etc/rsyslog.d/00-base.conf, change:
# file="/var/log/messages" to file="-/var/log/messages"

# Check remote forwarding queue depth
tail -n 50 /var/log/rsyslog-stats.log | grep -A2 'action 0'

# Test TCP connectivity to log server
nc -zv logserver.internal 514
nc -zv logserver.internal 6514

# Check TLS cert expiry
openssl s_client -connect logserver.internal:6514 \
  -CAfile /etc/pki/rsyslog/ca.pem 2>/dev/null | \
  openssl x509 -noout -dates
// advertisement