Understanding the Three Layers You Actually Need
Intrusion detection on Linux splits into two domains: host-based (HIDS) and network-based (NIDS). Most teams pick one and ignore the other, which leaves obvious blind spots. A rootkit that never touches the network evades Suricata entirely. A port scan that never touches disk evades AIDE. You need both.
The third layer, auditd, sits underneath both. It records syscalls - who opened which file, which process executed which binary, which user escalated privileges. When AIDE flags a changed file, auditd tells you what process changed it and under which UID. When Suricata fires on outbound traffic, auditd tells you which process opened that socket.
auditd ships with the kernel audit subsystem, so it has near-zero overhead and cannot be silently disabled by a userspace process without leaving a trace. AIDE runs as a scheduled job and is offline by design - it compares a known-good database against the current filesystem. Suricata runs inline or as a passive tap on your network interface.
On a typical VM with 4 vCPUs and 8GB RAM, we measured: auditd at under 1% CPU even with aggressive rules, AIDE scans at 2-4% CPU during the scan window (not constant), and Suricata at 3-8% CPU depending on ruleset size and traffic volume. These numbers are acceptable for production.
# Verify kernel audit subsystem is available
auditctl -s
# Expected output includes:
# enabled 1
# pid
# rate_limit 0 Setting Up auditd with Useful Rules
Install auditd first because it is the foundation the other tools depend on for context.
On RHEL/Rocky/Alma 9: `dnf install audit audit-libs` On Debian/Ubuntu: `apt install auditd audispd-plugins`
The default configuration logs almost nothing useful. The real work is writing rules. Drop rules into `/etc/audit/rules.d/` as `.rules` files. They are loaded in filename order, so prefix them with numbers.
The ruleset below covers the events that matter most: privilege escalation, file permission changes on sensitive paths, execution of new binaries, and modifications to the audit rules themselves. The `-F auid!=4294967295` filter excludes unset audit UIDs, which eliminates noise from kernel threads.
After loading rules, verify them with `auditctl -l`. Any rule that fails to load silently is a gap in your coverage. Always check.
For log shipping, configure `/etc/audit/auditd.conf` to set `log_format = ENRICHED` - this resolves UIDs and GIDs to names at write time, which matters if accounts are deleted before you investigate.
# /etc/audit/rules.d/99-harden.rules
# Delete all existing rules first
-D
# Buffer size
-b 8192
# Failure mode: 1=log, 2=panic
-f 1
# Monitor writes to /etc/passwd, /etc/shadow, /etc/sudoers
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k priv_escalation
-w /etc/sudoers.d/ -p wa -k priv_escalation
# Monitor SSH config changes
-w /etc/ssh/sshd_config -p wa -k sshd_config
# Monitor cron directories
-w /etc/cron.d/ -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
# Monitor setuid/setgid execution
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k root_commands
# Monitor module loading
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k modules
# Monitor audit rule changes (detect tampering)
-w /etc/audit/ -p wa -k audit_rules
-w /sbin/auditctl -p x -k audit_tools
# Lock rules - must be last, requires reboot to change
-e 2AIDE: File Integrity Monitoring That Works
AIDE (Advanced Intrusion Detection Environment) works by building a cryptographic database of your filesystem at a known-good point, then comparing against it on schedule. Version 0.18 introduced parallel hashing which cuts initial database build time roughly in half on SSD-backed systems.
Install: `apt install aide` or `dnf install aide`
The default configuration at `/etc/aide/aide.conf` (Debian) or `/etc/aide.conf` (RHEL) monitors too much and too little simultaneously. It includes `/var/log` (noisy, low signal) and misses custom application directories. Edit it before initializing.
The custom ruleset below defines a `PERMS` check for files where content changes are expected but permission changes are not, and a `CONTENT` check for binaries where any change is suspicious.
Build the initial database immediately after a clean install, before you deploy application code:
``` aide --init mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db ```
On our test server with a standard Ubuntu 24.04 install plus a LAMP stack, the initial scan took 4 minutes and produced a 47MB database. Subsequent checks against that baseline take under 90 seconds.
Schedule checks with systemd timers, not cron - timers survive clock jumps and give you proper logging. The timer unit below runs AIDE at 03:00 daily and mails output to root. Set `MAILTO` in your MTA or pipe to a Slack/PagerDuty webhook via a wrapper script. If you are building automation around this, task orchestration platforms like taskbotshub.ai can route AIDE alert output to incident tickets without writing custom glue code.
# /etc/aide/aide.conf.d/99-custom.conf
# Define check groups
PERMS = p+u+g+acl+selinux
CONTENT = sha512+size+p+u+g
FULL = sha512+size+p+u+g+n+i+acl+selinux
# System binaries - any change is an incident
/bin CONTENT
/sbin CONTENT
/usr/bin CONTENT
/usr/sbin CONTENT
/usr/local/bin CONTENT
/lib CONTENT
/lib64 CONTENT
# Config files - monitor everything
/etc FULL
# Exclude noisy paths
!/etc/mtab
!/etc/.*~
!/var/lib/aide
!/var/log
!/tmp
!/proc
!/sys
!/dev
!/runAIDE Systemd Timer for Daily Checks
Cron-based AIDE checks are common but fragile. If your system clock drifts or the cron daemon restarts during a check, you get partial output or silence. Systemd timers handle these cases correctly and log to journald automatically.
Create both a service unit and a timer unit. The service runs the check and sends output. The timer triggers the service. Check timer status with `systemctl list-timers aide-check.timer`.
# /etc/systemd/system/aide-check.service
[Unit]
Description=AIDE filesystem integrity check
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/bin/aide --check
StandardOutput=journal
StandardError=journal
SyslogIdentifier=aide-check
---
# /etc/systemd/system/aide-check.timer
[Unit]
Description=Daily AIDE integrity check
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target
---
# Enable and start
systemctl daemon-reload
systemctl enable --now aide-check.timer
# Verify
systemctl list-timers aide-check.timerInstalling and Configuring Suricata 7.x
Suricata 7.0 (released late 2023, still the stable branch in 2026 with 7.0.x point releases) added significant improvements to HTTP/2 and TLS fingerprinting that matter for modern threat detection. Use the OISF PPA or the official COPR repo rather than distro packages - Ubuntu 24.04 ships 6.x which is missing several rule categories.
On Ubuntu 24.04: ``` add-apt-repository ppa:oisf/suricata-stable apt update apt install suricata ```
On RHEL 9: ``` dnf copr enable @oisf/suricata dnf install suricata ```
After install, update the Emerging Threats ruleset before starting: ``` suricata-update ```
This pulls ~50,000 rules. The default configuration enables all of them, which will flood you with false positives on most networks. Use the rule management workflow below to suppress noisy categories and tune thresholds.
Set the `HOME_NET` variable correctly in `/etc/suricata/suricata.yaml` - this is the single most important configuration item. Suricata uses it to determine which side of a connection is internal and which rules apply directionally.
For a server (not a network gateway), run Suricata in AF_PACKET mode against your primary interface. On our test server with a 1Gbps interface, AF_PACKET with 4 worker threads handled 600Mbps sustained without drops. For higher throughput, use AF_XDP or DPDK, but those require additional kernel module setup.
# /etc/suricata/suricata.yaml - critical sections to configure
vars:
address-groups:
HOME_NET: "[192.168.1.0/24, 10.0.0.0/8]"
EXTERNAL_NET: "!$HOME_NET"
HTTP_SERVERS: "$HOME_NET"
SQL_SERVERS: "$HOME_NET"
af-packet:
- interface: eth0
threads: 4
cluster-id: 99
cluster-type: cluster_flow
defrag: yes
use-mmap: yes
mmap-locked: yes
tpacket-v3: yes
ring-size: 200000
block-size: 32768
logging:
outputs:
- eve-log:
enabled: yes
filetype: regular
filename: /var/log/suricata/eve.json
types:
- alert:
payload: yes
payload-printable: yes
metadata: yes
- http:
extended: yes
- dns:
- tls:
extended: yesTuning Suricata Rules to Reduce False Positives
Raw Emerging Threats rules on a production server will alert on legitimate software behavior - package managers hitting CDN infrastructure, monitoring agents making unusual TCP connections, TLS fingerprints that match known-bad but also known-good software. Untuned Suricata becomes noise you ignore, which defeats the purpose.
The practical workflow: run Suricata for 72 hours in IDS mode (not IPS), collect all alerts from `/var/log/suricata/eve.json`, then build a suppression list for any rule that fires more than 50 times against known-good traffic.
Query the JSON log with `jq` to find your top firing rules:
``` jq -r 'select(.event_type=="alert") | .alert.signature_id' /var/log/suricata/eve.json | sort | uniq -c | sort -rn | head -20 ```
For each high-volume rule SID, decide: suppress entirely, suppress for specific source/dest, or tune the threshold. Add suppressions to `/etc/suricata/threshold.conf`.
After tuning, reload rules without restarting the daemon: ``` systemctl kill -s USR2 suricata ```
For rules you want to keep but reduce alert volume, use the `threshold` directive with `type limit` to get one alert per source IP per hour rather than one per packet. This preserves signal without creating 10,000 alert events from a single port scan.
# /etc/suricata/threshold.conf
# Suppress noisy DNS rule globally
suppress gen_id 1, sig_id 2027758
# Suppress specific rule for known monitoring agent IP
suppress gen_id 1, sig_id 2013028, track by_src, ip 10.0.0.15
# Rate-limit port scan alert to 1 per hour per source
threshold gen_id 1, sig_id 2010935, type limit, track by_src, count 1, seconds 3600
# Reload after editing:
# systemctl kill -s USR2 suricataWiring the Three Tools Together with Centralized Logging
Three separate log streams are manageable on one server. At five servers it becomes unworkable. The standard approach in 2026 is shipping all three log sources to a central collector. We use Vector (from Datadog, not the now-defunct startup) because it handles both auditd and Suricata's EVE JSON natively with zero parsing configuration.
Install Vector 0.40.x on each monitored host: ``` curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | bash apt install vector ```
The Vector config below reads from journald (which captures auditd output via the journal), reads Suricata's EVE JSON file directly, and ships both to a central syslog endpoint or Elasticsearch cluster. AIDE output goes to journald via the systemd service we configured earlier, so it arrives through the same journald source.
For teams running automated incident response, the EVE JSON stream from Suricata is particularly easy to process programmatically. If your DevOps workflow already uses AI-assisted automation via platforms like taskbotshub.ai, the structured JSON output from Suricata can feed directly into automated triage playbooks without custom parsing.
One important detail: set `encoding.codec = "json"` on your Vector sink, not text. Downstream tools, including Elasticsearch, Splunk, and Loki, handle structured JSON far better than raw syslog strings for security event correlation.
# /etc/vector/vector.yaml
sources:
journald_source:
type: journald
include_units:
- auditd
- aide-check
current_boot_only: true
suricata_eve:
type: file
include:
- /var/log/suricata/eve.json
read_from: beginning
data_dir: /var/lib/vector
transforms:
parse_eve:
type: remap
inputs:
- suricata_eve
source: |
. = parse_json!(.message)
.source = "suricata"
tag_audit:
type: remap
inputs:
- journald_source
source: |
.source = "auditd"
sinks:
central_loki:
type: loki
inputs:
- parse_eve
- tag_audit
endpoint: "http://loki.internal:3100"
labels:
host: "{{ host }}"
source: "{{ source }}"
encoding:
codec: jsonResponding to an Alert: Practical Investigation Workflow
When AIDE flags `/usr/bin/curl` as modified, the first question is not 'was this a legitimate update?' but 'what process touched this file and when?' That is where auditd earns its place.
Search auditd logs for file writes to the flagged path: ``` auditgrep /usr/bin/curl # or with ausearch: ausearch -f /usr/bin/curl --start today ```
The output will include the syscall, the PID, the UID, the AUID (the original login UID that survives sudo), and a timestamp. Cross-reference the timestamp with Suricata alerts from the same window:
``` jq 'select(.event_type=="alert" and (.timestamp | fromdateiso8601) > 1753900000)' /var/log/suricata/eve.json ```
If the auditd record shows `auid=1001` (a normal user account) modified a system binary, and Suricata shows outbound traffic from the same host to a known C2 IP within the same five-minute window, you have a confirmed incident. Isolate the host immediately.
For isolation on a cloud VM, use your provider's security group API to drop all traffic except your management IP before you begin forensics. On bare metal, `iptables -I INPUT 1 -s YOUR_MGMT_IP -j ACCEPT && iptables -I INPUT 2 -j DROP` achieves the same result. Do not power off - you lose volatile memory including process list, open file handles, and network connections.
Capture volatile state first: ``` ps auxef > /tmp/forensics-ps.txt ss -tlnp > /tmp/forensics-ss.txt lsof -nP > /tmp/forensics-lsof.txt find /proc/*/exe -ls 2>/dev/null > /tmp/forensics-proc-exe.txt ```
Then copy these files off the host before you do anything else.
# Quick volatile state capture - run as root before any remediation
mkdir -p /tmp/forensics-$(date +%Y%m%d-%H%M%S)
cd /tmp/forensics-$(date +%Y%m%d-%H%M%S)
ps auxef > ps.txt
ss -tlnp > ss.txt
lsof -nP > lsof.txt
netstat -rn > routes.txt
find /proc/*/exe -ls 2>/dev/null > proc_exe.txt
cat /proc/*/net/tcp 2>/dev/null | sort -u > tcp_connections.txt
w > logged_in_users.txt
lastlog > lastlog.txt
cat /var/log/auth.log | grep -E '(Failed|Accepted|sudo)' | tail -200 > auth.txt
# Hash everything
sha256sum * > checksums.sha256
# Copy off-host immediately
tar czf /tmp/forensics.tar.gz /tmp/forensics-*
scp /tmp/forensics.tar.gz analyst@forensics.internal:/evidence/Hardening the IDS Stack Itself
An IDS that can be silently disabled by an attacker is theater. A few specific hardening steps matter.
For auditd: the `-e 2` flag in your rules file locks the audit configuration until reboot. Without it, any root process can `auditctl -D` to flush all rules and stop logging. Verify the lock is set: `auditctl -s | grep enabled` should return `enabled 2`.
For AIDE: the database must live somewhere the attacker cannot reach. Storing `/var/lib/aide/aide.db` on the same filesystem it is monitoring is only slightly better than nothing. On a cloud host, write the initial database to an S3 bucket or equivalent with write-once policies. On bare metal, write it to read-only NFS or a USB drive that you physically remove. At minimum, compare a fresh database against a copy stored on a separate host.
For Suricata in IPS mode (inline, not just detection): use `nfqueue` mode if you want blocking. Be conservative - a misconfigured IPS rule that blocks legitimate traffic is an outage. Run in IDS mode for at least 30 days, tune thoroughly, then switch specific high-confidence rule categories to `drop` rather than `alert`. Never switch the entire ruleset to drop at once.
The auditd daemon itself should run as a protected service. Verify it is configured to respawn if killed: ``` grep -E '(active_when_full|flush|space_left_action)' /etc/audit/auditd.conf ``` `space_left_action = SYSLOG` and `disk_full_action = SUSPEND` are the safe defaults. `disk_full_action = HALT` is appropriate for high-security environments that prefer downtime over losing audit coverage.
# Verify auditd lock is active
auditctl -s | grep enabled
# Should return: enabled 2
# Verify AIDE database is NOT world-readable
ls -la /var/lib/aide/aide.db
# Should be: -rw------- root root
# Check Suricata is running and processing packets
suricata-sc --connect /var/run/suricata/suricata-command.socket
# At the prompt:
# uptime
# iface-stat
# Verify auditd is not disableable by non-root
id
# Confirm you are root, then:
auditctl -s | grep enabled
# 'enabled 2' means locked - even root cannot disable without rebooting