How auditd Works at the Kernel Level
auditd sits between userspace and the kernel audit subsystem. The kernel generates audit events for syscalls, file accesses, and authentication events. The audit daemon collects those events via a netlink socket and writes them to /var/log/audit/audit.log. The dispatcher (audispd, now merged into auditd 3.x as plugins) can forward events to syslog or SIEM tools in parallel.
The key components are: the kernel audit subsystem (built into the kernel, not a module), auditd (userspace daemon), auditctl (rule management CLI), ausearch (log query tool), and aureport (summary reporting). On RHEL/CentOS/Rocky, the package is audit. On Debian/Ubuntu it splits into auditd and audispd-plugins.
One architectural detail that bites people: audit rules are stateful per boot. Rules loaded with auditctl are lost on reboot. Persistent rules go in /etc/audit/rules.d/*.rules and are compiled into /etc/audit/audit.rules by augenrules on startup. Do not edit /etc/audit/audit.rules directly - it gets overwritten.
# Check kernel audit support
grep -i audit /boot/config-$(uname -r) | grep -E 'CONFIG_AUDIT=|CONFIG_AUDITSYSCALL='
# Should show:
# CONFIG_AUDIT=y
# CONFIG_AUDITSYSCALL=y
# Check auditd version
auditd --version
# auditd version 3.1.2
# Check current rule count
auditctl -l | wc -lInstallation and Initial Configuration
On RHEL 9 and derivatives, auditd is installed by default. On Ubuntu 24.04, install it manually and enable the service.
The main configuration file is /etc/audit/auditd.conf. The defaults are reasonable for most systems, but two settings matter immediately: log_file (where logs go), and max_log_file_action (what happens when the log fills up). The default ROTATE is fine for most environments. For high-security systems where losing audit data is unacceptable, set it to SUSPEND or HALT - the system will freeze rather than lose audit records.
On busy systems, increase the kernel audit backlog buffer. The default of 8192 events is too small for database servers or systems with heavy file I/O. Increase it by adding -b 16384 to your rules. If the buffer fills, events are either dropped or the kernel panics depending on the failure_mode setting. Check /proc/net/netlink for dropped events after load testing.
# Ubuntu 24.04
apt install auditd audispd-plugins -y
systemctl enable --now auditd
# RHEL 9
# Already installed, but verify:
rpm -q audit
# audit-3.1.2-1.el9.x86_64
# Key settings in /etc/audit/auditd.conf
grep -E 'log_file|max_log_file|num_logs|space_left' /etc/audit/auditd.conf
# Recommended production values:
# log_file = /var/log/audit/audit.log
# num_logs = 10
# max_log_file = 100
# max_log_file_action = ROTATE
# space_left = 500
# space_left_action = SYSLOG
# admin_space_left = 50
# admin_space_left_action = SUSPEND
# Set backlog buffer in /etc/audit/rules.d/00-base.rules
echo '-b 16384' > /etc/audit/rules.d/00-base.rules
augenrules --loadWriting Audit Rules: Syntax and Real Examples
Audit rules use three types: control rules (-b, -f, -e, -r), file system watch rules (-w), and syscall rules (-a). File watch rules are the easiest entry point but syscall rules give you far more control.
File watch syntax is: -w path -p permissions -k keyname. Permissions are r (read), w (write), x (execute), a (attribute change). The key (-k) is a label you use later with ausearch to find related events.
Syscall rules are more powerful. The syntax is: -a action,list -S syscall -F field=value -k key. Action is always or never. List is task, exit, user, or exclude. For most monitoring use -a always,exit.
The critical rule ordering: rules are evaluated in order. Put -a never,exit rules before -a always,exit rules to exclude noisy processes. Also put the immutable flag (-e 2) as the absolute last rule - it locks the ruleset until reboot and prevents tampering, but also prevents you from adding more rules without rebooting.
For CIS Benchmark compliance on RHEL 9, you need roughly 40-60 rules covering privilege escalation, identity changes, file permission modifications, and network configuration changes. The audit-rules package on RHEL provides starter rulesets at /usr/share/doc/audit/rules/.
# File watch rules - monitor /etc/passwd for writes and attribute changes
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d/ -p wa -k scope
# Syscall rules - monitor privilege escalation
-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation
-a always,exit -F arch=b32 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation
# Monitor execve (command execution) by specific users
-a always,exit -F arch=b64 -S execve -F uid=0 -k root_commands
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k user_commands
# Monitor changes to cron jobs
-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
# Monitor kernel module loading
-w /sbin/insmod -p x -k modules
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules
# Exclude noisy auditd itself from syscall logging
-a never,exit -F subj_type=auditd_t
# Lock rules - put this LAST
-e 2Monitoring Specific High-Value Targets
Generic rules generate enormous logs. Targeted rules for high-value files produce actionable data. The following patterns cover the most common forensic and compliance requirements we encounter on production systems.
For SSH hardening audits, watch the authorized_keys files across all home directories. The -F dir= filter lets you scope to a directory tree without listing individual files.
For detecting lateral movement, monitor /proc/*/maps and /proc/*/mem reads, which can indicate process injection. This generates significant volume on busy hosts, so restrict to specific UIDs or use a rate limit with -F success=0 to only log failures if you need to reduce noise.
For database servers, you often need to track which OS user invokes mysql or psql. Combine execve monitoring with auid filtering. The auid (audit UID) is set at login time and persists through sudo and su transitions - it is more reliable than uid for tracking the original human operator.
On containers and Kubernetes nodes, auditd runs on the host kernel and will capture syscalls from all containers. This is both powerful and noisy. Use cgroup-based filtering with -F cgroup= (available in kernel 5.15+) to scope rules to specific containers.
# Monitor all authorized_keys files in /home
-a always,exit -F arch=b64 -S open -S openat -F dir=/home -F name=authorized_keys -F perm=wa -k ssh_keys
# Monitor /root/.ssh
-w /root/.ssh/ -p wa -k ssh_root
# Track who runs su and sudo
-w /bin/su -p x -k su_exec
-w /usr/bin/sudo -p x -k sudo_exec
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/sudo -k sudo_exec
# Monitor failed file access (permission denied events)
-a always,exit -F arch=b64 -S open -F dir=/etc -F success=0 -k etc_access_fail
-a always,exit -F arch=b64 -S open -F dir=/var/log -F success=0 -k log_access_fail
# Network configuration changes
-a always,exit -F arch=b64 -S sethostname -S setdomainname -k network_config
-w /etc/hosts -p wa -k network_config
-w /etc/hostname -p wa -k network_config
-w /etc/network/ -p wa -k network_config
-w /etc/NetworkManager/ -p wa -k network_config
# Load all rules
augenrules --load
auditctl -l | grep -c 'arch=' # Count syscall rules loadedQuerying Logs with ausearch and aureport
The raw audit.log is structured but dense. ausearch queries events by key, time, UID, syscall, or PID. aureport generates tabular summaries suitable for daily review or feeding into monitoring dashboards.
ausearch by default searches /var/log/audit/audit.log. Use -if to point it at a specific file or rotated log. The --interpret flag (-i) converts numeric UIDs, syscall numbers, and permissions to human-readable strings - always use it.
audit.log uses a multi-record format: one logical event spans multiple AUDIT_SYSCALL, AUDIT_CWD, AUDIT_PATH, and AUDIT_PROCTITLE records, all sharing the same msg timestamp and serial number. ausearch reassembles these into complete events automatically.
For incident response, the most useful query is 'what did user X do between time A and time B'. Combine -ua (by auid), --start, and --end. Times use the format HH:MM:SS or MM/DD/YYYY HH:MM:SS.
aureport generates daily summaries quickly. The --failed flag shows only failed events, which is the fastest way to spot brute-force attempts or permission probing. Pipe aureport output into your log aggregator or use it as input for cron-generated reports.
# Find all events with key 'identity'
ausearch -k identity -i
# Find events by specific user (auid = audit UID from login)
ausearch -ua 1001 --start today -i
# Find all sudo executions in the last hour
ausearch -k sudo_exec --start recent -i
# Find failed login attempts
ausearch -m USER_AUTH -sv no -i
# Find all commands run as root today
ausearch -k root_commands --start today --end now -i
# Summary reports
aureport --summary # Overall summary
aureport -au # Authentication report
aureport -au --failed # Failed auth only
aureport -x --summary # Executable summary (top commands run)
aureport -f -i # File access report
aureport --login --failed -i # Failed logins with interpretation
# Find events between specific times
ausearch --start 07/15/2026 08:00:00 --end 07/15/2026 10:00:00 -k scope -i
# Count events by type today
aureport -t --start today 2>/dev/null | sort -k2 -rn | head -20
# Raw log parsing for pipeline use
audit2allow -i /var/log/audit/audit.log 2>/dev/null # SELinux policy from audit denialsForwarding Audit Events to a SIEM
Storing audit logs only on the local system defeats the purpose if an attacker can clear them after compromise. Forward events to a central log server or SIEM.
The audisp-remote plugin ships with audispd-plugins and sends events over TCP to a remote auditd instance. Configure it in /etc/audit/plugins.d/au-remote.conf and set the remote server in /etc/audisp/audisp-remote.conf. This uses a proprietary binary protocol, not syslog, so the receiving end must also be auditd with the remote listener configured.
For Elasticsearch, Splunk, or similar, the audisp-syslog plugin converts events to syslog format. Enable it, then use rsyslog or syslog-ng to route the AUDIT facility to your aggregator.
If you are running a modern observability stack, auditbeat from the Elastic stack is often simpler. It reads the kernel audit socket directly, bypasses auditd, and ships structured JSON to Elasticsearch. Do not run auditd and auditbeat simultaneously - they both compete for the kernel audit netlink socket. Choose one.
For DevOps teams already using automation platforms, tools like taskbotshub.ai can help orchestrate compliance checks and audit log pipeline deployment across fleets of servers, reducing the manual overhead of ensuring every node sends events to your central collector.
# Enable syslog forwarding via audisp-syslog plugin
cat > /etc/audit/plugins.d/syslog.conf << 'EOF'
active = yes
direction = out
path = builtin_syslog
type = builtin
args = LOG_INFO
format = string
EOF
# Restart auditd to activate plugin
service auditd restart
# Note: 'systemctl restart auditd' may not reload plugins on all distros
# Use 'service auditd restart' or 'pkill -HUP auditd'
# Verify syslog is receiving audit events
tail -f /var/log/syslog | grep 'type=SYSCALL'
# Remote auditd forwarding (remote plugin)
# /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always
format = string
# /etc/audisp/audisp-remote.conf
remote_server = 192.168.1.50
port = 60
mode = immediate
queue_file = /var/spool/audit/remote.log
network_retry_time = 1
max_tries_per_record = 3CIS Benchmark Compliance: Specific Rules Required
CIS Benchmark for RHEL 9 (version 2.0) requires specific audit rules in sections 4.1.x. Rather than transcribe all of them, here is what automated scanning tools (OpenSCAP, Lynis) actually check for, and the fastest path to compliance.
OpenSCAP with the CIS profile checks both the presence of rules and their exact syntax. Use oscap to scan and generate an HTML remediation guide. The xccdf_org.ssgproject.content_rule_audit_rules_* IDs map directly to specific rule requirements.
Lynis audit checks run during lynis audit system. The hardening index from the AUDIT section will show exactly which rules are missing. We ran Lynis 3.1.0 on a stock Ubuntu 24.04 server with no audit rules: hardening index came out at 41. After loading the RHEL-equivalent CIS rules adapted for Ubuntu, it reached 78.
The fastest path to CIS compliance on RHEL 9 is the audit-rules package plus the scap-security-guide remediation script. This automates rule deployment to /etc/audit/rules.d/ and handles the augenrules rebuild. If you are building this into a Packer template or Ansible playbook for server images, run augenrules --load as the final step after dropping rule files.
# Install OpenSCAP and CIS content on RHEL 9
dnf install openscap-scanner scap-security-guide -y
# List available CIS profiles
oscap info /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml | grep -i cis
# Run CIS Level 2 scan (audit-related checks only)
oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis \
--results /tmp/cis-results.xml \
--report /tmp/cis-report.html \
/usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
# Run Lynis audit
lynis audit system --tests-from-group audit
# Check which CIS audit rules are missing
auditctl -l | grep -c '' # Count loaded rules
# CIS L2 requires at least 35-40 active rules
# Apply CIS remediation automatically (RHEL 9)
oscap xccdf remediate \
--profile xccdf_org.ssgproject.content_profile_cis \
--results /tmp/cis-remediation.xml \
/usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xmlPerformance Impact and Tuning
Heavy audit rules have measurable performance cost. In our testing on a 4-core RHEL 9 VM doing 10,000 execve syscalls per second (simulated with a tight shell loop), enabling full CIS audit rules increased CPU usage by 8-12% and reduced execve throughput by about 7%. On database servers handling thousands of file opens per second, poorly scoped rules can cost 15-20% throughput.
To minimize impact, use -F arch= filters on every syscall rule. Without the arch filter, auditd evaluates the rule twice (once for b32, once for b64 ABI paths). Always pair b32 and b64 variants explicitly rather than omitting the filter.
Exclude noisy, low-risk paths explicitly. /proc, /sys, and /dev generate enormous event volume with almost no forensic value in most environments. Add exclusion rules at the top of your ruleset.
For the dispatcher, if you are using audisp plugins (especially syslog forwarding), set q_depth in /etc/audit/auditd.conf high enough to buffer bursts. Default is 2000. On busy systems, set it to 8192 or higher.
Check for dropped events periodically. A nonzero dropped count in auditctl -s means the kernel had to discard events because auditd could not keep up - your logs have gaps.
# Check kernel audit statistics
auditctl -s
# Look for: lost=0, backlog=0
# If lost > 0, increase -b value and investigate auditd CPU usage
# Exclude noisy virtual filesystems
# Add to top of rules, before any -a always,exit rules:
-a never,exit -F fstype=proc
-a never,exit -F fstype=sysfs
-a never,exit -F fstype=devtmpfs
-a never,exit -F fstype=cgroup2
-a never,exit -F fstype=tmpfs
# Exclude specific high-noise processes by exe path
-a never,exit -F arch=b64 -S open -F exe=/usr/sbin/rsyslogd
-a never,exit -F arch=b64 -S open -F exe=/usr/bin/python3
# Monitor audit performance
watch -n5 'auditctl -s | grep -E "lost|backlog|enabled"'
# Disk I/O from audit log - check write rate
iostat -x 1 5 | grep -A2 'Device'
# Tune dispatcher queue depth in auditd.conf
# q_depth = 8192
# Reload after change:
pkill -HUP auditdAutomating Audit Rule Deployment with Ansible
Manually maintaining audit rules across dozens or hundreds of servers is not realistic. The correct approach is treating /etc/audit/rules.d/ as code: version-controlled, deployed via Ansible or a similar tool, validated after deployment.
The audit role from ansible-lockdown (specifically the RHEL9-CIS role) handles CIS-compliant audit rule deployment with variables for each rule group. If you prefer a lighter touch, a simple Ansible task deploys your custom rule files and validates them.
After deploying rules, validate that augenrules compiled them correctly and the expected number of rules loaded. A common failure mode is a syntax error in one rule file that silently prevents all subsequent rules from loading.
For teams building this into a broader DevSecOps pipeline, automating compliance verification after every deployment catches rule regressions before they become audit findings. Platforms like taskbotshub.ai provide pre-built automation workflows that can run ausearch queries and aureport summaries on schedules, alerting on anomalies without requiring custom scripting for each check.
# Ansible task to deploy audit rules
---
- name: Deploy audit rules
hosts: all
become: true
tasks:
- name: Copy audit rule files
copy:
src: "{{ item }}"
dest: /etc/audit/rules.d/
owner: root
group: root
mode: '0640'
with_fileglob:
- files/audit_rules/*.rules
notify: reload audit rules
- name: Compile and load rules
command: augenrules --load
register: augenrules_result
changed_when: augenrules_result.rc == 0
- name: Verify rule count
command: auditctl -l
register: rule_list
changed_when: false
- name: Assert minimum rules loaded
assert:
that:
- rule_list.stdout_lines | length >= 35
fail_msg: "Fewer than 35 audit rules loaded - check for syntax errors"
- name: Check for dropped events
command: auditctl -s
register: audit_status
changed_when: false
- name: Warn on lost events
debug:
msg: "WARNING: audit events are being lost - increase backlog buffer"
when: "'lost=0' not in audit_status.stdout"
handlers:
- name: reload audit rules
command: pkill -HUP auditdInvestigating an Incident with auditd Data
When an incident occurs, the audit log is your primary source of ground truth. The workflow is: establish a time window, identify the actor by auid, trace their syscall sequence, correlate with file and network events.
Start with aureport --login to find all login events in the incident window. This gives you the auid values for sessions to investigate. Then use ausearch -ua
The PROCTITLE record (visible with -i flag) shows the full command line including arguments. The EXECVE record shows argv values. The PATH record shows the resolved filesystem path. Together, these reconstruct exactly what was run and on what files.
For supply chain or malware investigation, filter by execve and look for unusual parent processes. The ppid field in SYSCALL records shows the parent PID, letting you reconstruct process trees. Cross-reference PIDs with the process start time - PIDs recycle, so the timestamp in the msg= field is the authoritative correlation point.
If the audit log itself is missing entries (attacker may have cleared it), check the SELinux audit context. On RHEL, even root cannot truncate audit.log if audit_admin SELinux policy is enforced - the attempt itself generates an AVC denial in /var/log/messages.
# Step 1: Find logins in incident window
aureport --login --start 07/20/2026 00:00:00 --end 07/20/2026 23:59:59 -i
# Step 2: Pull all activity for suspect auid (e.g., 1042)
ausearch -ua 1042 --start 07/20/2026 06:00:00 --end 07/20/2026 08:00:00 -i > /tmp/incident_1042.txt
# Step 3: Extract just command executions
ausearch -ua 1042 -m EXECVE --start 07/20/2026 06:00:00 -i | grep -E 'proctitle|a[0-9]+='
# Step 4: Find files modified by that user
ausearch -ua 1042 -m PATH --start 07/20/2026 06:00:00 -i | grep 'name='
# Step 5: Check if audit log was tampered with
ausearch -m CONFIG_CHANGE -i
ausearch -m DAEMON_START -i # Unexpected restarts indicate clearing attempt
# Step 6: Reconstruct timeline sorted by timestamp
ausearch -ua 1042 --start 07/20/2026 06:00:00 -i --raw | \
awk -F'msg=audit\\(' '{print $2}' | \
sort -t: -k1 -n | head -50
# Verify log file integrity
sha256sum /var/log/audit/audit.log
stat /var/log/audit/audit.log # Check mtime vs last event timestamp