Start with a Snapshot: Capture System State Before You Change Anything

Before running any remediations, record baseline state. This gives you a diff after the audit and protects you if a change breaks something.

Capture the current list of listening ports, active users, and loaded kernel modules in one pass. Store the output somewhere outside the server being audited - an audit log on the same machine is only useful if it has not been tampered with.

# Capture baseline
date >> /tmp/audit-$(hostname)-$(date +%F).txt
ss -tulpn >> /tmp/audit-$(hostname)-$(date +%F).txt
who -a >> /tmp/audit-$(hostname)-$(date +%F).txt
lsmod >> /tmp/audit-$(hostname)-$(date +%F).txt
ps auxf >> /tmp/audit-$(hostname)-$(date +%F).txt

# Then SCP to your audit workstation immediately
scp /tmp/audit-*.txt auditor@10.0.0.5:/audits/

User Account Audit: Who Has Access and Why

User accounts are the most common entry point that survives infrastructure rebuilds. Check for accounts with valid login shells that are not supposed to be interactive, accounts with empty passwords, and users with UID 0 other than root.

The /etc/passwd file lists all accounts. Pipe it through awk to extract accounts with login shells, then cross-reference against /etc/shadow for password status.

# All accounts with interactive shells
grep -v '/nologin\|/false' /etc/passwd | awk -F: '{print $1, $3, $7}'

# Accounts with UID 0 (should only be root)
awk -F: '($3 == 0) {print}' /etc/passwd

# Accounts with empty passwords
sudo awk -F: '($2 == "" || $2 == "!") {print $1}' /etc/shadow

# Users currently logged in with last login timestamps
lastlog | grep -v 'Never logged in' | sort -k3

SSH Configuration: The Most Common Misconfiguration Surface

In our experience, default sshd_config values cause more compromises than anything else. Root login permitted, password authentication enabled, and no idle timeout are the three we see most often on servers that have been running for over a year without a review.

Check the active sshd configuration - not just the file, but what the daemon actually loaded, which may differ if include directives are in play.

# Show effective sshd config (includes all Include directives)
sshd -T 2>/dev/null | grep -E 'permitrootlogin|passwordauthentication|pubkeyauthentication|clientaliveinterval|clientalivecountmax|allowusers|permitemptypasswords|x11forwarding|protocol'

# What keys are authorized for root
cat /root/.ssh/authorized_keys 2>/dev/null

# Check for any authorized_keys files on the system
find / -name authorized_keys 2>/dev/null
// advertisement

Network Exposure: Open Ports and Firewall Rules

Every open port is an attack surface. The goal here is to identify ports that are listening but should not be, and to verify that your firewall rules actually match what you intended.

Use ss rather than netstat on modern systems. netstat is deprecated on Linux and missing from many distributions by default. Pay attention to the Local Address column - a service bound to 0.0.0.0 is reachable from any interface, while 127.0.0.1 is local only.

# All listening TCP and UDP sockets with process names
ss -tulpn

# Cross-reference with firewall rules (iptables)
sudo iptables -L -n -v --line-numbers

# For nftables (default on RHEL 9, Ubuntu 22.04+)
sudo nft list ruleset

# For ufw-managed systems
sudo ufw status verbose

# Identify which process owns a specific port
sudo ss -tulpn | grep ':443'
fuser 443/tcp

SUID and SGID Binaries: Privilege Escalation Vectors

SUID binaries run with the file owner's privileges regardless of who executes them. A misconfigured SUID binary owned by root gives any local user a path to root. This is one of the most reliable privilege escalation techniques, and it is frequently introduced by poorly written install scripts.

Run this find command and compare the output against a known good baseline. On a fresh Ubuntu 24.04 install, you should see around 30 SUID binaries. More than that warrants investigation.

# Find all SUID files
find / -perm -4000 -type f 2>/dev/null | sort

# Find all SGID files
find / -perm -2000 -type f 2>/dev/null | sort

# World-writable files and directories (excluding /proc, /sys)
find / -xdev -perm -0002 -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null | sort

# World-writable SUID files - high priority
find / -xdev -perm -4002 -type f 2>/dev/null

Cron Jobs and Scheduled Tasks: Persistence Mechanisms

Attackers frequently use cron to maintain persistence. A job that runs every 5 minutes calling a remote script keeps access alive even after you change passwords and rotate SSH keys. Audit all cron locations: system crontab, /etc/cron.d, per-user crontabs, and anacron.

Do not skip /var/spool/cron - this is where user-specific crons live and it is easy to miss. Also check systemd timers, which are used as cron replacements on modern systemd-based distributions.

# System-level cron
cat /etc/crontab
ls -la /etc/cron.d/ && cat /etc/cron.d/*
ls -la /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/

# Per-user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
  crontab -l -u $user 2>/dev/null && echo "--- $user ---"
done

# Systemd timers
systemctl list-timers --all

# at jobs
atq
// advertisement

Kernel and Sysctl Hardening: Check What Is Actually Running

The kernel configuration controls critical security behaviors: IP forwarding, ICMP redirects, SYN flood protection, and address space layout randomization. These settings should be locked down in /etc/sysctl.d/ and verified to match what is currently active in the kernel.

The key distinction is between what is in the config file and what the running kernel actually has. sysctl -a reads live kernel values. A mismatch means either the config was not applied after the last boot or something changed it at runtime.

# Check live kernel security parameters
sysctl kernel.randomize_va_space
sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.all.accept_redirects
sysctl net.ipv4.conf.all.send_redirects
sysctl net.ipv4.tcp_syncookies
sysctl kernel.dmesg_restrict
sysctl kernel.kptr_restrict

# Dump all sysctl values to a file for comparison
sysctl -a 2>/dev/null > /tmp/sysctl-live.txt

# Recommended hardening block for /etc/sysctl.d/99-hardening.conf
cat << 'EOF'
kernel.randomize_va_space = 2
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.tcp_syncookies = 1
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
net.ipv4.conf.all.rp_filter = 1
EOF

Installed Packages and Known Vulnerable Versions

Outdated packages with public CVEs are a leading cause of server compromise. The audit step here is two parts: identify packages that need updates, and identify packages that should not be installed at all - development tools, compilers, and debugging utilities rarely belong on production servers.

On Debian-based systems, use apt-get and debsecan. On RHEL-based systems, use dnf with the security plugin. Both give you CVE-mapped output when the vulnerability data is available.

# Debian/Ubuntu: packages with available updates
apt list --upgradable 2>/dev/null

# Debian/Ubuntu: CVE check (requires debsecan)
debsecan --suite noble --format detail 2>/dev/null | head -50

# RHEL/CentOS/Fedora: security updates only
dnf updateinfo list security --available

# Find compilers and dev tools on production systems
which gcc g++ cc make python3 perl ruby 2>/dev/null
dpkg -l | grep -E 'gcc|g\+\+|build-essential' 2>/dev/null   # Debian
rpm -qa | grep -E 'gcc|gcc-c\+\+|make|kernel-devel' 2>/dev/null   # RHEL

Log Audit: Evidence of Prior Access and Ongoing Intrusion

Logs tell you what has already happened. On a server with no active intrusion detection, the logs are often the only forensic record available. Focus on authentication logs, sudo usage, and any indication of lateral movement.

On Ubuntu and Debian, authentication events are in /var/log/auth.log. On RHEL and CentOS, they are in /var/log/secure. Both feed into journald on modern systems, so journalctl is the canonical way to query them regardless of distribution.

Look specifically for failed login bursts followed by a successful login - this is the fingerprint of a successful brute-force. Also look for sudo usage from accounts that should not have sudo access.

# Failed SSH logins in the last 24 hours
journalctl _SYSTEMD_UNIT=sshd.service --since '24 hours ago' | grep 'Failed password' | awk '{print $11}' | sort | uniq -c | sort -rn | head -20

# Successful root logins
journalctl _SYSTEMD_UNIT=sshd.service --since '30 days ago' | grep 'Accepted' | grep root

# Sudo usage
journalctl _COMM=sudo --since '7 days ago'

# Last 100 logins
last -n 100 | head -100

# New user accounts created recently
find /home -maxdepth 1 -newer /etc/passwd -type d 2>/dev/null
grep 'useradd\|adduser' /var/log/auth.log 2>/dev/null | tail -50
// advertisement

Automated Auditing with Lynis

Lynis is a hardening and compliance tool that automates most of the checks in this guide and adds hundreds more. Version 3.1.x added CIS Benchmark mappings and improved systemd unit file auditing. We run it on every server we onboard and use the hardening index as a baseline score.

Install directly from the Lynis GitHub releases rather than your distribution's package manager - the packaged version in Ubuntu 24.04 repos is 3.0.8, while the current release is 3.1.2, which includes fixes for false positives on systemd-resolved configurations.

# Install latest Lynis from source
cd /opt
git clone https://github.com/CISOfy/lynis.git
cd lynis

# Run full system audit
sudo ./lynis audit system --quiet 2>&1 | tee /tmp/lynis-$(hostname)-$(date +%F).txt

# Check the hardening index score
grep 'Hardening index' /tmp/lynis-*.txt

# Show only warnings and suggestions
grep -A1 'Warning\|Suggestion' /tmp/lynis-*.txt | head -60

# Run with specific profile (CIS)
sudo ./lynis audit system --profile ./default.prf

Network-Level Exposure: Audit from Outside the Host

An audit from inside the server only shows you what the kernel reports. A firewall misconfiguration or a NAT rule can expose ports that ss -tulpn does not show because the process is not on that host. Always scan from an external perspective.

Run an nmap scan from a separate machine on the same network segment. If you are auditing a server in a cloud environment or remote data center, you need a scan point in the same network tier - either a bastion host or a dedicated audit instance.

For servers accessible over the public internet, running your audit workstation behind NordVPN's Linux client (available at https://nordvpn.com/?ref=PLACEHOLDER) lets you verify what is visible from arbitrary geographic locations using their CLI without spinning up cloud instances. The nordvpn CLI supports country and city targeting with nordvpn connect --country and nordvpn connect --city flags, which is useful for testing geo-restriction rules.

# From external audit host - TCP SYN scan
sudo nmap -sS -p- -T4 --open 10.0.0.10 -oN /tmp/nmap-tcp-$(date +%F).txt

# UDP scan on common ports
sudo nmap -sU -p 53,67,68,69,123,161,500,1194,4500 10.0.0.10

# Service version detection and OS fingerprint
sudo nmap -sV -O -p 22,80,443,3306,5432,6379,27017 10.0.0.10

# Script scan for common vulnerabilities
sudo nmap --script vuln -p 80,443 10.0.0.10 2>/dev/null | grep -v '^#'

File Integrity and Rootkit Detection

AIDE (Advanced Intrusion Detection Environment) maintains a cryptographic database of file hashes and attributes. Running a check compares the current state against the baseline and flags any changes. This is how you detect a rootkit that replaced system binaries.

chkrootkit and rkhunter are complementary tools. They use signature-based detection for known rootkits and heuristic checks for suspicious patterns. Neither is comprehensive on its own, but both add signal. We run all three as part of a monthly scheduled audit, which you could automate through a workflow tool like taskbotshub.ai if you are managing multiple servers and want centralized scheduling with alerting.

Initialize the AIDE database on a clean system immediately after provisioning. A database initialized after a compromise is worthless.

# Install and initialize AIDE
sudo apt install aide aide-common -y   # Debian/Ubuntu
sudo dnf install aide -y               # RHEL

# Initialize database (do this on a clean system)
sudo aideinit
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Run integrity check
sudo aide --check 2>&1 | tee /tmp/aide-check-$(date +%F).txt

# rkhunter
sudo rkhunter --update
sudo rkhunter --check --skip-keypress 2>&1 | grep -E 'Warning|Found'

# chkrootkit
sudo chkrootkit 2>&1 | grep -v 'not infected' | grep -v '^$'
// advertisement

sudo Configuration and Privilege Review

Misconfigured sudo rules are one of the fastest paths to root on a server that is otherwise well-secured. The most dangerous patterns are NOPASSWD rules, rules that allow running shells or editors, and rules using wildcards in command paths.

Parse the effective sudoers configuration with visudo -c to check syntax first, then review the output of sudo -l for each user account that has sudo access.

# Validate sudoers syntax
sudo visudo -c

# Show full effective sudoers
sudo cat /etc/sudoers
ls -la /etc/sudoers.d/ && sudo cat /etc/sudoers.d/*

# Check sudo permissions for a specific user
sudo -l -U www-data
sudo -l -U deploy

# Find dangerous patterns: NOPASSWD, shells, editors
sudo grep -r 'NOPASSWD' /etc/sudoers /etc/sudoers.d/ 2>/dev/null
sudo grep -rE '\b(vi|vim|nano|less|more|bash|sh|zsh|python|perl|ruby|find|awk|sed)\b' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

Audit Reporting: What to Produce and Who Needs It

An audit that produces no artifact is an audit that cannot be verified, tracked, or compared over time. At minimum, produce a findings file sorted by severity, a list of remediation actions with owner assignments, and a before/after comparison if this is a re-audit.

For teams managing multiple servers, storing audit results with consistent naming makes grep and diff trivial. If you are naming your audit projects or documenting servers in a wiki, using a clean, descriptive naming convention matters more than people acknowledge - a tool like nicename.me can help generate readable project and host naming schemes that stay consistent across teams.

Generate a structured summary from the Lynis output for rapid review:

# Extract Lynis findings into a sorted report
sudo grep -E '\[WARNING\]|\[SUGGESTION\]' /var/log/lynis.log | \
  sed 's/^.*\[/[/' | sort | uniq > /tmp/lynis-findings.txt

# Count findings by category
echo "Warnings: $(grep -c WARNING /tmp/lynis-findings.txt)"
echo "Suggestions: $(grep -c SUGGESTION /tmp/lynis-findings.txt)"

# Generate diff between two audits
diff /tmp/lynis-$(hostname)-2026-01-01.txt /tmp/lynis-$(hostname)-2026-06-23.txt | \
  grep '^[<>]' | head -40