Audit What You Have Before Changing Anything
The first mistake sysadmins make is hardening a server they have not fully inventoried. Run ss before touching firewall rules.
Check listening ports and the processes behind them. On a fresh VPS you should see sshd on 22 and maybe systemd-resolved on 127.0.0.53. On a server that has been running for six months, you will often find forgotten services nobody remembers enabling.
Also pull the OS version, kernel, and installed package list before you start. Knowing the baseline matters when you need to diff against a future audit.
ss -tlnp
systemctl list-units --type=service --state=running
uname -r
dpkg -l | grep -v '^rc' | wc -l # Debian/Ubuntu: count installed packages
rpm -qa | wc -l # RHEL/FedoraSSH Hardening: The Minimum Required Config
SSH is the entry point attackers hit first. A correctly configured sshd drops the attack surface to near zero. Make these changes in /etc/ssh/sshd_config and reload the daemon. Do not disconnect your current session before testing in a second terminal.
Disable password authentication entirely. If you lose your key, restore access through the console. That inconvenience is the point. Set PermitRootLogin to no and create a dedicated non-root user with sudo. Change the default port only if it meaningfully reduces log noise on your end - it does not stop a determined attacker, but it cuts automated scanning traffic by roughly 90% in our experience.
The AllowUsers directive is underused. Listing exactly which local usernames may authenticate via SSH is a hard whitelist that nothing bypasses.
# /etc/ssh/sshd_config
Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizationKeysFile .ssh/authorized_keys
AllowUsers deploy ansible
X11Forwarding no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
UseDNS no
# Reload
systemctl reload sshdGenerate and Deploy Ed25519 Keys
Stop generating RSA-4096 keys as if it is 2012. Ed25519 keys are shorter, faster, and offer equivalent security. Generate them on the client, never on the server.
Copy the public key to the server with ssh-copy-id or paste it manually into ~/.ssh/authorized_keys on the target user. Permissions on that file must be 600, and the .ssh directory must be 700 or sshd will silently ignore the key.
For teams managing credentials across multiple servers, 1Password CLI (op) integrates directly into SSH agent workflows. You can store SSH keys in 1Password vaults and use op run or the 1Password SSH agent so private keys never touch disk in plaintext. Their Linux client supports apt and rpm repos at https://1password.com/PLACEHOLDER.
# On your client machine
ssh-keygen -t ed25519 -C "deploy@myserver" -f ~/.ssh/id_ed25519_myserver
# Copy to server
ssh-copy-id -i ~/.ssh/id_ed25519_myserver.pub -p 2222 deploy@203.0.113.10
# Verify permissions on server
stat ~/.ssh && stat ~/.ssh/authorized_keysFirewall Rules with nftables
iptables is legacy. On any kernel 5.2 or newer, use nftables directly. ufw is acceptable for simple setups but hides too much for production. Write the ruleset explicitly.
The ruleset below drops everything inbound by default, accepts established connections, and opens only SSH on the custom port. Extend it per service. Commit the ruleset to your config management repo - Ansible, Salt, or Puppet - and treat manual firewall changes as a bug.
Check your active ruleset with nft list ruleset. If you see iptables rules coexisting with nftables rules, you have a conflict to resolve. On Debian 12 and Ubuntu 22.04+ nftables is the default backend for ufw, but the raw nft interface is cleaner for complex setups.
# /etc/nftables.conf
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
iif lo accept
ct state established,related accept
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
tcp dport 2222 accept
# tcp dport { 80, 443 } accept # uncomment for web
}
chain forward {
type filter hook forward priority 0; policy drop;
}
chain output {
type filter hook output priority 0; policy accept;
}
}
# Apply and persist
nft -f /etc/nftables.conf
systemctl enable nftablesAutomatic Security Updates Without Breaking Production
Unpatched packages are responsible for more breaches than misconfigured firewalls. The counterargument - that automatic updates break things - is valid but does not justify running unpatched servers. The solution is automatic security-only updates, not automatic dist-upgrades.
On Debian and Ubuntu, unattended-upgrades handles this. Configure it to apply only the security pocket, email on errors, and reboot only if required (kernel updates). Set the reboot time to a maintenance window.
On RHEL/Rocky/AlmaLinux, dnf-automatic with apply_updates = security is the equivalent. Check that the timer is active after installation.
# Debian/Ubuntu
apt install unattended-upgrades
dpkg-reconfigure --priority=low unattended-upgrades
# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";
Unattended-Upgrade::Mail "ops@example.com";
# RHEL family
dnf install dnf-automatic
# /etc/dnf/automatic.conf: apply_updates = security
systemctl enable --now dnf-automatic-install.timerDisable Root and Lock Down sudo
Lock the root account password. The account can still be used via su from a sudo-enabled user, which is what you want for emergencies. What you do not want is direct root login over SSH (already disabled above) or root password-based escalation.
Create a specific sudo policy. Never give users NOPASSWD across all commands in production. If automation requires passwordless sudo, scope it to exactly the commands needed. Ansible playbooks, for example, often need only specific systemctl and apt commands.
Audit your sudoers file regularly. Developers added to wheel or sudo groups during debugging are a common persistence vector.
# Lock root password (disables password auth for root)
passwd -l root
# Create a user with scoped sudo
useradd -m -s /bin/bash deploy
usermod -aG sudo deploy
# Scoped NOPASSWD example in /etc/sudoers.d/deploy
# deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx, /usr/bin/apt-get update
# Audit current sudo permissions
grep -r '' /etc/sudoers /etc/sudoers.d/
getent group sudo wheelFail2ban Configuration for SSH and Web Services
Fail2ban reads log files and bans IPs that exceed failed authentication thresholds. Install it, configure a jail for sshd, and set a meaningful bantime. The default 10-minute ban is too short. We use 24 hours for SSH on public-facing servers, with a findtime of 10 minutes and maxretry of 3.
If you changed the SSH port, tell fail2ban. The sshd jail needs to know the actual port. Create overrides in /etc/fail2ban/jail.local, never edit jail.conf directly.
For web servers, enable the nginx-http-auth and nginx-limit-req jails. Check jail status with fail2ban-client to confirm it is reading the right log files.
# /etc/fail2ban/jail.local
[DEFAULT]
bantime = 86400
findtime = 600
maxretry = 3
backend = systemd
[sshd]
enabled = true
port = 2222
logpath = %(sshd_log)s
[nginx-http-auth]
enabled = true
# Check status
fail2ban-client status
fail2ban-client status sshdKernel Hardening via sysctl
The default kernel parameters are tuned for compatibility, not security. Several sysctl values meaningfully reduce attack surface with zero performance cost.
Disable IP forwarding unless this is a router. Harden against SYN flood attacks with SYN cookies. Disable ICMP redirects - they serve legitimate network purposes in some environments but are more commonly used in MITM attacks. Prevent processes from seeing other users' processes via hidepid on /proc.
Write these to /etc/sysctl.d/99-hardening.conf and run sysctl --system to apply without reboot. Verify each setting took effect.
# /etc/sysctl.d/99-hardening.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv6.conf.all.accept_redirects = 0
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
# Apply
sysctl --system
# Verify
sysctl net.ipv4.ip_forwardFile Integrity Monitoring with AIDE
AIDE (Advanced Intrusion Detection Environment) takes a snapshot of filesystem checksums and reports changes. It will not stop an attack, but it will tell you what changed and when, which is critical for incident response.
Initialize the database after completing your hardening steps, not before. Any subsequent run compares against that baseline. Run AIDE checks via cron and email the diff. A daily check is sufficient for most servers.
Focus AIDE on high-value paths: /bin, /sbin, /usr/bin, /usr/sbin, /etc, /boot. Exclude volatile paths like /var/log, /tmp, and /proc to avoid false positives flooding your reports.
apt install aide
# /etc/aide/aide.conf additions
/bin NORMAL
/sbin NORMAL
/usr/bin NORMAL
/usr/sbin NORMAL
/etc NORMAL
/boot NORMAL
!/var/log
!/tmp
# Initialize baseline (do this after full hardening)
aide --init
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# Check (run from cron)
aide --check
# Cron: daily at 2am
# 0 2 * * * root /usr/bin/aide --check | mail -s "AIDE report $(hostname)" ops@example.comAudit Logging with auditd
auditd writes kernel-level audit events to /var/log/audit/audit.log. Default rules capture little of value. Add rules for actions that matter: privilege escalation attempts, changes to /etc/passwd and /etc/shadow, execution of su and sudo, and changes to cron jobs.
ausearch and aureport are your primary tools for reading the log. aureport --summary gives a quick overview. Use ausearch -k to filter by rule key.
Send audit logs to a remote syslog server. A local audit log that an attacker can modify after a breach has limited forensic value. Configure audisp-remote or use rsyslog with TLS to ship logs off-box immediately.
apt install auditd audispd-plugins
# /etc/audit/rules.d/hardening.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands
-w /var/spool/cron -p wa -k cron
-w /etc/cron.d -p wa -k cron
# Load rules
augenrules --load
systemctl restart auditd
# Query
ausearch -k identity --start today
ausearch -k sudoers -iSecrets Management: No Plaintext Credentials on Disk
Finding plaintext passwords in .env files, bash history, or config files is the single most common finding in post-breach reviews. Fix this before it becomes relevant.
For teams, 1Password CLI (op) handles secrets injection at runtime. You reference a secret by vault path and it is substituted into the environment or file at execution time. Nothing hits disk. The op inject command reads a template file and replaces references on the fly.
For server-side secrets that applications need at runtime, use HashiCorp Vault or systemd credentials (systemd 247+). The systemd-creds tool encrypts credentials with the machine's TPM or a key derived from /etc/machine-id, making them useless if the disk is removed. Clear bash history of any commands containing passwords immediately.
# 1Password CLI: inject secrets into environment
op inject -i .env.tpl -o .env
# .env.tpl example
# DB_PASSWORD={{ op://Production/postgres/password }}
# systemd credentials (systemd 247+)
systemd-creds encrypt --name=db-password - /etc/credstore/db-password
# Reference in unit file:
# LoadCredential=db-password:/etc/credstore/db-password
# Nuke bash history entries with passwords
history | grep -i password
history -d VPN for Administrative Access
If you are managing servers in multiple datacenters or across cloud providers, routing admin traffic through a VPN before it hits the server cuts exposure significantly. The SSH port does not need to be public if all admins connect via VPN first.
NordVPN has a native Linux CLI client with full support on Debian, Ubuntu, and RHEL-based distros. It installs via their apt or yum repository and the nordvpn CLI manages connections, killswitch, and split tunneling. For infrastructure teams that need a quick, audited VPN tunnel without running their own WireGuard endpoint, it is a practical option. Find Linux setup docs and the CLI reference at https://nordvpn.com/?ref=PLACEHOLDER.
If you run your own WireGuard setup, the same principle applies: restrict SSH to the WireGuard interface IP range only in your nftables rules and remove the public SSH rule entirely.
# WireGuard: restrict SSH to VPN interface only
# Replace wg0 with your interface name
# In nftables, source-restrict the SSH rule:
# iifname "wg0" tcp dport 2222 accept
# NordVPN CLI quickstart
apt install nordvpn
nordvpn login
nordvpn set killswitch on
nordvpn connectLynis: Full Security Audit in 60 Seconds
Run Lynis on every new server before putting it into service. It audits 300+ controls across SSH config, filesystem permissions, running services, kernel parameters, and more, then scores the system and lists specific remediation items.
Install from the upstream repository or your distro's package manager. The upstream version is more current. Run as root for full output. Pipe to a file and track the hardening index score over time - you want to see it increase, 80+ is realistic for a well-maintained server.
For automated security scanning in CI/CD pipelines, DevOps teams are integrating tools like Lynis and Trivy into workflow automation platforms. taskbotshub.ai includes pre-built automation bots for security scanning pipelines that can trigger Lynis audits post-deploy and route findings to Slack or a ticket system.
# Install upstream version
apt install lynis
# Full system audit
lynis audit system
# Score and suggestions
grep 'Hardening index' /var/log/lynis.log
grep 'Suggestion' /var/log/lynis-report.dat | wc -l
# Non-interactive for CI
lynis audit system --no-colors --quiet 2>/dev/null | tee /tmp/lynis-$(date +%F).txtDisable Unnecessary Services and Remove Orphan Packages
Every running service is a potential vulnerability. On a dedicated web server, you do not need avahi-daemon, cups, rpcbind, or bluetooth. Disable and mask them so they cannot be started accidentally.
Removing unused packages reduces the update surface and eliminates software that could be exploited even if not running. On Debian/Ubuntu, deborphan finds packages with no reverse dependencies. Run it after removing known-unnecessary packages to catch anything that was only installed as a dependency.
Check for SUID and SGID binaries. Any world-executable SUID binary that is not required is a privilege escalation risk. Know what is on your system.
# Disable and mask unnecessary services
for svc in avahi-daemon cups rpcbind bluetooth; do
systemctl stop $svc 2>/dev/null
systemctl disable $svc 2>/dev/null
systemctl mask $svc
done
# Find orphan packages (Debian/Ubuntu)
apt install deborphan
deborphan
# Find SUID/SGID binaries
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -exec ls -la {} \; 2>/dev/null
# Compare against expected list
find / -xdev -perm -4000 -type f 2>/dev/null | sort > /root/suid-baseline.txtTLS, Certificate Management, and Protocol Hardening
For any service exposing HTTPS, configure TLS 1.2 minimum, TLS 1.3 preferred, and disable weak cipher suites. Use Mozilla's SSL Configuration Generator to produce correct nginx or Apache configs for your server version - it outputs production-ready blocks.
Automate certificate renewal with certbot and systemd timers. The certbot.timer unit from the certbot package handles this on Debian/Ubuntu. Verify the timer is active and do a dry-run to confirm renewal works before the certificate expires.
Test your TLS configuration with testssl.sh locally before exposing a service. It runs comprehensive checks including BEAST, POODLE, ROBOT, and Heartbleed from your own machine.
# Certbot with systemd timer
apt install certbot python3-certbot-nginx
certbot --nginx -d example.com
systemctl status certbot.timer
# Dry run to verify renewal
certbot renew --dry-run
# testssl.sh local test
git clone --depth 1 https://github.com/drwetter/testssl.sh.git
cd testssl.sh
./testssl.sh https://example.com
# nginx TLS hardening
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:...;
# ssl_prefer_server_ciphers off;Process Isolation: AppArmor and namespaces
AppArmor is enabled by default on Ubuntu and Debian. Confirm it is active and that profiles are loaded for high-risk services like nginx, mysql, and sshd. aa-status shows loaded profiles and their enforcement mode. Move profiles from complain to enforce once you have confirmed they do not break functionality.
For containerized workloads, run containers without --privileged, drop capabilities explicitly, and use read-only root filesystems where the application allows it. Docker defaults grant more capabilities than most applications need. A minimal capability set for a web process is CAP_NET_BIND_SERVICE and nothing else.
On servers where AppArmor is absent (some RHEL-based distros default to SELinux), ensure SELinux is set to enforcing, not permissive. getenforce returns the current mode. Never set it to disabled without a documented exception.
# AppArmor status
aa-status
# Set profile to enforce
aa-enforce /etc/apparmor.d/usr.sbin.nginx
# SELinux status
getenforce
setenforce 1 # temporary
# Permanent: SELINUX=enforcing in /etc/selinux/config
# Docker: drop capabilities
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE \
--read-only \
--security-opt no-new-privileges \
nginx:alpine