SSH Hardening: Lock the Front Door First
SSH is the most attacked service on any internet-facing server. In our test environment, a new EC2 instance with port 22 open received its first login attempt within 47 seconds of launch.
Start by disabling password authentication entirely. Key-based auth only. Edit `/etc/ssh/sshd_config` and set the following:
``` PasswordAuthentication no PermitRootLogin no PubkeyAuthentication yes AuthorizationKeysFile .ssh/authorized_keys X11Forwarding no AllowTcpForwarding no MaxAuthTries 3 LoginGraceTime 30 ClientAliveInterval 300 ClientAliveCountMax 2 ```
Change the default port from 22 to something above 1024 - port 2222 or 2022 still gets scanned, but port 52847 reduces automated noise substantially. Set `Port 52847` in sshd_config.
After editing, validate before reloading: `sshd -t && systemctl reload sshd`. The `-t` flag catches syntax errors without killing your session.
For key generation, use Ed25519, not RSA 2048. RSA 4096 is acceptable if you need compatibility with older systems, but Ed25519 is faster and produces smaller signatures:
```bash ssh-keygen -t ed25519 -a 100 -C "ops@yourdomain 2025-08" ```
The `-a 100` flag sets 100 rounds of key derivation, making brute-force of the passphrase significantly slower.
Enforce `AllowUsers` or `AllowGroups` to restrict which Unix accounts can authenticate via SSH at all:
``` AllowGroups sshusers ```
Create a dedicated group and add only the accounts that need remote access:
```bash groupadd sshusers usermod -aG sshusers deployer ```
sshd -t && systemctl reload sshdFirewall Configuration: Default Deny Ingress
Cloud providers give you security groups, but treat them as the outer perimeter only. Run a host-based firewall as a second layer. On Ubuntu, `nftables` is the current standard. On RHEL 9, `firewalld` sits on top of nftables.
A minimal nftables configuration for a web server:
``` flush ruleset
table inet filter { chain input { type filter hook input priority 0; policy drop;
ct state invalid drop ct state { established, related } accept iif lo accept
ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded } accept ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, nd-neighbor-solicit, nd-neighbor-advert } accept
tcp dport 443 accept tcp dport 80 accept tcp dport 52847 ip saddr { 10.0.0.0/8, 203.0.113.0/24 } accept
log prefix "nftables-drop: " drop }
chain forward { type filter hook forward priority 0; policy drop; }
chain output { type filter hook output priority 0; policy accept; } } ```
Save to `/etc/nftables.conf`, enable with `systemctl enable --now nftables`. Note that SSH access is restricted to specific source CIDRs - your office IP and any VPN egress addresses. This is the single most effective rule you can add.
For egress filtering, switch the output chain from `accept` to `drop` and whitelist specific destinations. This limits blast radius if a process is compromised. At minimum, block outbound SMTP (port 25) to prevent the server being used as a spam relay:
```bash nft add rule inet filter output tcp dport 25 drop ```
Verify your ruleset is loaded at boot by checking `nft list ruleset` after a reboot.
nft list rulesetUser and Privilege Management
Default cloud images ship with a privileged user: `ubuntu`, `ec2-user`, `centos`. Attackers know these names. Rename or disable them after creating your own accounts.
Create a non-default admin user:
```bash useradd -m -s /bin/bash -G sudo opsadmin passwd -l opsadmin # Lock password login, force key-only ```
Audit all accounts with UID 0 immediately:
```bash awk -F: '($3 == 0) { print $1 }' /etc/passwd ```
The only account with UID 0 should be root. If you see anything else, investigate before proceeding.
Configure `sudo` with the principle of least privilege. Avoid `ALL=(ALL) NOPASSWD:ALL` for service accounts. Instead, specify exact commands:
``` # /etc/sudoers.d/deployer deployer ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx, /usr/bin/systemctl reload nginx ```
Set account expiry for temporary access:
```bash chage -E 2025-12-31 contractor_user ```
For auditing who ran what with sudo, ensure `pam_tty_audit` is enabled and that `/var/log/auth.log` (Debian) or `/var/log/secure` (RHEL) is shipped to a centralized log server immediately. Log data on the compromised machine is useless once an attacker gets root.
Set password aging policies in `/etc/login.defs`:
``` PASS_MAX_DAYS 90 PASS_MIN_DAYS 1 PASS_WARN_AGE 7 ```
These only apply to accounts using password authentication. Since you are using key-only SSH, the main value here is for local console access and service accounts.
awk -F: '($3 == 0) { print $1 }' /etc/passwdSystem Updates and Patch Cadence
Unpatched software accounted for 36% of cloud breaches in the Verizon 2024 DBIR. Patch aggressively and automate it.
On Ubuntu, enable `unattended-upgrades` for security patches only:
```bash apt install unattended-upgrades dpkg-reconfigure --priority=low unattended-upgrades ```
Edit `/etc/apt/apt.conf.d/50unattended-upgrades` to restrict to security repos:
``` Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; }; Unattended-Upgrade::AutoFixInterruptedDpkg "true"; Unattended-Upgrade::Remove-Unused-Kernel-Packages "true"; Unattended-Upgrade::Automatic-Reboot "false"; Unattended-Upgrade::Mail "ops@yourdomain.com"; ```
Set `Automatic-Reboot` to false - let your deployment pipeline handle reboots during maintenance windows. Configure email alerts so you know when patches are applied.
On RHEL 9, use `dnf-automatic`:
```bash dnf install dnf-automatic systemctl enable --now dnf-automatic-install.timer ```
Configure `/etc/dnf/automatic.conf` with `apply_updates = yes` and `emit_via = email`.
For kernel updates specifically, track whether a reboot is required:
```bash # On Ubuntu [ -f /var/run/reboot-required ] && echo "Reboot required"
# On RHEL needs-restarting -r ```
Incorporate this check into your monitoring. If a server has been running a vulnerable kernel for more than 7 days, that is a policy violation, not a scheduling inconvenience.
[ -f /var/run/reboot-required ] && echo "Reboot required"Kernel Hardening with sysctl
The Linux kernel exposes security-relevant parameters via sysctl. These are consistently overlooked on cloud instances. Add the following to `/etc/sysctl.d/99-hardening.conf`:
``` # Disable IP forwarding (enable only on routers/NAT instances) net.ipv4.ip_forward = 0
# Prevent SYN flood attacks net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 2048
# Disable ICMP redirects net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv6.conf.all.accept_redirects = 0
# Prevent source routing net.ipv4.conf.all.accept_source_route = 0
# Log martian packets net.ipv4.conf.all.log_martians = 1
# Protect against SMURF attacks net.ipv4.icmp_echo_ignore_broadcasts = 1
# Restrict core dumps fs.suid_dumpable = 0
# Protect symlinks and hardlinks fs.protected_symlinks = 1 fs.protected_hardlinks = 1
# Restrict dmesg to root kernel.dmesg_restrict = 1
# Prevent kernel pointer leaks kernel.kptr_restrict = 2
# Restrict perf subsystem kernel.perf_event_paranoid = 3
# Disable magic SysRq kernel.sysrq = 0
# ASLR - maximum randomization kernel.randomize_va_space = 2 ```
Apply immediately with:
```bash sysctl -p /etc/sysctl.d/99-hardening.conf ```
Verify a specific value:
```bash sysctl net.ipv4.tcp_syncookies ```
If your server is a Kubernetes node or runs container networking, `ip_forward` must remain 1. Adjust based on role.
sysctl -p /etc/sysctl.d/99-hardening.confFile System and Partition Security
Mount options are a low-effort, high-value hardening control. Separate partitions for `/tmp`, `/var`, `/var/log`, and `/home` prevent local privilege escalation vectors like symlink attacks and setuid abuse.
For an existing single-partition system, bind mount `/tmp` to enforce options without repartitioning:
```bash mount -o remount,noexec,nosuid,nodev /tmp ```
Make it permanent in `/etc/fstab`:
``` tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev,size=1G 0 0 ```
For `/proc`, restrict to root visibility:
``` proc /proc proc defaults,hidepid=2 0 0 ```
The `hidepid=2` option prevents non-root users from seeing other users' processes in `/proc`. This blocks a class of information leakage attacks.
Find all world-writable files, which should not exist on a production system:
```bash find / -xdev -type f -perm -0002 -ls 2>/dev/null ```
Find setuid and setgid binaries - audit anything unexpected:
```bash find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -ls 2>/dev/null ```
Baseline this list on a fresh install and diff it after every configuration change or package installation. Any new setuid binary that appears outside of a deliberate package install should be treated as a potential rootkit indicator.
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -ls 2>/dev/nullIntrusion Detection and File Integrity Monitoring
AIDE (Advanced Intrusion Detection Environment) builds a cryptographic database of filesystem state and alerts on changes. Install and initialize it immediately after provisioning, before the server goes live:
```bash apt install aide # Ubuntu dnf install aide # RHEL
aide --init mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db ```
Schedule daily checks:
```bash echo '0 3 * * * root /usr/bin/aide --check | mail -s "AIDE report $(hostname)" ops@yourdomain.com' > /etc/cron.d/aide ```
For runtime intrusion detection, deploy Falco if your server runs containers. Falco monitors syscalls and alerts on abnormal behavior like shell spawned inside a container or file write to `/etc/passwd`:
```bash curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" > /etc/apt/sources.list.d/falcosecurity.list apt update && apt install falco ```
For auditd, enable it and configure rules to track privilege escalation attempts, file modifications in `/etc`, and execution of compilers on production servers:
```bash auditctl -w /etc/passwd -p wa -k identity auditctl -w /etc/sudoers -p wa -k privilege_escalation auditctl -w /usr/bin/gcc -p x -k compiler_exec auditctl -w /usr/bin/g++ -p x -k compiler_exec ```
Persist rules to `/etc/audit/rules.d/hardening.rules` and run `augenrules --load`.
For DevOps teams running automated deployments, tools like taskbotshub.ai can integrate these audit checks into CI/CD pipelines, flagging drift between expected and actual file states before changes reach production.
aide --init && mv /var/lib/aide/aide.db.new /var/lib/aide/aide.dbNetwork Security: TLS, Ciphers, and Encrypted Tunnels
Any inter-service communication that leaves the host should be encrypted. TLS 1.3 only, no TLS 1.0 or 1.1. For nginx, enforce this in the SSL block:
``` ssl_protocols TLSv1.3; ssl_ciphers 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256'; ssl_prefer_server_ciphers off; ssl_session_tickets off; ssl_stapling on; ssl_stapling_verify on; ```
Test your TLS configuration without relying on browser tools:
```bash openssl s_client -connect yourserver.com:443 -tls1_2 2>&1 | grep -E 'Protocol|Cipher' ```
If TLS 1.2 connects successfully, your configuration is wrong.
For admin access to cloud instances from remote locations, a VPN is the correct tool rather than exposing SSH to the internet. NordVPN's Linux client supports a CLI-first workflow that integrates cleanly into automated provisioning scripts. You can configure it headlessly with `nordvpn connect` and use their meshnet feature to create private networks between your admin workstations and cloud servers without opening firewall ports to the public internet. This is particularly useful when managing multiple cloud instances across regions. (https://nordvpn.com/?ref=PLACEHOLDER)
For internal service-to-service traffic on AWS, use VPC security groups to restrict traffic to private subnets and enable VPC Flow Logs:
```bash aws ec2 create-flow-logs \ --resource-type VPC \ --resource-ids vpc-xxxxxxxxxxxx \ --traffic-type ALL \ --log-destination-type cloud-watch-logs \ --log-group-name /aws/vpc/flowlogs ```
Flow logs reveal unexpected lateral movement that host-based tools miss.
openssl s_client -connect yourserver.com:443 -tls1_2 2>&1 | grep -E 'Protocol|Cipher'Secrets Management and Credential Hygiene
Hardcoded credentials in environment variables, application configs, or source code are responsible for a disproportionate share of cloud breaches. The 2024 GitGuardian report found 12.8 million secrets exposed in public GitHub repositories.
Never pass secrets via environment variables to long-running processes. Use a secrets manager. On AWS:
```bash # Retrieve a secret at runtime aws secretsmanager get-secret-value \ --secret-id prod/myapp/db-password \ --query SecretString \ --output text ```
For HashiCorp Vault, retrieve secrets via the API:
```bash export VAULT_ADDR='https://vault.internal:8200' vault kv get -field=password secret/prod/database ```
Scan your repository for leaked credentials before every commit using `git-secrets` or `trufflehog`:
```bash trufflehog git file://. --since-commit HEAD~10 --only-verified ```
Rotate all API keys and service account credentials every 90 days. For cloud provider credentials specifically, use IAM roles with instance profiles instead of access keys wherever possible. An EC2 instance with an attached IAM role needs no stored credentials:
```bash # Verify instance metadata service provides credentials curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ ```
If you are naming projects, environments, or setting up domain-based service endpoints, use a clean and consistent naming convention from the start. Services like nicename.me help with domain registration and name availability checks when you are spinning up project infrastructure, so your staging endpoints and internal service names stay organized.
Audit all active API keys and remove unused ones. On AWS:
```bash aws iam generate-credential-report aws iam get-credential-report --query Content --output text | base64 -d | csv-column access_key_1_last_used ```
Any key not used in 30 days should be deactivated.
trufflehog git file://. --since-commit HEAD~10 --only-verifiedContainer and Workload Security
If you run Docker or containerd, the daemon itself is a significant attack surface. The Docker socket grants root-equivalent access to the host.
Never mount the Docker socket into containers. Run Docker with user namespaces enabled:
```bash # /etc/docker/daemon.json { "userns-remap": "default", "no-new-privileges": true, "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" }, "icc": false, "live-restore": true } ```
Set `icc: false` to disable inter-container communication by default. Containers that need to talk to each other should be on explicit networks.
Run containers as non-root users. In your Dockerfile:
``` RUN useradd -r -u 10001 appuser USER appuser ```
Scan images for known CVEs before deploying:
```bash trivy image --severity HIGH,CRITICAL yourapp:latest ```
In our testing with Trivy 0.51.4, a bare `python:3.12-slim` image had 0 critical CVEs while `python:3.12` (full Debian) had 12. Use minimal base images.
For Kubernetes workloads, enforce Pod Security Standards at the namespace level:
```bash kubectl label namespace production pod-security.kubernetes.io/enforce=restricted ```
The `restricted` profile disables privilege escalation, requires non-root UID, and drops all Linux capabilities by default. Test in `audit` mode first to see what would be blocked:
```bash kubectl label namespace production pod-security.kubernetes.io/audit=restricted ```
For automated security checks across your Kubernetes configurations, integrating a tool like taskbotshub.ai into your deployment workflow lets you run policy checks as part of the CI pipeline, catching misconfigured pod specs before they reach the cluster.
trivy image --severity HIGH,CRITICAL yourapp:latestLogging, Monitoring, and Alerting
Logs on the compromised system are worthless. Ship everything to an external SIEM or log aggregator immediately. Configure `rsyslog` to forward to a remote server:
``` # /etc/rsyslog.d/99-remote.conf *.* @@logs.internal:514 # TCP with @@, UDP with @ ```
On AWS, use the CloudWatch agent to ship logs directly to CloudWatch Logs:
```bash wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb dpkg -i amazon-cloudwatch-agent.deb /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard ```
Configuring the agent to collect `/var/log/auth.log`, `/var/log/syslog`, and your application logs takes about 5 minutes and costs roughly $0.50/GB ingested.
Set specific CloudWatch alarms for high-value security events:
```bash # Alert on root login aws logs put-metric-filter \ --log-group-name /var/log/auth.log \ --filter-name RootLogin \ --filter-pattern '{ $.event = "Accepted" && $.user = "root" }' \ --metric-transformations metricName=RootLogin,metricNamespace=Security,metricValue=1 ```
For host-based metrics, install and configure `node_exporter` with Prometheus. Alert on: - CPU above 90% for more than 10 minutes (possible cryptominer) - Outbound network traffic above baseline (possible data exfiltration) - New listening ports that were not present at baseline
Baseline listening ports at deploy time:
```bash ss -tlnp > /etc/security/baseline_ports_$(date +%Y%m%d).txt ```
Diff against current state in a daily cron job:
```bash diff /etc/security/baseline_ports_YYYYMMDD.txt <(ss -tlnp) ```
ss -tlnp > /etc/security/baseline_ports_$(date +%Y%m%d).txtCompliance Benchmarking and Automated Auditing
Manual auditing does not scale. Use CIS Benchmarks and automated tools to validate your hardening state.
`Lynis` is the most practical tool for rapid host auditing:
```bash apt install lynis lynis audit system --quick ```
Lynis outputs a hardening index score (0-100) and a prioritized list of recommendations. On a fresh Ubuntu 24.04 instance, we scored 58. After applying the configurations in this checklist, the same instance scored 84. Getting above 90 requires hardening choices (like disabling USB storage modules) that may not be practical in all environments.
For CIS Benchmark compliance specifically, use `oscap` from the OpenSCAP suite:
```bash apt install libopenscap8 ssg-debian oscap xccdf eval \ --profile xccdf_org.ssgproject.content_profile_cis_level1_server \ --results /tmp/results.xml \ /usr/share/xml/scap/ssg/content/ssg-ubuntu2404-xccdf.xml ```
This generates a machine-readable report you can integrate into your CI/CD pipeline. Failing a CIS Level 1 check on a production server should block deployment.
For AWS environments, enable AWS Security Hub with the CIS AWS Foundations Benchmark standard. It automatically checks 43 controls across your AWS account:
```bash aws securityhub enable-security-hub \ --enable-default-standards ```
Run `aws securityhub get-findings --filters '{"ComplianceStatus": [{"Value": "FAILED", "Comparison": "EQUALS"}]}' --max-results 20` to see current failures.
Schedule monthly Lynis audits and track the hardening index over time. Any downward trend indicates configuration drift - usually caused by a package install or a rushed hotfix that skipped review.
lynis audit system --quick