Why Key-Only SSH Is No Longer Enough

A stolen laptop or a compromised secrets manager exposes your private key. Without a second factor, that key gives an attacker direct root-escalation paths through any SUID binary or misconfigured sudoers entry. The 2023 CircleCI breach and the 2024 Cloudflare Okta incident both demonstrated that credential theft happens at scale and faster than most teams rotate keys.

The most practical second factor for Linux servers is TOTP (Time-based One-Time Password) via RFC 6238. It requires no network call from the server, works offline, and the libpam-google-authenticator module is maintained, audited, and ships in every major distribution's default repos. Hardware tokens like YubiKey are better for the highest-security environments, but TOTP covers 95% of production needs with far less operational overhead.

Before touching PAM or SSH config, confirm your fallback access path. On cloud instances, that means your provider's serial console (AWS EC2 Instance Connect, GCP Cloud Shell, Azure Serial Console). On bare metal, it means IPMI or iDRAC. Test that path now. We have seen sysadmins lock themselves out of 40-node clusters because they assumed out-of-band access worked without verifying it.

# Verify your out-of-band access works before any PAM changes
# On AWS:
aws ec2-instance-connect send-ssh-public-key \
  --instance-id i-0abc123def456 \
  --instance-os-user ubuntu \
  --ssh-public-key file://~/.ssh/id_ed25519.pub

# Then from a second terminal, confirm you can SSH in normally
ssh -i ~/.ssh/id_ed25519 ubuntu@

Installing libpam-google-authenticator

On Debian/Ubuntu, the package is in main. On RHEL/Rocky/AlmaLinux, it comes from EPEL. Install it on each server you intend to protect - not just a jump host.

After installation, each user who will authenticate must run `google-authenticator` interactively as themselves, not as root. This generates a per-user secret stored in `~/.google_authenticator`. If you run it as root, the secret lands in root's home directory and non-root users get authentication failures.

The interactive prompts matter. Answer 'y' to time-based tokens, 'y' to updating the `.google_authenticator` file, 'y' to disallowing multiple uses of the same token (prevents replay attacks), 'n' to the 30-second window increase unless your servers have significant clock skew, and 'y' to rate limiting (3 attempts per 30 seconds). Save the emergency scratch codes somewhere offline - a password manager entry or a printed sheet in a safe works. We store ours in Bitwarden with the entry tagged 'emergency-2fa'.

# Debian / Ubuntu 24.04
apt install libpam-google-authenticator -y

# RHEL 9 / Rocky 9 / AlmaLinux 9
dnf install epel-release -y
dnf install google-authenticator -y

# Run as the user who will log in (not root)
google-authenticator

# Non-interactive for automation testing only (dev environments)
google-authenticator -t -d -f -r 3 -R 30 -w 3

Configuring PAM for SSH

PAM configuration is where most mistakes happen. The order of lines and the control flags (required, requisite, sufficient, optional) determine whether a failed 2FA check blocks login or silently passes. Get this wrong and you either lock everyone out or bypass 2FA entirely.

On Ubuntu 24.04, edit `/etc/pam.d/sshd`. Add the google-authenticator line after the `@include common-auth` line. The `nullok` flag allows users who have not yet enrolled to log in - remove it once all users have enrolled. The `secret` option lets you move the secret file out of the home directory, which matters if home directories are NFS-mounted.

On RHEL 9, the file is also `/etc/pam.d/sshd` but the existing content differs. The `auth substack password-auth` line is your equivalent of `@include common-auth`. Insert the google-authenticator line immediately after it.

The `[success=done default=ignore]` control flag is worth understanding: if google-authenticator succeeds, PAM skips the rest of the auth stack for that module type. This is the correct flag when you want TOTP as a hard requirement combined with SSH keys. Do not use `optional` in production - it degrades to single-factor if the module fails to load.

# /etc/pam.d/sshd - Ubuntu 24.04
# Add this line AFTER @include common-auth
auth required pam_google_authenticator.so

# For NFS home directories, redirect the secret file:
auth required pam_google_authenticator.so secret=/etc/google-authenticator/${USER}/.google_authenticator

# Full recommended block for Ubuntu:
@include common-auth
auth required pam_google_authenticator.so nullok

# RHEL 9 - insert after 'auth substack password-auth'
auth substack password-auth
auth required pam_google_authenticator.so
// advertisement

Configuring sshd to Require Both Key and TOTP

PAM configuration alone is not enough. You must also update `/etc/ssh/sshd_config` to tell OpenSSH to use PAM for authentication and to require multiple authentication methods. Without `AuthenticationMethods`, SSH accepts a valid key and never invokes PAM's keyboard-interactive challenge.

`ChallengeResponseAuthentication` was renamed to `KbdInteractiveAuthentication` in OpenSSH 8.7. Ubuntu 24.04 ships OpenSSH 9.6p1 and RHEL 9 ships 8.7p1, so both use the new name. If you have older servers on Ubuntu 22.04 (OpenSSH 8.9p1), both names work but the old name generates a deprecation warning in the logs.

After editing sshd_config, always validate the syntax before reloading. A syntax error in sshd_config with an active reload kills new connections but leaves existing sessions alive - you have a window to fix it. A full restart with a syntax error on some init systems drops existing sessions too.

Test the new configuration from a second terminal before closing your current session. The expected flow: SSH presents your key, the server accepts it, then prompts 'Verification code:' before granting access.

# /etc/ssh/sshd_config additions
UsePAM yes
KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

# Validate before reload
sshd -t

# Reload (not restart) to preserve existing sessions
systemctl reload sshd

# Test from a second terminal
ssh -v user@yourserver
# You should see:
# debug1: Authentications that can continue: publickey
# debug1: Next authentication method: keyboard-interactive
# Verification code:

Extending 2FA to sudo

Requiring 2FA at SSH login is good. Requiring it again at sudo elevation is better, because it catches an attacker who compromises a low-privilege session (via a web shell, for example) without having gone through SSH at all.

Edit `/etc/pam.d/sudo` and add the google-authenticator line. The `forward_pass` option passes the password+token combination through so PAM can check both without prompting twice. Without `forward_pass`, sudo prompts for the system password first, then immediately prompts for the TOTP code as a separate challenge - which works but confuses users.

One important caveat: the `NOPASSWD` directive in sudoers bypasses PAM entirely. If you have `NOPASSWD: ALL` for your deploy user, adding google-authenticator to `/etc/pam.d/sudo` does nothing for that user. Audit your sudoers with `sudo -l -U username` for every privileged user.

# /etc/pam.d/sudo - add before @include common-auth
auth required pam_google_authenticator.so
@include common-auth

# Audit NOPASSWD entries across all sudoers files
grep -r 'NOPASSWD' /etc/sudoers /etc/sudoers.d/

# Check what a specific user can run
sudo -l -U deploy

Handling Service Accounts and CI/CD Pipelines

Service accounts used by CI/CD pipelines cannot complete an interactive TOTP challenge. The correct approach is to exempt these accounts from 2FA at the PAM level, not at the SSH level, and compensate with stricter controls elsewhere.

PAM supports per-user exemptions via the `nullok` flag combined with a missing `.google_authenticator` file, but that is fragile - if someone accidentally runs `google-authenticator` as the service account, 2FA activates and breaks the pipeline. A more robust method uses a PAM access control file or a dedicated sshd instance on a non-standard port for machine-to-machine connections.

The dedicated sshd instance approach is what we use in production. Run a second sshd on port 2222 with a separate config file that omits the `AuthenticationMethods publickey,keyboard-interactive` line. Firewall that port to known CI/CD source IPs only. This keeps human and machine authentication paths completely separate.

For teams running automated deployment workflows, tools like taskbotshub.ai can manage SSH key rotation and audit CI/CD authentication events across fleets, reducing the manual overhead of keeping service account keys current without weakening the authentication policy.

If a dedicated sshd instance is too complex for your setup, use the `Match User` block in sshd_config to apply different authentication requirements per user.

# Option 1: Match block in sshd_config for service accounts
Match User deploy,jenkins,gitlab-runner
    AuthenticationMethods publickey
    AllowUsers deploy jenkins gitlab-runner

# Option 2: Dedicated sshd instance for CI/CD
cp /etc/ssh/sshd_config /etc/ssh/sshd_config_cicd
# Edit sshd_config_cicd:
#   Port 2222
#   AuthenticationMethods publickey
#   AllowUsers deploy jenkins

/usr/sbin/sshd -f /etc/ssh/sshd_config_cicd

# Lock port 2222 to specific IPs
ufw allow from 10.0.1.50 to any port 2222
ufw allow from 10.0.1.51 to any port 2222
ufw deny 2222
// advertisement

NFS Home Directories and Secret File Management

If home directories are NFS-mounted, the `.google_authenticator` file lives on a network share. This has two problems: a brief NFS outage blocks all logins, and the secrets are accessible to anything with NFS mount access. The fix is to move secrets to local storage on each server.

Create a local directory with strict permissions and use the `secret` option in the PAM configuration to point google-authenticator at the local path. The directory must be owned by root and contain per-user subdirectories owned by each user with mode 0700.

For large fleets, the enrollment step - running `google-authenticator` per user per server - becomes a problem. We solve this by generating the secret server-side with a script and distributing it via an Ansible playbook. The TOTP secret is a base32-encoded string; you can generate it, create the `.google_authenticator` file programmatically, and then display the QR code URL for the user to scan. The file format is simple: first line is the base32 secret, followed by options like `" TOTP_AUTH"`, `" DISALLOW_REUSE"`, and `" RATE_LIMIT 3 30 "` with their associated data.

# Local secret directory setup
mkdir -p /etc/google-authenticator
chmod 755 /etc/google-authenticator

# Per-user subdirectory (run in a loop for all users)
username=alice
mkdir -p /etc/google-authenticator/${username}
chown ${username}:${username} /etc/google-authenticator/${username}
chmod 700 /etc/google-authenticator/${username}

# /etc/pam.d/sshd with local secret path
auth required pam_google_authenticator.so \
  secret=/etc/google-authenticator/${USER}/.google_authenticator \
  user=root

# The google_authenticator file format
# Line 1: base32 secret (no padding)
# Line 2: " TOTP_AUTH"
# Line 3: " DISALLOW_REUSE"
# Line 4: " RATE_LIMIT 3 30 [timestamps]"
# Lines 5+: emergency scratch codes (8-digit numbers, one per line)

# Generate a secret programmatically (requires python3-pyotp)
python3 -c "import pyotp; print(pyotp.random_base32())"

Ansible Playbook for Fleet Enrollment

Rolling 2FA out to 50 servers manually is not realistic. The following Ansible approach generates a unique TOTP secret per user per host, writes the google_authenticator file with correct ownership and permissions, updates PAM, and updates sshd_config - all idempotent.

One non-obvious detail: the `google_authenticator` file must be owned by the user it belongs to and have mode 0600. If root owns it, PAM silently fails and the user cannot log in. The Ansible `file` module handles this, but confirm with `ls -la ~/.google_authenticator` after the play runs.

Store the generated secrets in Ansible Vault, not in plain-text group_vars. After the play runs, the secrets need to reach users securely - we email QR code provisioning URIs via an encrypted channel or display them through your internal portal. The provisioning URI format is `otpauth://totp/Label?secret=BASE32SECRET&issuer=YourOrg`, which any TOTP app (Google Authenticator, Authy, 1Password, Bitwarden) can scan.

# tasks/main.yml excerpt
- name: Install libpam-google-authenticator
  package:
    name: libpam-google-authenticator
    state: present

- name: Generate TOTP secret for user
  command: python3 -c "import pyotp; print(pyotp.random_base32())"
  register: totp_secret
  changed_when: false
  no_log: true

- name: Write google_authenticator file
  copy:
    content: |
      {{ totp_secret.stdout }}
      " TOTP_AUTH"
      " DISALLOW_REUSE"
      " RATE_LIMIT 3 30"
    dest: "/home/{{ target_user }}/.google_authenticator"
    owner: "{{ target_user }}"
    group: "{{ target_user }}"
    mode: '0600'
  no_log: true

- name: Configure sshd for 2FA
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: '^AuthenticationMethods'
    line: 'AuthenticationMethods publickey,keyboard-interactive'
    state: present
  notify: reload sshd

- name: Validate sshd config
  command: sshd -t
  changed_when: false

Auditing and Monitoring 2FA Events

Authentication events go to `/var/log/auth.log` on Debian-based systems and `/var/log/secure` on RHEL-based systems. Successful TOTP authentication logs as `Accepted keyboard-interactive/pam for user from IP port PORT ssh2`. Failed TOTP appears as `Failed keyboard-interactive/pam for user from IP`.

Set up a log alert for repeated TOTP failures from a single IP - this pattern indicates an attacker who has obtained a valid key and is attempting to brute-force the TOTP window. The rate limiting in google-authenticator (3 attempts per 30 seconds) provides local protection, but fail2ban adds IP-level blocking and alerting.

The fail2ban filter for SSH already catches many patterns. For TOTP-specific failures, add a custom filter targeting the PAM keyboard-interactive failure string. Set `maxretry = 5` and `bantime = 3600` - stricter than the default SSH filter because an attacker who has your key is a higher-severity event than a generic brute-force attempt.

For compliance environments (SOC 2, PCI-DSS, ISO 27001), you need tamper-evident logs shipped off the server in real time. Ship auth logs to a centralized SIEM - Graylog, Elasticsearch with Filebeat, or Loki with Promtail all work. The key requirement is that the log destination is write-only from the server's perspective.

# Watch auth events in real time
tail -f /var/log/auth.log | grep -E '(keyboard-interactive|google_authenticator)'

# Count TOTP failures per IP in the last hour
grep 'Failed keyboard-interactive' /var/log/auth.log \
  | awk '{print $11}' \
  | sort | uniq -c | sort -rn | head -20

# /etc/fail2ban/filter.d/sshd-totp.conf
[Definition]
failregex = Failed keyboard-interactive/pam for .* from 
ignoreregex =

# /etc/fail2ban/jail.d/sshd-totp.conf
[sshd-totp]
enabled = true
filter = sshd-totp
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600
findtime = 300

fail2ban-client reload
fail2ban-client status sshd-totp
// advertisement

Clock Synchronization - The Silent Killer

TOTP codes are valid for 30 seconds and generated from the current Unix timestamp. If your server clock drifts more than 30 seconds from your phone's clock, every code will be rejected and users will be locked out with no obvious error message. The auth log shows `Failed keyboard-interactive` which looks identical to a wrong code.

Verify NTP is running and synchronized on every server in scope. On systemd-based systems, `timedatectl` shows the sync status. The `NTP synchronized: yes` line is what you need. On servers that have been running for years without attention, we have found clock drift exceeding 5 minutes - enough to make TOTP completely non-functional.

For high-security environments, use a local NTP server synced to multiple upstream stratum-1 sources rather than relying on internet NTP pools. This avoids the scenario where an attacker manipulates internet NTP responses to cause clock drift (a theoretical but documented attack vector). The `iburst` option in ntpd/chrony configuration speeds up initial synchronization after a restart.

# Check time sync status
timedatectl status
# Look for: NTP synchronized: yes

# Install and enable chrony (preferred over ntpd on modern systems)
apt install chrony -y  # Debian/Ubuntu
dnf install chrony -y  # RHEL

systemctl enable --now chronyd

# /etc/chrony.conf - recommended configuration
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst
makestep 1.0 3
rtcsync

# Check current sync status and offset
chronyc tracking
chronyc sources -v

# Force sync if drift is significant
chronyc makestep

Emergency Recovery Procedures

Document your recovery procedure before you need it. When a sysadmin loses their phone at 2am during an incident, the worst time to figure out recovery is at that moment.

Google Authenticator generates 5 emergency scratch codes during initial enrollment. Each code is single-use. Store them in your password manager and in a physical location. When a user uses a scratch code, verify with `grep` that the file updated - used codes are removed from the `.google_authenticator` file automatically.

For full lockout scenarios - lost phone, no scratch codes, no out-of-band access - you need a documented process that does not itself become an attack vector. A common pattern: require a video call with two team members present, verify identity via a pre-shared challenge question stored in your secrets manager, then have an admin temporarily add a `nullok` entry to PAM so the user can log in with key only and re-enroll.

For users who change phones frequently (common with BYOD policies), consider using Authy or 1Password's TOTP implementation instead of Google Authenticator. Both support encrypted cloud backup of TOTP secrets, which means a new phone does not require re-enrollment. The security tradeoff is that the TOTP secret now exists in a cloud service, but for most threat models that is acceptable.

If you are managing server names or project identifiers as part of a larger infrastructure build-out, clean naming conventions matter for audit trails - tools like nicename.me can help generate sensible, consistent hostnames and project slugs that make auth logs readable at a glance.

# Check remaining scratch codes for a user
cat /home/alice/.google_authenticator
# Scratch codes appear as 8-digit numbers after the options lines

# Temporarily disable 2FA for a specific user during emergency
# Option 1: Add nullok to PAM (affects all users without .google_authenticator)
# Option 2: Rename their secret file
mv /home/alice/.google_authenticator /home/alice/.google_authenticator.bak

# After re-enrollment, restore or replace
# DO NOT forget to re-enable - set a reminder immediately

# Re-generate scratch codes without changing the TOTP secret
# Run as the affected user
google-authenticator
# Answer 'n' to generate new key
# Answer 'y' to update file - this regenerates scratch codes

Security Hardening Beyond Basic 2FA

Two-factor auth is one layer. Several additional controls significantly improve the overall posture without adding operational complexity.

Limit which users can authenticate via SSH using `AllowUsers` or `AllowGroups` in sshd_config. This prevents any newly created service account or compromised system user from being a login target. We use `AllowGroups ssh-users` and manage group membership via Ansible, so new users do not get SSH access by default.

Disable `PermitRootLogin` completely. Set it to `no`, not `prohibit-password`. Root logins should go through a regular user account and then `sudo su -` or `sudo -i`, which creates an audit trail linking the TOTP-authenticated user to the root session. With `prohibit-password`, a root TOTP bypass via a misconfigured PAM module is still theoretically possible.

Set `LoginGraceTime 30` to reduce the window for authentication. The default 120 seconds gives attackers more time to probe. With key plus TOTP, 30 seconds is sufficient for any human user.

Consider `MaxAuthTries 4` - enough for two failed TOTP entries plus one success, covering a mistyped code. Combined with fail2ban, this provides defense in depth against code guessing without frustrating legitimate users who mistype a digit.

# /etc/ssh/sshd_config - hardened configuration
Port 22
AddressFamily inet
ListenAddress 0.0.0.0

PermitRootLogin no
MaxAuthTries 4
LoginGraceTime 30
MaxSessions 5

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no
PermitEmptyPasswords no
KbdInteractiveAuthentication yes
UsePAM yes
AuthenticationMethods publickey,keyboard-interactive

AllowGroups ssh-users

X11Forwarding no
PrintMotd no
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server

# Add user to ssh-users group
usermod -aG ssh-users alice

# Verify the group exists
getent group ssh-users
// advertisement