Understanding the CVE Lifecycle and Your Exposure Window
The National Vulnerability Database publishes CVEs with a CVSS score, affected package ranges, and sometimes a working PoC reference within days of disclosure. Your exposure window is the gap between NVD publication and the moment you deploy a patched package. On average, that window stretches 12 days for enterprise Linux deployments according to Qualys telemetry from 2025 - long enough for mass exploitation of high-profile vulnerabilities.
Each major distro maintains its own security advisory feed. Red Hat uses RHSA advisories, Debian publishes DSAs, and Ubuntu uses USNs. These feeds lag NVD by hours to days while the distro packages and tests the fix. Subscribing directly to the distro feed rather than polling NVD avoids chasing upstream CVEs that do not affect your specific package builds.
CVSS scores alone are a poor triage tool. A CVSS 9.8 vulnerability in a library you do not expose to the network is less urgent than a CVSS 7.5 vulnerability in your public-facing nginx build. Always cross-reference the attack vector (AV:N vs AV:L), privileges required, and whether a working exploit is publicly available. The CISA Known Exploited Vulnerabilities catalog is a faster signal than CVSS for prioritization - if a CVE appears there, patch it within 24 hours regardless of score.
# Subscribe to Ubuntu security notices via RSS or email
# For command-line monitoring, use unattended-upgrades with security-only origin
grep -r 'Unattended-Upgrade::Allowed-Origins' /etc/apt/apt.conf.d/50unattended-upgradesScanning Your Systems for Vulnerable Packages
Before you patch anything, you need an accurate inventory of what is vulnerable. Three tools cover most use cases: `trivy`, `grype`, and the distro-native scanners.
`trivy` from Aqua Security scans installed packages, container images, and filesystem paths against multiple vulnerability databases including NVD, GitHub Advisory, and distro-specific advisories. It is the most practical choice for mixed environments. Install the 0.52+ release to get EPSS score support, which gives you exploit prediction probability alongside CVSS.
`grype` from Anchore is a strong alternative with SBOM input support - useful if you already generate SBOMs in your CI pipeline. For RHEL/CentOS environments, `openscap-scanner` with the SCAP Security Guide content gives you CVE scanning tied directly to CIS and DISA STIG benchmarks, which matters for compliance-driven environments.
For a quick scan of a running Debian/Ubuntu host, `debsecan` queries the Debian security tracker directly and reports which installed packages have open CVEs, including severity and whether a fix is available. It is lightweight and needs no daemon.
# Install trivy on Ubuntu 24.04
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.52.2
# Scan the local filesystem for OS package vulnerabilities
trivy fs --scanners vuln --severity HIGH,CRITICAL /
# Scan with EPSS scores enabled
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL --format json --output /var/reports/trivy-$(date +%F).json /
# debsecan on Debian/Ubuntu
apt-get install -y debsecan
debsecan --suite noble --format detail --only-fixedTriage: Separating Critical Patches from Noise
A production server running Ubuntu 24.04 LTS will typically show 30-80 open CVEs at any given time after a fresh install. Most of these are low or medium severity with no public exploit. Patching all of them immediately creates unnecessary change risk. A structured triage process gets you to a defensible patch schedule.
Tier 1 - patch within 24 hours: Any CVE on the CISA KEV catalog, any CVSS 9.0+ with AV:N and no authentication required, any vulnerability in services directly exposed to the internet (sshd, nginx, apache2, kernel network stack).
Tier 2 - patch within 7 days: CVSS 7.0-8.9 with AV:N, or any vulnerability in a service reachable from internal networks. Also includes kernel privilege escalation vulnerabilities even if AV:L, because local exploit means any compromised process can escalate.
Tier 3 - next maintenance window: Everything else. Low and medium severity, AV:L with privileges required, libraries with no direct exposure.
To pull only kernel CVEs with available fixes on Ubuntu, run the following. The `ubuntu-advantage-tools` package (ua / pro) also surfaces kernel livepatch availability, which lets you close kernel CVEs without a reboot on production systems enrolled in Ubuntu Pro.
# List kernel-related CVEs with available fixes on Ubuntu
ubuntu-security-status --unavailable 2>/dev/null || true
apt-cache show linux-image-$(uname -r) | grep -i cve
# Ubuntu Pro: check livepatch status
pro status --format json | jq '.services[] | select(.name=="livepatch")'
# RHEL: check available security errata
yum updateinfo list security --cve CVE-2024-6387
# Check if a specific binary links against a vulnerable library
ldd /usr/sbin/sshd | grep libcryptoApplying Patches: Safe Procedures for Production
The fastest way to apply security patches on Debian/Ubuntu is `apt-get upgrade` with a security-only filter. Do not run `apt-get dist-upgrade` during an emergency patch cycle unless you have tested the full upgrade in staging - dist-upgrade resolves new dependencies and can change package versions beyond the security fix.
On RHEL 8/9 and derivatives, `dnf upgrade --security` applies only packages with security advisories. Adding `--bugfix` also pulls in bug fix updates, which sometimes contain security-relevant fixes that were not filed as advisories.
After applying kernel patches, you need a reboot to activate the new kernel. For services like sshd, nginx, or glibc, a service restart (or process restart for glibc) is sufficient. The `needrestart` tool checks which running processes link against updated libraries and tells you exactly what needs restarting - use it after every security patch run.
Always capture a pre-patch snapshot. On cloud instances this is a one-command operation. On bare metal with LVM, take an LVM snapshot of the root volume before patching. If the patched kernel fails to boot, you need a rollback path that does not require a console session.
# Ubuntu: apply security updates only
apt-get update
apt-get upgrade -y --with-new-pkgs -o Dpkg::Options::='--force-confdef' \
$(apt-get upgrade --dry-run 2>/dev/null | grep -i security | awk '{print $2}')
# Simpler: use unattended-upgrades for security-only
unattended-upgrades --dry-run --debug 2>&1 | grep 'Packages that will be upgraded'
unattended-upgrades -v
# RHEL/Rocky 9: security-only upgrade
dnf upgrade --security -y
# Check what needs restarting after library updates
apt-get install -y needrestart
needrestart -r l # list mode, no restart
# LVM snapshot before patching (adjust vg/lv names)
lvcreate -L5G -s -n root-pre-patch /dev/vg0/root
# AWS: create AMI snapshot before patching
aws ec2 create-image --instance-id i-0abc123 --name "pre-patch-$(date +%F)" --no-rebootKernel Live Patching Without Reboots
Kernel CVEs are among the most disruptive to patch because they require reboots. Three live patching solutions are in active use in 2026: Ubuntu Livepatch (part of Ubuntu Pro), RHEL's kpatch, and the upstream `kpatch` project for other distros.
Ubuntu Pro is free for up to 5 machines. Livepatch covers the Ubuntu kernel and provides patches within hours of a kernel CVE disclosure for supported LTS releases. On Ubuntu 24.04, enrollment takes two commands. After enrollment, `canonical-livepatch status` shows which CVEs have been patched without a reboot.
For RHEL 8/9, Red Hat provides kernel live patches via the `kpatch` package. These are available through the standard subscription and cover RHEL-specific CVEs. The patch is delivered as a kernel module that hot-patches running kernel functions.
Live patching does not eliminate the need to eventually reboot and run a fully patched kernel. The live patch is a bridge, not a permanent fix. Schedule reboots for the next maintenance window after applying a live patch. Running a live-patched kernel for more than 60 days without a full reboot is not recommended - the patch state can accumulate complexity.
# Ubuntu Pro: enroll and enable livepatch
pro attach
pro enable livepatch
# Check livepatch status and covered CVEs
canonical-livepatch status --verbose
# RHEL 9: install kpatch and apply live patches
dnf install kpatch
dnf install kpatch-patch-$(uname -r | sed 's/-/_/g')
kpatch list
# Verify a specific CVE is covered by live patch
kpatch list | grep CVE-2024-1086 Automating CVE Scanning and Patching in CI/CD
Manual patch cycles do not scale past 20 servers. The standard automation pattern chains three components: scheduled vulnerability scanning, alerting with severity gating, and automated patch application with rollback capability.
For the scanning layer, run trivy or grype on a nightly cron across your fleet and output JSON results to a central store (S3, a database, or an object store on your own infrastructure). A simple Python script can parse the JSON and generate alerts for Tier 1 findings. Pipe those alerts into PagerDuty, OpsGenie, or your team's Slack channel with the CVE ID, affected host, package, and CVSS score.
For automated patching, Ansible is the most common choice in sysadmin environments. A playbook that runs `apt-get upgrade` with security-only filtering, calls `needrestart`, and then validates service health covers 80% of patch automation needs. Keep the playbook idempotent and add a pre-task that takes a snapshot via the cloud provider's API.
If you are building more complex patch orchestration workflows - managing approval gates, rollback logic, multi-cloud fleet coordination, and audit trails - AI-assisted DevOps platforms like taskbotshub.ai can reduce the engineering overhead of wiring these pipelines together, particularly for teams that need to patch heterogeneous fleets across AWS, GCP, and on-premises hardware simultaneously.
For container workloads, integrate trivy into your CI pipeline so no image with a CVSS 9.0+ CVE can be pushed to production. GitHub Actions, GitLab CI, and Jenkins all have trivy integrations. Set `--exit-code 1` for critical findings to fail the build.
# Ansible playbook excerpt: security patch + health check
# roles/security_patch/tasks/main.yml
---
- name: Update apt cache
apt:
update_cache: yes
cache_valid_time: 3600
- name: Apply security upgrades only
apt:
upgrade: safe
default_release: "{{ ansible_distribution_release }}-security"
register: patch_result
- name: Check services that need restart
command: needrestart -r l -q
register: needrestart_output
changed_when: false
- name: Restart affected services
service:
name: "{{ item }}"
state: restarted
loop: "{{ services_to_restart }}"
when: needrestart_output.stdout | length > 0
- name: Verify sshd is running
service:
name: ssh
state: started
check_mode: no
# Trivy in GitHub Actions CI
# .github/workflows/trivy-scan.yml snippet
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@0.20.0
with:
image-ref: 'myapp:${{ github.sha }}'
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
exit-code: '1'Rollback: When a Patch Breaks Production
Security patches occasionally break things. glibc updates have broken symbol compatibility. Kernel updates have caused driver failures on specific hardware. nginx security patches have changed default behaviors. Having a tested rollback procedure before you patch is not paranoia - it is the difference between a 10-minute recovery and a 4-hour incident.
On Debian/Ubuntu, `apt-get` allows downgrading a specific package with `apt-get install package=version`. The previous version must still be in the apt cache or available in the package manager's history. `apt-cache showpkg openssh-server` lists available versions. If the cache has been cleaned, you need to pull the .deb from snapshot.debian.org (Debian) or the Ubuntu package archive.
For kernel rollbacks, the old kernel is not removed by default on most systems - it remains in GRUB. Boot into the previous kernel by editing GRUB at startup or by setting `GRUB_DEFAULT` to the old entry and running `update-grub`. On systems where you cannot access the console directly, set the next boot to use a specific kernel via `grub-reboot` before initiating a reboot.
On RHEL systems, `dnf history` shows a numbered transaction log. `dnf history undo
# Debian/Ubuntu: downgrade a specific package
apt-get install openssh-server=1:9.6p1-3ubuntu13.4
# List available versions in cache
apt-cache showpkg openssh-server
# Set GRUB to boot previous kernel on next boot only
grub-reboot "1>2" # submenu index - verify with: grep -n menuentry /boot/grub/grub.cfg
reboot
# RHEL: view dnf transaction history
dnf history list
dnf history info 42
dnf history undo 42
# LVM snapshot rollback (if pre-patch snapshot exists)
lvconvert --merge /dev/vg0/root-pre-patch
rebootPatch Compliance Reporting and Audit Trails
For environments subject to PCI-DSS, SOC 2, or ISO 27001 audits, you need evidence that patches were applied within required timeframes. The tooling for this is separate from the patching toolchain itself.
`lynis` is the most practical audit tool for generating patch compliance reports on Linux. It checks installed package versions against known CVEs, reviews system configuration, and produces a hardening index score. Run it post-patch and archive the report with a timestamp.
For fleet-wide compliance visibility, combine your nightly trivy JSON output with a simple dashboard. Grafana with a JSON/CSV data source, or even a static HTML report generated by a Python script, gives you a patch status heatmap across your server fleet. Label each server with its environment (production, staging, development) and SLA tier so the report surfaces priority gaps immediately.
When naming internal patch management projects, dashboards, or tooling, clear and searchable names help teams find documentation faster - a consideration worth making early. Services like nicename.me can help you check name availability across domains and namespaces if you are standing up an internal portal or external-facing compliance dashboard for clients.
For audit trail integrity, log all patch operations to a write-once store. On AWS, CloudTrail plus S3 with Object Lock covers this. On self-hosted infrastructure, ship your shell history and apt/dnf logs to a central syslog server over TLS where local users cannot modify them.
# Install and run lynis audit
apt-get install -y lynis # or: dnf install lynis
lynis audit system --quiet --report-file /var/log/lynis-$(date +%F).dat
lynis show details | grep -i cve
# Generate a patch compliance summary from apt history
grep 'Upgrade:' /var/log/apt/history.log | tail -50
# Ship apt/dnf logs to remote syslog
# /etc/rsyslog.d/50-apt-audit.conf
module(load="imfile")
input(type="imfile" File="/var/log/apt/history.log"
Tag="apt-history" Severity="info" Facility="local6")
local6.* @@syslog.internal:6514 # TCP with TLS
# Trivy: output JSON for compliance database ingestion
trivy fs --scanners vuln --format json \
--output /var/reports/trivy-$(hostname)-$(date +%F).json /Hardening Against Zero-Days While Patches Are Pending
Between CVE disclosure and patch availability, you need mitigations. For network-exposed services, the fastest mitigation is restricting access via firewall rules while you wait for the patched package.
For sshd vulnerabilities specifically, setting `LoginGraceTime 0` in sshd_config mitigated the regreSSHion race condition before patches were available - though this has its own trade-offs for connection handling under load. For application vulnerabilities, WAF rules targeting known exploit patterns can block active exploitation. AWS WAF, Cloudflare, and ModSecurity all deployed regreSSHion mitigation rules within 48 hours of disclosure.
Kernel exploit mitigations built into the kernel itself - ASLR, SMEP, SMAP, stack canaries - reduce the exploitability of kernel CVEs. Verify these are active on your systems. AppArmor or SELinux profiles for exposed services constrain what a successful exploit can actually do, reducing blast radius even when a vulnerability exists.
For container environments, running containers as non-root with a read-only root filesystem and dropped capabilities (`--cap-drop ALL`) means a compromised container has minimal access to the host kernel attack surface. This is not a substitute for patching but it makes many kernel CVEs unexploitable from within a container.
# Verify kernel security mitigations are active
cat /proc/sys/kernel/randomize_va_space # should be 2 (full ASLR)
grep -r '' /sys/kernel/security/lsm 2>/dev/null || cat /sys/kernel/security/lsm
# Check AppArmor status
aa-status --json | jq '{profiles_loaded: .profiles | length, processes_confined: .processes | length}'
# SELinux status (RHEL)
getenforce
sestatus -v | head -20
# Verify SMEP/SMAP on x86_64
grep -m1 'flags' /proc/cpuinfo | grep -oE 'smep|smap'
# Quick firewall mitigation: restrict sshd to known CIDRs while patch is pending
nft add rule inet filter input tcp dport 22 ip saddr != 10.0.0.0/8 drop
# Run container with minimal capabilities
docker run --rm --cap-drop ALL --read-only --user 1000:1000 myapp:latest