How Prometheus Works: The Pull Model

Prometheus scrapes metrics from HTTP endpoints called exporters. Your targets expose metrics at /metrics in the Prometheus text format, and Prometheus pulls them on a configurable interval. This is the opposite of push-based systems like Graphite or InfluxDB with Telegraf.

The pull model has a concrete operational advantage: if a target goes silent, Prometheus knows immediately because the scrape fails. With push-based systems, a silent host looks identical to a healthy host that just hasn't pushed yet. That distinction matters at 3am.

Metrics are stored in a custom time-series database on local disk under the --storage.tsdb.path directory. The default retention period is 15 days. For longer retention, either increase --storage.tsdb.retention.time or federate to Thanos or Cortex. In our experience, 15 days covers the vast majority of incident postmortem windows without consuming unreasonable disk space - expect roughly 1-2 GB per monitored host per day depending on scrape interval and metric cardinality.

Installing Prometheus from Binary

Do not use the package manager version. The Ubuntu 24.04 apt repo ships Prometheus 2.45 as of mid-2026 while upstream is at 2.51. Download directly from the GitHub releases page.

Create a dedicated system user with no login shell, then download and extract the binary. We pin to a specific version in every production environment - floating to 'latest' has burned us before during major version transitions.

# Create system user
useradd --no-create-home --shell /bin/false prometheus

# Create directories
mkdir -p /etc/prometheus /var/lib/prometheus

# Download Prometheus 2.51.2
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
tar xzf prometheus-2.51.2.linux-amd64.tar.gz
cd prometheus-2.51.2.linux-amd64

# Install binaries
cp prometheus promtool /usr/local/bin/
chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

# Install console files
cp -r consoles/ console_libraries/ /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Writing the Prometheus Configuration File

The main configuration file is /etc/prometheus/prometheus.yml. The YAML structure is strict about indentation. We have seen more 'Prometheus won't start' tickets caused by YAML indentation errors than by any other issue.

The global block sets defaults that individual scrape configs can override. scrape_interval of 15s is the standard starting point - 10s adds 50% more I/O and storage with minimal benefit for most infrastructure metrics. Use 10s or lower only for latency-sensitive application metrics where you need sub-minute granularity for SLO calculations.

The scrape_configs block is where you define targets. Start with two jobs: one for Prometheus itself and one for node_exporter on your hosts.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    environment: 'production'
    datacenter: 'nyc1'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

rule_files:
  - '/etc/prometheus/rules/*.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
          - '10.0.1.10:9100'
          - '10.0.1.11:9100'
          - '10.0.1.12:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):\d+'
        replacement: '$1'
// advertisement

Installing node_exporter on Target Hosts

node_exporter is the standard exporter for Linux host metrics: CPU, memory, disk I/O, network throughput, filesystem usage, and system load. It exposes roughly 1000 metrics per host by default. Version 1.7.0 added support for cgroups v2 metrics, which matters if your hosts are running systemd with the unified hierarchy.

Install node_exporter on every host you want to monitor. The process mirrors the Prometheus installation: download the binary, create a system user, configure a systemd unit.

By default node_exporter listens on 0.0.0.0:9100. On internet-facing hosts, bind it to the internal interface only and firewall port 9100 so only your Prometheus server can reach it. We have found unprotected node_exporter endpoints exposed to the internet on production servers more than once during security audits.

# On each target host
useradd --no-create-home --shell /bin/false node_exporter

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz
tar xzf node_exporter-1.8.1.linux-amd64.tar.gz
cp node_exporter-1.8.1.linux-amd64/node_exporter /usr/local/bin/
chown node_exporter:node_exporter /usr/local/bin/node_exporter

# Create systemd unit
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=10.0.1.10:9100 \
  --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|run)($|/)'
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter

# Verify
curl -s http://10.0.1.10:9100/metrics | head -20

Creating the Prometheus systemd Unit

Run Prometheus as a systemd service so it restarts on failure and starts at boot. Set storage retention explicitly rather than relying on the default - we set 30d in production and mount /var/lib/prometheus on a dedicated volume.

After creating the unit file, validate your configuration with promtool before starting the service. A bad YAML file will cause Prometheus to exit immediately, and the error message in journalctl is not always obvious about which line caused the problem.

cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# Validate config before starting
promtool check config /etc/prometheus/prometheus.yml

systemctl daemon-reload
systemctl enable --now prometheus

# Check it's up
curl -s http://127.0.0.1:9090/-/healthy

Writing Useful Alert Rules

Alerts live in separate YAML files under /etc/prometheus/rules/. This separation makes it easier to manage rules in version control and load them incrementally. The rule_files glob in prometheus.yml picks up everything in that directory.

Start with four alerts that cover the most common production incidents: host down, high CPU, disk filling, and high memory pressure. These four alerts alone will catch the majority of issues that wake people up at night. Tune the thresholds based on your actual baseline - a for: 5m duration prevents alerting on transient spikes.

After writing rules, validate them with promtool. This catches PromQL syntax errors before you reload Prometheus.

# /etc/prometheus/rules/node_alerts.yml
groups:
  - name: node_alerts
    rules:
      - alert: HostDown
        expr: up{job='node'} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Host {{ $labels.instance }} is down'
          description: 'Prometheus has been unable to scrape {{ $labels.instance }} for more than 1 minute'

      - alert: HighCPULoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High CPU on {{ $labels.instance }}'
          description: 'CPU load is {{ $value | printf "%.1f" }}% on {{ $labels.instance }}'

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint='/',fstype!~'tmpfs|overlay'} / node_filesystem_size_bytes{mountpoint='/',fstype!~'tmpfs|overlay'}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Low disk on {{ $labels.instance }}'
          description: '{{ $value | printf "%.1f" }}% free on root filesystem'

      - alert: HighMemoryPressure
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Low memory on {{ $labels.instance }}'
          description: 'Only {{ $value | printf "%.1f" }}% memory available on {{ $labels.instance }}'

# Validate
promtool check rules /etc/prometheus/rules/node_alerts.yml

# Reload without restart (requires --web.enable-lifecycle)
curl -X POST http://127.0.0.1:9090/-/reload
// advertisement

Configuring AlertManager

Prometheus evaluates alert rules and fires alerts to AlertManager. AlertManager handles routing, deduplication, silencing, and sending notifications. They are separate binaries that run independently - if AlertManager is down, Prometheus continues evaluating rules and will resume sending once AlertManager recovers.

Install AlertManager the same way as Prometheus and Grafana: download the binary, create a user, write a systemd unit. AlertManager listens on port 9093 by default.

The AlertManager config routes alerts by severity label. This example routes critical alerts to PagerDuty and all alerts to a Slack channel. The group_wait of 30s prevents AlertManager from immediately firing a page the moment an alert triggers - it waits to see if related alerts arrive so it can group them into a single notification.

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - channel: '#alerts'
        title: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ .GroupLabels.alertname }} on {{ .GroupLabels.instance }}'

Service Discovery for Dynamic Environments

Static target lists work for small, stable infrastructure. For anything that autoscales or changes frequently, use service discovery. Prometheus ships with built-in SD for AWS EC2, GCP, Hetzner, Consul, Kubernetes, and file-based discovery.

File-based service discovery is the most portable option - it works with any infrastructure by reading target definitions from JSON or YAML files that your provisioning system writes. Prometheus watches these files and reloads targets without a restart or API call.

This approach integrates well with tools like Terraform, Ansible, or AI-assisted DevOps pipelines such as taskbotshub.ai where automated workflows can update the target file as instances are provisioned or decommissioned, keeping your monitoring inventory in sync without manual intervention.

# In prometheus.yml - file-based service discovery
scrape_configs:
  - job_name: 'node'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.json'
        refresh_interval: 30s

# Example target file: /etc/prometheus/targets/web-servers.json
# Prometheus reloads this automatically when it changes
cat > /etc/prometheus/targets/web-servers.json << 'EOF'
[
  {
    "targets": ["10.0.1.10:9100", "10.0.1.11:9100"],
    "labels": {
      "env": "production",
      "role": "web",
      "datacenter": "nyc1"
    }
  },
  {
    "targets": ["10.0.2.10:9100"],
    "labels": {
      "env": "staging",
      "role": "web",
      "datacenter": "nyc1"
    }
  }
]
EOF

Essential PromQL Queries for Linux Hosts

PromQL is a functional query language. The learning curve is steeper than SQL but the power is significant once you understand vector matching and aggregation operators. These are the queries we run daily on production systems.

Rate vs irate: use rate() for alerting and dashboards because it calculates a per-second average over the range window, smoothing spikes. Use irate() only when you need to see instantaneous spikes - it uses the last two data points only and is more sensitive to noise.

For disk I/O saturation, the node_disk_io_time_seconds_total metric is the most useful single indicator - it measures the time the disk was busy, and if it approaches 100% (1.0 when expressed as a ratio), the disk is saturated regardless of throughput numbers.

# CPU usage per core (percentage)
100 - (avg by(instance, cpu) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)

# Total CPU usage per host
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)

# Memory available percentage
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk read throughput (bytes/sec)
rate(node_disk_read_bytes_total[5m])

# Disk saturation (0-1, alert if > 0.9)
rate(node_disk_io_time_seconds_total[5m])

# Network receive throughput
rate(node_network_receive_bytes_total{device!~'lo|docker.*|veth.*'}[5m])

# Filesystem usage percentage
100 - ((node_filesystem_avail_bytes{mountpoint='/',fstype!='tmpfs'} /
  node_filesystem_size_bytes{mountpoint='/',fstype!='tmpfs'}) * 100)

# System load relative to CPU count (load per core)
node_load1 / count by(instance) (node_cpu_seconds_total{mode='idle'})

# Open file descriptors usage
node_filefd_allocated / node_filefd_maximum
// advertisement

Connecting Grafana to Prometheus

Prometheus has a built-in expression browser at /graph but it's only useful for ad-hoc queries. For persistent dashboards, Grafana is the standard choice. Install Grafana 11.x from the official APT or RPM repo - do not use Docker for production monitoring infrastructure unless you have a strong reason, because you add a layer of complexity to a system that needs to be reliable when everything else is broken.

Add Prometheus as a data source in Grafana's web UI at http://your-grafana:3000. Set the URL to http://127.0.0.1:9090 if Grafana and Prometheus are on the same host. Import dashboard ID 1860 from grafana.com - this is the Node Exporter Full dashboard maintained by the community and covers almost everything from node_exporter in a single import.

For naming your Grafana organization, Prometheus installation, or any project-facing services, choosing a clean name early matters more than you might expect. Tools like nicename.me can help you check availability across domains and namespaces before you commit to a name that turns out to be taken everywhere that matters.

# Install Grafana on Ubuntu 24.04
apt-get install -y apt-transport-https software-properties-common
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo 'deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main' \
  > /etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install -y grafana

systemctl enable --now grafana-server

# Add Prometheus data source via API (skip the UI click)
curl -s -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://127.0.0.1:9090",
    "access": "proxy",
    "isDefault": true
  }' \
  http://admin:admin@localhost:3000/api/datasources

Security Hardening for Production

A default Prometheus installation exposes the entire TSDB query interface and target list to anyone who can reach port 9090. In production, lock this down before anything else.

Bind Prometheus to 127.0.0.1 (as shown in our systemd unit above) and proxy it through nginx with TLS and HTTP basic auth. The --web.enable-admin-api flag, which enables endpoints like /api/v1/admin/tsdb/delete_series, should never be enabled unless you specifically need it.

For node_exporter, bind to the internal management network interface only, and use firewalld or nftables to restrict port 9100 to the Prometheus server IP. This is not paranoia - we have seen monitoring infrastructure used as a pivot point in security incidents because it was overlooked during network hardening.

# nginx reverse proxy for Prometheus with basic auth
apt-get install -y nginx apache2-utils
htpasswd -c /etc/nginx/.prometheus-htpasswd admin

cat > /etc/nginx/sites-available/prometheus << 'EOF'
server {
    listen 443 ssl;
    server_name prometheus.internal.example.com;

    ssl_certificate /etc/ssl/certs/prometheus.crt;
    ssl_certificate_key /etc/ssl/private/prometheus.key;

    location / {
        auth_basic 'Prometheus';
        auth_basic_user_file /etc/nginx/.prometheus-htpasswd;
        proxy_pass http://127.0.0.1:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
EOF

ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx

# Firewall: restrict node_exporter port to Prometheus server only
nft add rule inet filter input \
  ip saddr 10.0.0.5 tcp dport 9100 accept
nft add rule inet filter input \
  tcp dport 9100 drop