Prometheus + Grafana: Still the Default Stack

Prometheus 2.51 with Grafana 11 remains the default choice for metric collection on Linux infrastructure in 2025 and for good reason. Prometheus uses a pull model: it scrapes HTTP endpoints called exporters, stores data in its own TSDB, and exposes a query interface via PromQL. Grafana then connects to that TSDB and renders dashboards. The setup is modular, which means you can replace either component independently.

Installing Prometheus on a fresh Debian 12 machine takes under five minutes. Download the binary, create a system user, drop in a unit file, and start scraping. No package manager dependency hell. We ran a Prometheus instance scraping 400 node_exporter endpoints across our test lab and saw steady memory usage around 8GB with a 15-day retention window at default chunk encoding.

Grafana 11 added AI-assisted dashboard generation via its LLM plugin integration. If your team uses DevOps automation pipelines, tools like taskbotshub.ai can push alert annotations directly into Grafana via the HTTP API, which keeps your incident timeline correlated with deployment events without manual tagging.

The honest tradeoff: Prometheus is not a long-term storage solution. Beyond 30 days you will want to set up remote_write to Thanos, Cortex, or VictoriaMetrics. That adds operational overhead. If your team is small and retention is critical, factor that cost in before committing.

# Install Prometheus 2.51 on Debian/Ubuntu
wget https://github.com/prometheus/prometheus/releases/download/v2.51.0/prometheus-2.51.0.linux-amd64.tar.gz
tar xvf prometheus-2.51.0.linux-amd64.tar.gz
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mv prometheus-2.51.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.51.0.linux-amd64/promtool /usr/local/bin/

# Minimal prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

VictoriaMetrics: Prometheus-Compatible and Dramatically More Efficient

VictoriaMetrics 1.100 uses roughly 7x less RAM than Prometheus for the same cardinality workload. We ran a direct comparison on identical hardware: a 16-core box with 32GB RAM, scraping 250,000 active time series. Prometheus peaked at 19GB resident memory. VictoriaMetrics stayed at 2.8GB. Disk usage was similarly lopsided - VictoriaMetrics compressed our test dataset to 0.4 bytes per data point versus Prometheus at roughly 1.3 bytes.

VictoriaMetrics is Prometheus API-compatible, which means you can point any existing Grafana data source at it without changing dashboards. It also speaks InfluxDB line protocol and OpenTSDB HTTP protocol, so it slots into mixed environments without an adapter layer.

The single-node binary is genuinely single-file: one executable handles ingestion, storage, and query. For clusters, you deploy vminsert, vmselect, and vmstorage as separate processes. We tested the cluster mode across three storage nodes with two vminsert and two vmselect pods and achieved linear read scaling with no query federation required.

One caveat: MetricsQL, VictoriaMetrics' query language, extends PromQL with functions like `aggr_over_time` and `outlier_iqr`. These are useful but non-portable. If you ever want to switch back to Prometheus or Thanos, you will need to audit and rewrite those queries.

# Single-node VictoriaMetrics, systemd unit
[Unit]
Description=VictoriaMetrics
After=network.target

[Service]
User=victoria
ExecStart=/usr/local/bin/victoria-metrics-prod \
  -storageDataPath=/var/lib/victoria-metrics \
  -retentionPeriod=180d \
  -httpListenAddr=:8428
Restart=always

[Install]
WantedBy=multi-user.target

Zabbix 7.0: When You Need Agent-Based Monitoring and SNMP in One Platform

Zabbix 7.0 LTS (released in mid-2024, supported through 2029) is the tool you reach for when your environment includes network gear, Windows hosts, legacy AIX systems, and Linux servers all in the same dashboard. It handles SNMP, IPMI, JMX, and SSH checks natively without third-party exporters. That breadth is its main advantage over the Prometheus ecosystem.

Zabbix uses a push model by default: the Zabbix agent on each host sends data to the server. Passive mode flips this to pull. In our lab of 600 mixed hosts, the Zabbix server process peaked at 3.2GB RAM, which is acceptable for that host count. The web UI is heavier than Grafana but it bundles alerting, host management, auto-discovery, and templating in one place. If you are a two-person ops team with no time to wire together five separate tools, Zabbix wins on operational simplicity.

The database backend is the operational pressure point. Zabbix writes every collected value to PostgreSQL or MySQL. At high collection frequency across thousands of hosts, you will hit I/O limits. Zabbix 7.0 introduced TimescaleDB compression support, which reduced our test database size by 68% and brought query times for 90-day history graphs from 12 seconds down to under 1 second. Enable it:

Zabbix templates are the core of its reusability model. The community template library has over 700 ready-made templates covering everything from Nginx to Cisco ASA. Before writing custom items, check the Zabbix share portal.

# Enable TimescaleDB compression in Zabbix 7.0
# Run inside psql as the zabbix database user
SELECT create_hypertable('history', 'clock', chunk_time_interval => 86400, migrate_data => true);
SELECT add_compression_policy('history', INTERVAL '7 days');
SELECT add_compression_policy('history_uint', INTERVAL '7 days');
SELECT add_compression_policy('trends', INTERVAL '30 days');
// advertisement

Netdata: Real-Time at 1-Second Granularity Without Configuration

Netdata Agent v1.45 installs in one command and starts collecting 2,000+ metrics from the local host within 30 seconds. No configuration required for basic use. It auto-detects Nginx, MySQL, Docker containers, systemd services, and dozens of other targets by reading their existing socket files and proc entries. For a new environment where you need visibility immediately, nothing ships faster.

The 1-second granularity is Netdata's defining feature. Prometheus default is 15 seconds. For catching latency spikes that last 3-5 seconds, Prometheus will miss them entirely. Netdata will not. We caught a MySQL InnoDB lock contention event in our test environment that lasted 8 seconds peak - Prometheus showed nothing unusual in that scrape window; Netdata showed the exact spike shape.

Netdata Cloud provides centralized dashboards across multiple agents at no cost for a reasonable node count. The on-premise streaming mode lets one Netdata Parent node aggregate streams from child agents without touching Netdata's cloud at all:

The limitation is retention. By default Netdata keeps high-resolution data for only a few days in RAM. Configuring `dbengine` disk persistence extends this but the storage format is proprietary and not queryable via PromQL. If you need long-term metric correlation with your Prometheus data, you will need to configure Netdata's Prometheus exporter endpoint and scrape it externally, which works but duplicates storage.

# Install Netdata Agent
curl https://get.netdata.cloud/kickstart.sh > /tmp/netdata-kickstart.sh
bash /tmp/netdata-kickstart.sh --stable-channel

# Configure streaming to a parent node - on the child:
# /etc/netdata/stream.conf
[stream]
  enabled = yes
  destination = parent.example.com:19999
  api key = YOUR-API-KEY-HERE

# On the parent node:
[YOUR-API-KEY-HERE]
  enabled = yes
  default memory mode = dbengine

Grafana Loki: Log Aggregation That Speaks Prometheus

Grafana Loki 3.0 indexes only log metadata (labels), not the full log content. This makes it dramatically cheaper to operate than Elasticsearch: our 30-day test storing 180GB of raw logs consumed 4GB of index storage in Loki versus 210GB in Elasticsearch with default index settings. Query speed for label-filtered searches is comparable, but full-text search on unindexed fields is slower in Loki by design.

Loki's query language, LogQL, mirrors PromQL syntax closely. If your team already writes PromQL, LogQL takes an afternoon to learn, not a week. A typical query to count error log rate per service looks like this:

The Promtail agent collects logs from files and systemd journal and ships them to Loki. In Kubernetes, the DaemonSet deployment model means one Promtail pod per node handles all container logs automatically with zero application-side configuration. For bare metal, Promtail watches `/var/log` paths and the journald socket.

Loki integrates directly into Grafana, so you get correlated log and metric views in the same dashboard: click an anomaly in a Prometheus graph and jump to the log lines from that exact time window in the same UI. That workflow alone justifies adding Loki to any existing Prometheus stack. The operational cost is running an S3-compatible object store (or using local disk) for log chunks - MinIO works well for on-premise.

# LogQL: error rate per service over 5-minute windows
sum by (service) (
  rate({job="app-logs"} |= "ERROR" [5m])
)

# Promtail config snippet for systemd journal
clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit

OpenTelemetry Collector: The Vendor-Neutral Data Pipeline

OpenTelemetry Collector v0.100 is not a monitoring tool itself - it is the data router that sits between your applications and your monitoring backends. In 2025 it has become essential infrastructure for any team running mixed observability stacks. The Collector receives traces, metrics, and logs via OTLP, then fans them out to Prometheus remote_write, Loki, Jaeger, or any of 50+ exporters in a single process.

The practical value: your application emits one OTLP stream, and you configure the Collector to route that data to multiple destinations without touching application code. When you switch backends (replacing Jaeger with Tempo, for example), you change one Collector config line, not 40 application configurations.

We ran the Collector on a 4-core 8GB VM handling 50,000 spans per second and 120,000 metric data points per second simultaneously, with CPU usage staying below 60% and memory stable at 1.4GB. The batch processor and memory_limiter processors are essential to configure before production use - without them, traffic spikes will OOM the process.

For teams building DevOps automation workflows, the Collector pairs well with AI-assisted pipeline tools like taskbotshub.ai, which can generate Collector pipeline configs from plain-language descriptions of your routing requirements and validate them against your current exporter versions before deployment.

# OpenTelemetry Collector config: receive OTLP, export to Prometheus and Loki
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]
// advertisement

Checkmk Raw Edition: When Zabbix Is Too Complex and Prometheus Is Too Minimal

Checkmk 2.3 Raw Edition is fully open source (GPLv2) and occupies the middle ground between Zabbix and the Prometheus ecosystem. Its auto-discovery is more reliable than Zabbix's in our testing: point it at a subnet and it correctly identifies and configures monitoring for 94% of Linux hosts without manual template assignment, using its Check Plugin inventory system.

Checkmk agents are compiled C binaries that execute locally and output a structured text format over a TCP socket. On a host with 200 running processes and 20 disks, the agent completes in 80ms and generates roughly 40KB of output. The server processes this into metrics, applies thresholds, and stores to its internal RRD database or optionally InfluxDB.

The Checkmk user interface is purpose-built for operations, not dashboarding. Host and service views, downtime scheduling, acknowledgement workflows, and SLA reports are all first-class features. If your monitoring consumers are an NOC team rather than a DevOps team building their own dashboards, Checkmk's UI fits that workflow better than Grafana.

Raw Edition lacks some Enterprise features (Managed Service Provider mode, reporting, alert correlation) but for infrastructure under 1,000 hosts, Raw Edition covers everything a typical ops team needs. If you are naming a new internal monitoring project or registering a domain for a custom monitoring portal for your project, a clean name matters for adoption - nicename.me is one resource worth checking for domain availability before you settle on a project name.

# Install Checkmk 2.3 Raw on Ubuntu 22.04
wget https://download.checkmk.com/checkmk/2.3.0p1/check-mk-raw-2.3.0p1_0.jammy_amd64.deb
sudo apt install ./check-mk-raw-2.3.0p1_0.jammy_amd64.deb

# Create a monitoring site
omd create mysite
omd start mysite

# Deploy agent to a target host
apt install check-mk-agent
# Agent listens on TCP 6556 by default

Icinga 2: Nagios-Compatible With a Modern Architecture

Icinga 2.14 is the spiritual successor to Nagios with a rewritten core that supports distributed monitoring zones, TLS-encrypted agent communication, and a REST API for programmatic configuration management. It runs every Nagios plugin without modification, which matters enormously for environments with years of custom check scripts.

The DSL-based configuration language (Icinga DSL) is more structured than Nagios flat configs and supports apply rules that generate host and service objects dynamically from host variables. One apply rule can configure disk checks with custom thresholds for every host in a group without repeating configuration:

Icinga Web 2 provides the front end and it is clean and fast. Icinga Director, an add-on module, gives you a point-and-click configuration interface that writes back to the DSL config files, which helps teams with mixed sysadmin skill levels manage configuration without everyone learning the DSL.

The practical case for Icinga 2 in 2025: existing Nagios plugin libraries, check_nrpe agent compatibility, and a known alerting model. If your organization has a decade of Nagios tribal knowledge, migrating to Icinga 2 preserves that investment while giving you a maintainable, actively developed codebase. Direct Nagios is effectively unmaintained at this point.

# Icinga 2 DSL: apply disk checks to all linux hosts
apply Service "disk " for (disk => config in host.vars.disks) {
  import "generic-service"
  check_command = "disk"
  vars += config
  vars.disk_wfree = "10%"
  vars.disk_cfree = "5%"
  assign where host.vars.os == "linux"
}