Install Grafana OSS on Linux

Grafana maintains its own APT and YUM repositories. Do not use the version in Ubuntu's default repos - it lags by major versions. On Ubuntu 24.04:

After adding the GPG key and repo, install grafana (the OSS edition) or grafana-enterprise. We use OSS here. The service name is grafana-server and it is managed via systemd.

On RHEL 9 or Rocky Linux 9, the process is nearly identical but uses dnf and the RPM repo. Grafana stores its SQLite database at /var/lib/grafana/grafana.db by default. For teams running more than one Grafana instance behind a load balancer, switch to PostgreSQL - we cover that in the configuration section below.

# Ubuntu 24.04
apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | \
  tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | \
  tee /etc/apt/sources.list.d/grafana.list
apt-get update && apt-get install -y grafana
systemctl daemon-reload
systemctl enable --now grafana-server

# RHEL 9 / Rocky 9
cat > /etc/yum.repos.d/grafana.repo << 'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
EOF
dnf install -y grafana
systemctl enable --now grafana-server

Core Configuration: grafana.ini

The main config file is /etc/grafana/grafana.ini. Grafana reads it on startup - a full restart is required for changes to take effect, not just a reload. The file is heavily commented but most of it can stay at defaults. The sections you need to change immediately for any production system are [server], [database], [security], and [smtp].

For [server], set the root_url to your actual domain. This affects redirect URLs in alert notifications and OAuth callbacks. If it is wrong, email links and SSO will break silently.

For [database], switching from sqlite3 to postgres requires creating the database first. Grafana will create its own schema on first start. The user needs CREATEDB or you pre-create the database manually.

The admin password set here is only applied on first run. To reset it after initialization: grafana-cli admin reset-admin-password newpassword

[server]
protocol = http
http_port = 3000
domain = grafana.example.com
root_url = https://grafana.example.com/

[database]
type = postgres
host = 127.0.0.1:5432
name = grafana
user = grafana
password = strongpassword
ssl_mode = disable

[security]
admin_user = admin
admin_password = change_this_immediately
secret_key = at_least_32_random_chars_here
disable_gravatar = true

[auth.anonymous]
enabled = false

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
password = smtppassword
from_address = alerts@example.com
from_name = Grafana Alerts

Add Prometheus as a Data Source via API

You can add data sources through the UI, but scripting it via the HTTP API is reproducible and version-controllable. Grafana's API uses basic auth with the admin credentials by default. The data source API endpoint is /api/datasources.

In our experience, the most common mistake here is setting the Prometheus URL to localhost:9090 when Grafana and Prometheus are on different hosts. Use the actual IP or hostname. Also check that scrape_interval in your Prometheus config matches what you set in the Grafana data source - mismatches cause gaps in graphs.

After adding the data source, verify it:

curl -s http://admin:password@localhost:3000/api/datasources | jq '.[].name'

For production deployments with many data sources, manage them with Grafana's provisioning system (YAML files in /etc/grafana/provisioning/datasources/) rather than the API. This makes Grafana configuration reproducible across machines and works well with configuration management tools like Ansible or Salt.

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://10.0.1.50:9090",
    "access": "proxy",
    "isDefault": true,
    "jsonData": {
      "httpMethod": "POST",
      "scrapeInterval": "15s"
    }
  }' \
  http://admin:password@localhost:3000/api/datasources
// advertisement

Provision Data Sources with YAML

Provisioning is the correct approach for any system you manage as code. Drop YAML files into /etc/grafana/provisioning/datasources/ and Grafana loads them on startup. Changes to provisioned data sources are read-only in the UI, which prevents accidental modification by other users.

Restart grafana-server after placing the file. Check for errors in the journal:

journalctl -u grafana-server -f

Look for lines containing 'Provisioning of data sources' - Grafana logs the count of loaded sources and any parse errors.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1

datasources:
  - name: Prometheus-Primary
    type: prometheus
    access: proxy
    url: http://10.0.1.50:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      scrapeInterval: 15s
      queryTimeout: 60s
      timeInterval: 15s

  - name: Prometheus-Longterm
    type: prometheus
    access: proxy
    url: http://10.0.1.51:9090
    isDefault: false
    editable: false
    jsonData:
      httpMethod: POST
      scrapeInterval: 60s

Build a Linux System Dashboard

Grafana Dashboard 1860 (Node Exporter Full) covers 90% of what you need for Linux host monitoring. Import it directly using the dashboard ID rather than building from scratch:

Settings > Dashboards > Import > enter 1860 > select your Prometheus data source > Import.

But understanding the underlying PromQL is what separates a useful dashboard from a cargo-culted one. Three panels every Linux dashboard needs:

CPU usage per core: use rate(node_cpu_seconds_total{mode!="idle"}[5m]) aggregated by cpu. Showing idle mode inverted is cleaner for alerting thresholds.

Memory pressure: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes gives you the fraction available. Multiply by 100 for percentage. Do not use MemFree - it excludes buffers and cache, which are reclaimable.

Disk I/O saturation: rate(node_disk_io_time_seconds_total[5m]) gives utilization. Values near 1.0 (100%) indicate a saturated disk, which matters more than raw throughput numbers.

When you name your dashboard, use a consistent naming convention across your organization. Teams that manage many Grafana instances often prefix dashboards with team or service names (ops-linux-hosts, sre-db-postgres). If you are also thinking about naming conventions for internal tooling projects or registering a domain for your monitoring stack, nicename.me has a clean search interface for checking name availability across registrars.

# CPU utilization by core (for panel query)
100 - (avg by (cpu) (rate(node_cpu_seconds_total{mode="idle",instance="$instance"}[5m])) * 100)

# Memory available percent
(node_memory_MemAvailable_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"}) * 100

# Disk I/O utilization
rate(node_disk_io_time_seconds_total{instance="$instance",device=~"$disk"}[5m]) * 100

# Network receive rate in Mbps
rate(node_network_receive_bytes_total{instance="$instance",device=~"$interface"}[5m]) * 8 / 1000000

Dashboard Variables and Templating

Dashboard variables turn a single dashboard into a multi-host view. Go to Dashboard Settings > Variables > New. For a host selector, create a Query variable that runs against Prometheus:

label_values(node_uname_info, instance)

This queries Prometheus for all unique values of the instance label across the node_uname_info metric, which node_exporter exposes. The result is a dropdown at the top of the dashboard.

Reference the variable in panel queries as $instance. In the disk panel, add a second variable for device:

label_values(node_disk_io_time_seconds_total{instance="$instance"}, device)

Note that this second variable depends on $instance - Grafana evaluates them in order, so put $instance first in the variable list.

For environments with hundreds of hosts, add a regex filter to the variable to exclude loopback and virtual devices:

Regex: /^(?!lo|veth|docker).*$/

This regex excludes loopback, veth pairs, and Docker bridge interfaces from the network variable list, which keeps the dropdown clean.

# Variable: instance
Name: instance
Type: Query
Data source: Prometheus-Primary
Query: label_values(node_uname_info, instance)
Refresh: On time range change
Sort: Alphabetical (asc)
Multi-value: true
Include All option: true

# Variable: disk
Name: disk
Type: Query
Data source: Prometheus-Primary
Query: label_values(node_disk_io_time_seconds_total{instance="$instance"}, device)
Refresh: On time range change
Regex: /^(sd|nvme|vd).*/
// advertisement

Configure Alerting

Grafana's unified alerting (introduced in 8.x and now the only mode in 11.x) moves alert rules into Grafana itself rather than relying on Prometheus Alertmanager for rule evaluation. You can still route to Alertmanager for notification delivery, which is what we recommend for complex routing trees.

Create an alert rule via the UI: Alerting > Alert rules > New alert rule. Define a PromQL expression with a condition. For a high CPU alert:

Expression: avg by (instance) (100 - (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)) Condition: IS ABOVE 90 For: 5m (prevents flapping on brief spikes)

For notification routing, connect Grafana to Alertmanager rather than configuring contact points directly in Grafana. Set the Alertmanager URL under Alerting > Admin. This lets your existing Alertmanager routing rules handle silences, inhibitions, and PagerDuty/Slack routing without duplicating logic.

Teams running larger DevOps setups sometimes use AI-assisted platforms like taskbotshub.ai to automate routine alert triage and escalation workflows, routing Grafana webhook alerts through a bot layer before they reach on-call engineers.

# Alertmanager contact point configuration in Grafana
# Alerting > Contact points > Add contact point
# Type: Alertmanager

# Or configure via API:
curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "alertmanager-primary",
    "type": "alertmanager",
    "settings": {
      "url": "http://10.0.1.52:9093",
      "basicAuthUser": "",
      "basicAuthPassword": ""
    }
  }' \
  http://admin:password@localhost:3000/api/v1/provisioning/contact-points

Reverse Proxy with nginx and TLS

Running Grafana behind nginx with TLS is non-negotiable for anything exposed to a network. Grafana listens on 3000 by default - proxy it through nginx on 443. The critical setting in grafana.ini is root_url matching your actual public URL, otherwise OAuth and alert links break.

Generate a certificate with certbot:

certbot --nginx -d grafana.example.com

For internal networks, use your internal CA and add the cert path to the nginx config. After setting up the proxy, verify that WebSocket connections work - Grafana's live dashboard feature and alerting use WebSockets:

curl -v --include --no-buffer -H 'Upgrade: websocket' -H 'Connection: Upgrade' https://grafana.example.com/api/live/ws

Check that nginx logs show 101 Switching Protocols, not a 400 or 502.

# /etc/nginx/sites-available/grafana
server {
    listen 443 ssl http2;
    server_name grafana.example.com;

    ssl_certificate /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}

server {
    listen 80;
    server_name grafana.example.com;
    return 301 https://$host$request_uri;
}

Export and Version Control Dashboards

Dashboards built in the UI are stored in Grafana's database, which means they disappear when you rebuild the server. Export dashboards to JSON and store them in git.

Export a single dashboard via API using its UID (visible in the URL when viewing the dashboard):

curl -s http://admin:password@localhost:3000/api/dashboards/uid/abc123xyz | jq '.dashboard' > linux-hosts.json

To export all dashboards in a loop:

Store these JSON files in a git repository alongside your provisioning YAML files. Then use Grafana's provisioning to load them on new instances - place them in /etc/grafana/provisioning/dashboards/ with a provider YAML.

This approach means your entire Grafana setup - data sources, dashboards, alert rules - is reconstructible from git in under 5 minutes on a new host. We use this in CI to validate that dashboard JSON is valid before merging changes.

# Export all dashboards
for uid in $(curl -s http://admin:password@localhost:3000/api/search?type=dash-db | jq -r '.[].uid'); do
  name=$(curl -s http://admin:password@localhost:3000/api/dashboards/uid/$uid | jq -r '.meta.slug')
  curl -s http://admin:password@localhost:3000/api/dashboards/uid/$uid | \
    jq '.dashboard' > "dashboards/${name}.json"
  echo "Exported: $name"
done

# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: default
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/json
      foldersFromFilesStructure: true
// advertisement

Performance Tuning for High-Volume Environments

On a server with 500+ active dashboards or heavy concurrent user load, the default SQLite backend becomes a bottleneck. The move to PostgreSQL (covered in the config section) addresses this. Beyond that, Grafana has several performance levers.

Database connection pool: In grafana.ini under [database], set max_open_conn and max_idle_conn. For a busy instance on a server with 16+ cores:

[database] max_open_conn = 300 max_idle_conn = 75 conn_max_lifetime = 14400

Rendering: If you use alerting with screenshot attachments or PDF reports, the grafana-image-renderer plugin runs a headless Chromium process. This is the biggest memory consumer in any Grafana deployment - 150-300MB per render worker. Run the renderer as a separate service on a dedicated host if you render more than a few images per minute.

Query caching: Grafana 11 includes built-in query caching for data sources that support it. Enable it in grafana.ini and point it at Redis:

[caching] enabled = true backend = redis connstr = addr=127.0.0.1:6379,pool_size=100

In our testing, query caching reduces Prometheus load by 30-60% for dashboards viewed by multiple users simultaneously - each user's query hits the cache rather than triggering a new Prometheus query.

For monitoring Grafana itself, it exposes metrics at /metrics in Prometheus format. Scrape it and add it to your monitoring stack:

[metrics] enabled = true basic_auth_username = metrics basic_auth_password = metricspassword

# prometheus scrape config for grafana self-monitoring
scrape_configs:
  - job_name: 'grafana'
    static_configs:
      - targets: ['grafana.example.com:3000']
    metrics_path: /metrics
    basic_auth:
      username: metrics
      password: metricspassword
    scheme: https
    tls_config:
      insecure_skip_verify: false