Promethues + Alertmanager + node_exporter の最小構成 docker compose

• 概要
• 環境
• compose.yml
• Promethues
• Alertmanager
• 動作確認
• 最後に

概要

メモがてら残しておきます

環境

• Ubuntu 22.04
• docker 24.0.4

compose.yml

• vim compose.yml

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    volumes:
      - ./alertmanager:/etc/alertmanager
    command: "--config.file=/etc/alertmanager/alertmanager.yml"
    ports:
      - 9093:9093
    restart: unless-stopped
volumes:
  prom_data:

Promethues

• vim promethues/prometheus.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093

rule_files:
- '/etc/prometheus/*.rules'

scrape_configs:
- job_name: 'node_exporter'
  static_configs:
  - targets:
    - '192.168.100.1:9100'

• vim prometheus/node.rules

groups:
- name: Node
  rules:
  - record: instance:node_cpus:count
    expr: count without(cpu, mode) (node_cpu{mode="idle"})

Alertmanager

• vim alertmanager/alertmanager.yml

oute:
  receiver: default-receiver
  routes:
  - receiver: email
    matchers:
    - alertname=""
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
receivers:
- name: default-receiver
- name: email
  email_configs:
  - send_resolved: true
    to: your_mail_address@here
    from: your_from_address@here
    hello: localhost
    smarthost: smtp.ess.nifcloud.com:465
    auth_username: xxx
    auth_password: xxx
    headers:
      Subject: '{{ template "my.email.text.subject" . }}'
    text: '{{ template "my.email.text.body" . }}'
    require_tls: false
templates:
- /etc/alertmanager/*.tmpl

• vim alertmanager/my_email_text.tmpl

{{ define "my.email.text.body" }}
{{ range .Alerts.Firing }}
{{ range .Annotations.SortedPairs }}
- {{ .Name }} = {{ .Value }}
{{ end }}
{{ end }}
{{ end }}

{{ define "my.email.text.subject" }}
{{ $alerts := "" }}
{{ $instance_id := (index .Alerts 0).Labels.instance_id }}
{{ range .Alerts }}
{{ $alerts = (printf "%v %v" $alerts .Labels.alertname) }}
{{ end }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] ({{ $instance_id }} ->{{ $alerts }})
{{ end }}

動作確認

• docker compose up -d
• docker compose ps

NAME                IMAGE                                     COMMAND                  SERVICE             CREATED             STATUS                    PORTS
alertmanager        prom/alertmanager                         "/bin/alertmanager -…"   alertmanager        13 minutes ago      Up 13 minutes             0.0.0.0:9093->9093/tcp, :::9093->9093/tcp
node_exporter       quay.io/prometheus/node-exporter:latest   "/bin/node_exporter …"   node_exporter       13 minutes ago      Up 13 minutes
prometheus          prom/prometheus                           "/bin/prometheus --c…"   prometheus          13 minutes ago      Up 13 minutes             0.0.0.0:9090->9090/tcp, :::9090->9090/tcp

最後に

あとは必要に応じて node.rules ファイルや prometheus.yml でメトリックスの取得先を変更すれば OK です

my_email_text.tmpl を変更すればテキストメールのテンプレートも変更できます