Configure Alerting Rules

Set up Prometheus alerting rules and Alertmanager for reliable, actionable incident notifications.

See Extended Examples for complete configuration files and templates.

When to Use

Implementing proactive monitoring with automated incident detection
Routing alerts to appropriate teams based on severity and service ownership
Reducing alert fatigue through intelligent grouping and deduplication
Integrating monitoring with on-call systems (PagerDuty, Opsgenie)
Establishing escalation policies for critical production issues
Migrating from legacy monitoring systems to Prometheus-based alerting
Creating actionable alerts that guide responders to resolution

Inputs

Required: Prometheus metrics to alert on (error rates, latency, saturation)
Required: On-call rotation and escalation policies
Optional: Existing alert definitions to migrate
Optional: Notification channels (Slack, email, PagerDuty)
Optional: Runbook documentation for common alerts

Procedure

Step 1: Deploy Alertmanager

Install and configure Alertmanager to receive alerts from Prometheus.

Docker Compose deployment (basic structure):

yaml

version: '3.8'
services:
  alertmanager:
    image: prom/alertmanager:v0.26.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    # ... (see EXAMPLES.md for complete configuration)

Basic Alertmanager configuration (

alertmanager.yml

excerpt):

yaml

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical

# ... (see EXAMPLES.md for complete routing, inhibition rules, and receivers)

Configure Prometheus to use Alertmanager (

prometheus.yml

yaml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      timeout: 10s
      api_version: v2

Expected: Alertmanager UI accessible at

http://localhost:9093

, Prometheus "Status > Alertmanagers" shows UP status.

On failure:

Check Alertmanager logs:
```
docker logs alertmanager
```

Verify Prometheus can reach Alertmanager:

curl http://alertmanager:9093/api/v2/status

Test webhook URLs:

curl -X POST <SLACK_WEBHOOK_URL> -d '{"text":"test"}'

Validate YAML syntax:
```
amtool check-config alertmanager.yml
```

Step 2: Define Alerting Rules in Prometheus

Create alerting rules that fire when conditions are met.

Create alerting rules file (

/etc/prometheus/rules/alerts.yml

excerpt):

yaml

groups:
  - name: instance_alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been down for >5min."
          runbook_url: "https://wiki.example.com/runbooks/instance-down"

      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          # ... (see EXAMPLES.md for complete alerts)

Alert design best practices:

for
duration: Prevents flapping alerts. Use 5-10 minutes for most alerts.
Descriptive annotations: Include current value, affected resource, and runbook link.
Severity levels: critical (pages on-call), warning (investigate), info (FYI)
Team labels: Enable routing to correct team/channel
Runbook links: Every alert should have a runbook URL

Load rules into Prometheus:

yaml

# prometheus.yml
rule_files:
  - "rules/*.yml"

Validate and reload:

bash

promtool check rules /etc/prometheus/rules/alerts.yml
curl -X POST http://localhost:9090/-/reload

Expected: Alerts visible in Prometheus "Alerts" page, alerts fire when thresholds exceeded, Alertmanager receives fired alerts.

On failure:

Check Prometheus logs for rule evaluation errors
Verify rule syntax with
```
promtool check rules
```
Test alert queries independently in Prometheus UI
Inspect alert state transitions: Inactive → Pending → Firing

Step 3: Create Notification Templates

Design readable, actionable notification messages.

Create template file (

/etc/alertmanager/templates/default.tmpl

excerpt):

gotmpl

{{ define "slack.default.title" }}
[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}
{{ end }}

{{ define "slack.default.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Summary:* {{ .Annotations.summary }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
{{ end }}
{{ end }}

# ... (see EXAMPLES.md for complete email and PagerDuty templates)

Use templates in receivers:

yaml

receivers:
  - name: 'slack-custom'
    slack_configs:
      - channel: '#alerts'
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'

Expected: Notifications formatted consistently, include all relevant context, actionable with runbook links.

On failure:

Test template rendering:

amtool template test --config.file=alertmanager.yml

Check template syntax errors in Alertmanager logs
Use
```
{{ . | json }}
```
to debug template data structure

Step 4: Configure Routing and Grouping

Optimize alert delivery with intelligent routing rules.

Advanced routing configuration (excerpt):

yaml

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s

  routes:
    - match:
        team: platform
      receiver: 'team-platform'
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty-platform'
          group_wait: 10s
          repeat_interval: 15m
          continue: true   # Also send to Slack

# ... (see EXAMPLES.md for complete routing with time intervals)

Grouping strategies:

yaml

# Group by alertname: All HighCPU alerts bundled together
group_by: ['alertname']

# Group by alertname AND cluster: Separate notifications per cluster
group_by: ['alertname', 'cluster']

Expected: Alerts routed to correct teams, grouped logically, timing appropriate for severity.

On failure:

Test routing:

amtool config routes test --config.file=alertmanager.yml --alertname=HighCPU --label=severity=critical

Check routing tree:

amtool config routes show --config.file=alertmanager.yml

Verify
```
continue: true
```
if alert should match multiple routes

Step 5: Implement Inhibition and Silencing

Reduce alert noise with inhibition rules and temporary silences.

Inhibition rules (suppress dependent alerts):

yaml

inhibit_rules:
  # Cluster down suppresses all node alerts in that cluster
  - source_match:
      alertname: 'ClusterDown'
      severity: 'critical'
    target_match_re:
      alertname: '(InstanceDown|HighCPU|HighMemory)'
    equal: ['cluster']

  # Service down suppresses latency and error alerts
  - source_match:
      alertname: 'ServiceDown'
    target_match_re:
      alertname: '(HighLatency|HighErrorRate)'
    equal: ['service', 'namespace']

# ... (see EXAMPLES.md for more inhibition patterns)

Create silences programmatically:

bash

# Silence during maintenance
amtool silence add \
  instance=app-server-1 \
  --author="ops-team" \
  --comment="Scheduled maintenance" \
  --duration=2h

# List and manage silences
amtool silence query
amtool silence expire <SILENCE_ID>

Expected: Inhibition reduces cascade alerts automatically, silences prevent notifications during planned maintenance.

On failure:

Test inhibition logic with live alerts
Check Alertmanager UI "Silences" tab
Verify silence matchers are exact (labels must match perfectly)

Step 6: Integrate with External Systems

Connect Alertmanager to PagerDuty, Opsgenie, Jira, etc.

PagerDuty integration (excerpt):

yaml

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'YOUR_INTEGRATION_KEY'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ range .Alerts.Firing }}{{ .Annotations.summary }}{{ end }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          alertname: '{{ .GroupLabels.alertname }}'
        # ... (see EXAMPLES.md for complete integration examples)

Webhook for custom integrations:

yaml

receivers:
  - name: 'webhook-custom'
    webhook_configs:
      - url: 'https://your-webhook-endpoint.com/alerts'
        send_resolved: true

Expected: Alerts create incidents in PagerDuty, appear in team communication channels, trigger on-call escalations.

On failure:

Verify API keys/tokens are valid
Check network connectivity to external services
Test webhook endpoints independently with curl
Enable debug mode:
```
--log.level=debug
```

Validation

Alertmanager receives alerts from Prometheus successfully
Alerts routed to correct teams based on labels and severity
Notifications delivered to Slack, email, or PagerDuty
Alert grouping reduces notification volume appropriately
Inhibition rules suppress dependent alerts correctly
Silences prevent notifications during maintenance windows
Notification templates include runbook links and context
Repeat interval prevents alert fatigue for long-running issues
Resolved notifications sent when alerts clear
External integrations (PagerDuty, Opsgenie) create incidents

Common Pitfalls

Alert fatigue: Too many low-priority alerts cause responders to ignore critical ones. Set strict thresholds, use inhibition.
Missing
for
duration: Alerts without
```
for
```
fire on transient spikes. Always use 5-10 minute windows.
Overly broad grouping: Grouping by
```
['...']
```
sends individual notifications. Use specific label grouping.
No runbook links: Alerts without runbooks leave responders guessing. Every alert needs a runbook URL.
Incorrect severity: Mislabeling warnings as critical desensitizes team. Reserve critical for emergencies.
Forgotten silences: Silences without expiration can hide real issues. Always set end times.
Single route: All alerts to one channel loses context. Use team-specific routing.
No inhibition: Cascade alerts during outages create noise. Implement inhibition rules.

Related Skills

```
setup-prometheus-monitoring
```
- Define metrics and recording rules that feed alerting rules
```
define-slo-sli-sla
```
- Generate SLO burn rate alerts for error budget management
```
write-incident-runbook
```
- Create runbooks linked from alert annotations
```
build-grafana-dashboards
```
- Visualize alert firing history and silence patterns

configure-alerting-rules

NPX Install

Tags

SKILL.md Content

Configure Alerting Rules

When to Use

Inputs

Procedure

Step 1: Deploy Alertmanager

Step 2: Define Alerting Rules in Prometheus

Step 3: Create Notification Templates

Step 4: Configure Routing and Grouping

Step 5: Implement Inhibition and Silencing

Step 6: Integrate with External Systems

Validation

Common Pitfalls

Related Skills