• Prometheus Server: Core component that scrapes and stores time-series data with local TSDB
• Alertmanager: Handles alerts, deduplication, grouping, routing, and notifications to receivers
• Pushgateway: Allows ephemeral jobs to push metrics (use sparingly - prefer pull model)
• Exporters: Convert metrics from third-party systems to Prometheus format (node, blackbox, etc.)
• Client Libraries: Instrument application code (Go, Java, Python, Rust, etc.)
• Prometheus Operator: Kubernetes-native deployment and management via CRDs
• Remote Storage: Long-term storage via Thanos, Cortex, Mimir for multi-cluster federation

• Metrics: Time-series data identified by metric name and key-value labels
• Format:

  metric_name{label1="value1", label2="value2"} sample_value timestamp

• Metric Types:
  • Counter: Monotonically increasing value (requests, errors) - use

    rate()

    or

    increase()

    for querying
  • Gauge: Value that can go up/down (temperature, memory usage, queue length)
  • Histogram: Observations in configurable buckets (latency, request size) - exposes

    _bucket

    ,

    _sum

    ,

    _count

  • Summary: Similar to histogram but calculates quantiles client-side - use histograms for aggregation

• "alerts/*.yml"
• "rules/*.yml"

Prometheus itself

• job_name: "prometheus" static_configs:
  • targets: ["localhost:9090"]

Application services

• job_name: "application" metrics_path: "/metrics" static_configs:
  • targets:
    • "app-1:8080"
    • "app-2:8080" labels: env: "production" team: "backend"

Kubernetes service discovery

• job_name: "kubernetes-pods" kubernetes_sd_configs:
  • role: pod relabel_configs:

  Only scrape pods with prometheus.io/scrape annotation

  • source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true

  Use custom metrics path if specified

  • source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: metrics_path regex: (.+)

  Use custom port if specified

  • source_labels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: address

  Add namespace label

  • source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace

  Add pod name label

  • source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name

  Add service name label

  • source_labels: [__meta_kubernetes_pod_label_app] action: replace target_label: app

Node Exporter for host metrics

• job_name: "node-exporter" static_configs:
  • targets:
    • "node-exporter:9100"

undefined

• "/etc/alertmanager/templates/*.tmpl"

Suppress warning alerts if critical alert is firing

• source_match: severity: "critical" target_match: severity: "warning" equal: ["alertname", "instance"]

Suppress instance alerts if entire service is down

• source_match: alertname: "ServiceDown" target_match_re: alertname: ".*" equal: ["service"]

• name: "default" slack_configs:
  • channel: "#alerts" title: "Alert: {{ .GroupLabels.alertname }}" text: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"
• name: "pagerduty" pagerduty_configs:
  • service_key: "YOUR_PAGERDUTY_SERVICE_KEY" description: "{{ .GroupLabels.alertname }}"
• name: "dba-team" slack_configs:
  • channel: "#database-alerts" email_configs:
  • to: "dba-team@example.com" headers: Subject: "Database Alert: {{ .GroupLabels.alertname }}"
• name: "slack-dev" slack_configs:
  • channel: "#dev-alerts" send_resolved: true

undefined

• seconds for duration (not milliseconds)
• bytes for size (not kilobytes)
• ratio for percentages (0.0-1.0, not 0-100)

undefined

• Keep label values to < 10 per label (ideally)
• Total unique time-series per metric should be < 10,000
• Use recording rules to pre-aggregate high-cardinality metrics
• Avoid labels with unbounded values (IDs, timestamps, user input)

• name: performance_rules interval: 30s rules:

  Pre-calculate request rates

  • record: job:http_requests:rate5m expr: sum(rate(http_requests_total[5m])) by (job)

  Pre-calculate error rates

  • record: job:http_request_errors:rate5m expr: sum(rate(http_request_errors_total[5m])) by (job)

  Pre-calculate error ratio

  • record: job:http_request_error_ratio:rate5m expr: | job:http_request_errors:rate5m / job:http_requests:rate5m

  Pre-aggregate latency percentiles

  • record: job:http_request_duration_seconds:p95 expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))
  • record: job:http_request_duration_seconds:p99 expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))
• name: aggregation_rules interval: 1m rules:

  Multi-level aggregation for dashboards

  • record: instance:node_cpu_utilization:ratio expr: | 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)
  • record: cluster:node_cpu_utilization:ratio expr: avg(instance:node_cpu_utilization:ratio)

  Memory aggregation

  • record: instance:node_memory_utilization:ratio expr: | 1 - ( node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes )

undefined

• name: symptom_alerts rules:

  GOOD: Alert on user-facing symptoms

  • alert: HighErrorRate expr: | ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > 0.05 for: 5m labels: severity: critical team: backend annotations: summary: "High error rate detected" description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)" runbook: "https://wiki.example.com/runbooks/high-error-rate"
  • alert: HighLatency expr: | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service) ) > 1 for: 5m labels: severity: warning team: backend annotations: summary: "High latency on {{ $labels.service }}" description: "P95 latency is {{ $value }}s (threshold: 1s)" impact: "Users experiencing slow page loads"

  GOOD: SLO-based alerting

  • alert: SLOBudgetBurnRate expr: | ( 1 - ( sum(rate(http_requests_total{status!~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) ) > (14.4 * (1 - 0.999)) # 14.4x burn rate for 99.9% SLO for: 5m labels: severity: critical team: sre annotations: summary: "SLO budget burning too fast" description: "At current rate, monthly error budget will be exhausted in {{ $value | humanizeDuration }}"

undefined

• name: infrastructure_alerts rules:

  Lower severity for infrastructure issues

  • alert: HighMemoryUsage expr: | ( node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes ) / node_memory_MemTotal_bytes > 0.9 for: 10m labels: severity: warning # Not critical unless symptoms appear team: infrastructure annotations: summary: "High memory usage on {{ $labels.instance }}" description: "Memory usage is {{ $value | humanizePercentage }}"
  • alert: DiskSpaceLow expr: | ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} ) < 0.1 for: 5m labels: severity: warning team: infrastructure annotations: summary: "Low disk space on {{ $labels.instance }}" description: "Only {{ $value | humanizePercentage }} disk space remaining" action: "Clean up logs or expand disk"

undefined

1. For duration: Use

   for

   clause to avoid flapping
2. Meaningful annotations: Include summary, description, runbook URL, impact
3. Proper severity levels: critical (page immediately), warning (ticket), info (log)
4. Actionable alerts: Every alert should require human action
5. Include context: Add labels for team ownership, service, environment

0.001 * 14.4 ) and ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) 0.001 * 14.4 )

undefined

• name: "default" slack_configs:
  • api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK" channel: "#alerts"

• job_name: "federate" scrape_interval: 15s honor_labels: true metrics_path: "/federate" params: "match[]": # Pull aggregated metrics only - '{job="prometheus"}' - '{name=~"job:.*"}' # Recording rules - "up" static_configs:
  • targets:
    • "prometheus-us-east-1:9090"
    • "prometheus-us-west-2:9090"
    • "prometheus-eu-west-1:9090" labels: region: "us-east-1"

undefined

• url: "http://thanos-receive:19291/api/v1/receive" queue_config: capacity: 10000 max_shards: 50 min_shards: 1 max_samples_per_send: 5000 batch_send_deadline: 5s min_backoff: 30ms max_backoff: 100ms write_relabel_configs:

  Drop high-cardinality metrics before remote write

  • source_labels: [name] regex: "go_.*" action: drop

• url: "http://thanos-query:9090/api/v1/read" read_recent: true

undefined

• job_name: "kubernetes-pods-shard-0" kubernetes_sd_configs:
  • role: pod relabel_configs:

  Hash pod name and keep only shard 0 (mod 3)

  • source_labels: [__meta_kubernetes_pod_name] modulus: 3 target_label: __tmp_hash action: hashmod
  • source_labels: [__tmp_hash] regex: "0" action: keep
• job_name: "kubernetes-pods-shard-1" kubernetes_sd_configs:
  • role: pod relabel_configs:
  • source_labels: [__meta_kubernetes_pod_name] modulus: 3 target_label: __tmp_hash action: hashmod
  • source_labels: [__tmp_hash] regex: "1" action: keep

shard-2 similar pattern...

undefined

Select services to monitor

Define namespaces to search

Endpoint configuration

  # Relabeling
  relabelings:
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: namespace

  # Metric relabeling (filter/modify metrics)
  metricRelabelings:
    - sourceLabels: [__name__]
      regex: "go_.*"
      action: drop # Drop Go runtime metrics
    - sourceLabels: [status]
      regex: "[45].."
      targetLabel: error
      replacement: "true"

Optional: TLS configuration

tlsConfig:

insecureSkipVerify: true

ca:

secret:

name: prometheus-tls

key: ca.crt

undefined

Select pods to monitor

Namespace selection

Pod metrics endpoints

  # Relabeling
  relabelings:
    - sourceLabels: [__meta_kubernetes_pod_label_version]
      targetLabel: version
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node

undefined

Service account for Kubernetes API access

Select ServiceMonitors

Select PodMonitors

Select PrometheusRules

Resource limits

Storage

Retention

Alertmanager configuration

External labels

Security context

Enable admin API for management operations

Additional scrape configs (from Secret)

undefined

• Set appropriate retention period (balance storage vs history needs)
• Configure persistent storage with adequate size
• Enable high availability (multiple Prometheus replicas or federation)
• Set up remote storage for long-term retention (Thanos, Cortex, Mimir)
• Configure service discovery for dynamic environments
• Implement recording rules for frequently-used queries
• Create symptom-based alerts with proper annotations
• Set up Alertmanager with appropriate routing and receivers
• Configure inhibition rules to reduce alert noise
• Add runbook URLs to all critical alerts
• Implement proper label hygiene (avoid high cardinality)
• Monitor Prometheus itself (meta-monitoring)
• Set up authentication and authorization
• Enable TLS for scrape targets and remote storage
• Configure rate limiting for queries
• Test alert and recording rule validity (

  promtool check rules

  )
• Implement backup and disaster recovery procedures
• Document metric naming conventions for the team
• Create dashboards in Grafana for common queries
• Set up log aggregation alongside metrics (Loki)

• name: aggregation_rules rules:

  Instance-level aggregation

  • record: instance:node_cpu_utilization:ratio expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)

  Job-level aggregation

  • record: job:http_requests:rate5m expr: sum(rate(http_requests_total[5m])) by (job)

  Job-level error ratio

  • record: job:http_request_errors:ratio expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job)

  Cluster-level aggregation

  • record: cluster:cpu_utilization:ratio expr: avg(instance:node_cpu_utilization:ratio)

undefined

• Prometheus Documentation
• PromQL Basics
• Best Practices
• Alerting Rules
• Recording Rules
• Prometheus Operator
• Thanos Documentation
• Google SRE Book - Monitoring