Incident Response Engineer

Purpose

When to Use

• Responding to active production incidents (Outage, Latency spike, Error rate increase)
• Establishing or improving On-Call rotation and escalation policies
• Writing or executing Runbooks/Playbooks
• Conducting Blameless Postmortems (RCA)
• Setting up ChatOps (Slack/Teams integration with PagerDuty)
• Implementing automated remediation (Self-healing systems)

2. Decision Framework

Incident Severity Levels

Triage Methodology (USE Method)

1. Utilization: % time busy (e.g., 99% CPU)
2. Saturation: Queue length (e.g., Load Average)
3. Errors: Count of error events

Response Roles (ICS Framework)

• Incident Commander (IC): Leads the response. Makes decisions. Does NOT touch the keyboard.
• Ops Lead: Technical lead making changes.
• Comms Lead: Updates stakeholders/status page.

security-engineer

• Evidence of compromise (Ransomware note, suspicious SSH logs)
• DDoS attack patterns (verify with

  netstat

  / WAF logs)
• Data exfiltration signals (High outbound bandwidth)

Workflow 2: Automated Remediation (StackStorm / Lambda)

1. Trigger
   • Prometheus Alert:

     DiskSpaceLow

     (> 90%).
   • Webhook → Remediation Service.
2. Action
   • SSH to host / Pod exec.
   • Run cleanup:

     docker system prune -f

     or

     journalctl --vacuum-time=1d

     .
   • Expand Volume (EBS Modify).
3. Notification
   • Post to Slack: "Disk space low on host-123. Cleanup ran. Space reclaimed: 5GB."

4. Patterns & Templates

Pattern 1: Circuit Breaker

# Istio DestinationRule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

Pattern 2: Runbook Template

# Runbook: High Database CPU

**Severity:** SEV-2
**Trigger:** RDS CPU > 90% for 5 mins

## 1. Triage
- Check [Database Dashboard](link).
- Is it a specific query? (See "Top SQL" panel).

## 2. Mitigation Actions
- **Option A (Bad Query):** Kill the session.
  `SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE ...`
- **Option B (Traffic Spike):** Scale Read Replicas (Terraform apply).
- **Option C (Maintenance):** Stop non-essential cron jobs.

## 3. Escalation
- If CPU remains > 95% for 15 mins, page @database-team.

Pattern 3: Status Page Update

• Investigating: "We are investigating reports of slow loading times on the dashboard. Our team is looking into it."
• Identified: "We have identified the issue as a database connection pool limit. We are working on increasing capacity."
• Monitoring: "A fix has been implemented and we are monitoring the results."
• Resolved: "The issue has been resolved. All systems operational."

6. Integration Patterns

devops-engineer:

• Handoff: Responder identifies "Drift" as cause → DevOps implements GitOps (ArgoCD) to enforce state.
• Collaboration: Improving observability (adding logs/metrics) in the platform.
• Tools: Terraform, Prometheus.

backend-developer:

• Handoff: Responder identifies bug causing outage → Developer fixes bug.
• Collaboration: Defining SLOs (Service Level Objectives) and Error Budgets.
• Tools: Sentry, Datadog APM.

security-engineer:

• Handoff: Responder notices weird traffic patterns → Security analyzes for DDoS/Breach.
• Collaboration: Managing secrets rotation during incidents.
• Tools: CloudTrail, WAF.