Chaos Engineer

Purpose

When to Use

• Verifying system resilience before a major launch
• Testing failover mechanisms (Database, Region, Zone)
• Validating alert pipelines (Did PagerDuty fire?)
• Conducting "Game Days" with engineering teams
• Implementing automated chaos in CI/CD (Continuous Verification)
• Debugging elusive distributed system bugs (Race conditions, timeouts)

2. Decision Framework

Experiment Design Matrix

What are we testing?
│
├─ **Infrastructure Layer**
│  ├─ Pods/Containers? → **Pod Kill / Container Crash**
│  ├─ Nodes? → **Node Drain / Reboot**
│  └─ Network? → **Latency / Packet Loss / Partition**
│
├─ **Application Layer**
│  ├─ Dependencies? → **Block Access to DB/Redis**
│  ├─ Resources? → **CPU/Memory Stress**
│  └─ Logic? → **Inject HTTP 500 / Delays**
│
└─ **Platform Layer**
   ├─ IAM? → **Revoke Keys**
   └─ DNS? → **Block DNS Resolution**

Tool Selection

Blast Radius Control

sre-engineer

• No "Stop Button" mechanism available
• Observability gaps (Blind spots)
• Cascading failure risk identified without mitigation
• Lack of backups for stateful data experiments

4. Core Workflows

Workflow 1: Kubernetes Pod Chaos (Chaos Mesh)

1. Define Experiment (

   backend-kill.yaml

   )
   yaml

   apiVersion: chaos-mesh.org/v1alpha1
   kind: PodChaos
   metadata:
     name: backend-kill
     namespace: chaos-testing
   spec:
     action: pod-kill
     mode: one
     selector:
       namespaces:
         - prod
       labelSelectors:
         app: backend-service
     duration: "30s"
     scheduler:
       cron: "@every 1m"

2. Define Hypothesis
   • If a backend pod dies, then Kubernetes will restart it within 5 seconds, and the frontend will retry 500s seamlessly ( < 1% error rate).
3. Execute & Monitor
   • Apply manifest.
   • Watch Grafana dashboard: "HTTP 500 Rate" vs "Pod Restart Count".
4. Verification
   • Did the pod restart? Yes.
   • Did users see errors? No (Retries worked).
   • Result: PASS.

Workflow 3: Zone Outage Simulation (Game Day)

1. Preparation
   • Notify on-call team (Game Day).
   • Ensure primary DB writes are active.
2. Execution (AWS FIS / Manual)
   • Block network traffic to Zone A subnets.
   • OR Stop RDS Primary instance (Simulate crash).
3. Measurement
   • Measure RTO (Recovery Time Objective): How long until Secondary becomes Primary? (Target: < 60s).
   • Measure RPO (Recovery Point Objective): Any data lost? (Target: 0).

5. Anti-Patterns & Gotchas

❌ Anti-Pattern 1: Testing in Production First

• Running a "delete database" script in prod without testing in staging.

• Catastrophic data loss.
• Resume Generating Event (RGE).

• Dev → Staging → Canary → Prod.
• Verify hypothesis in lower environments first.

❌ Anti-Pattern 2: No Observability

• Running chaos without dashboards open.
• "I think it worked, the app is slow."

• You don't know why it failed.
• You can't prove resilience.

• Observability First: If you can't measure it, don't break it.

❌ Anti-Pattern 3: Random Chaos (Chaos Monkey Style)

• Killing random things constantly without purpose.

• Causes alert fatigue.
• Doesn't test specific failure modes (e.g., network partition vs crash).

• Thoughtful Experiments: Design targeted scenarios (e.g., "What if Redis is slow?"). Random chaos is for maintenance, targeted chaos is for verification.

7. Quality Checklist

• Hypothesis: Clearly defined ("If X happens, Y should occur").
• Blast Radius: Limited (e.g., 1 zone, 1% users).
• Approval: Stakeholders notified (or scheduled Game Day).

• Stop Button: Automated abort script ready.
• Rollback: Plan to restore state if needed.
• Backup: Data backed up before stateful experiments.

• Monitoring: Dashboards visible during experiment.
• Logging: Experiment start/end times logged for correlation.

• Fix: Action items assigned (Jira).
• Report: Findings shared with engineering team.

Examples

Example 1: Kubernetes Pod Failure Recovery

1. Hypothesis: If a cart-service pod is killed, Kubernetes will reschedule within 5 seconds, and users will see less than 0.1% error rate
2. Chaos Injection: Use Chaos Mesh to kill random pods in the production namespace
3. Monitoring: Track error rates, pod restart times, and user-facing failures

• Pod restart time: 3.2 seconds average (within SLA)
• Error rate during experiment: 0.02% (below 0.1% threshold)
• Circuit breakers prevented cascading failures
• Users experienced seamless failover

• Retry logic was working but needed exponential backoff
• Added fallback response for stale cart data
• Created runbook for pod failure scenarios

Example 2: Database Failover Validation

1. Preparation: Notified all stakeholders, backed up current state
2. Primary Zone Blockage: Used AWS FIS to simulate zone failure
3. Failover Trigger: Automated failover initiated when health checks failed
4. Measurement: Tracked RTO, RPO, and application recovery

• DNS TTL was too high (5 minutes), reduced to 30 seconds
• Application connection pooling needed pre-warming
• Added health check for database replication lag

Example 3: Third-Party API Dependency Testing

1. Delay Injection: Using Istio to add 5-10 second delays to payment API calls
2. Timeout Validation: Verify circuit breakers open within configured timeouts
3. Fallback Testing: Ensure users see appropriate error messages

• 50% of requests delayed 10s: Circuit breaker opens, fallback shown
• 100% delay: System degrades gracefully with queue-based processing
• Recovery: System reconnects properly after fault cleared

• Circuit breaker threshold: 5 consecutive failures (needed adjustment)
• Fallback UI: 94% of users completed purchase via alternative method
• Alert tuning: Reduced false positives by tuning latency thresholds

Best Practices

Experiment Design

• Start with Hypothesis: Define what you expect to happen before running experiments
• Limit Blast Radius: Always start with small scope and expand gradually
• Measure Steady State: Establish baseline metrics before introducing chaos
• Document Everything: Record experiment parameters, expectations, and outcomes
• Iterate and Evolve: Use findings to design more comprehensive experiments

Safety and Controls

• Always Have a Stop Button: Can you abort the experiment immediately?
• Define Rollback Plan: How do you restore normal operations?
• Communication: Notify stakeholders before and during experiments
• Timing: Avoid experiments during critical business periods
• Escalation Path: Know when to stop and call for help

Tool Selection

• Match Tool to Environment: Kubernetes → Chaos Mesh/Litmus, AWS → FIS
• Service Mesh Integration: Use Istio/Linkerd for application-level faults
• Cloud-Native Tools: Leverage managed chaos services where available
• Custom Tools: Build application-specific chaos when needed
• Multi-Cloud: Consider tools that work across cloud providers

Observability Integration

• Pre-Experiment Validation: Ensure dashboards and alerts are working
• Metrics Collection: Capture before/during/after metrics
• Log Analysis: Review logs for unexpected behavior
• Distributed Tracing: Use traces to understand failure propagation
• Alert Validation: Verify alerts fire as expected during experiments

Cultural Aspects

• Blame-Free Post-Mortems: Focus on system improvement, not finger-pointing
• Regular Game Days: Schedule chaos exercises as routine team activities
• Cross-Team Participation: Include on-call, developers, and operations
• Share Learnings: Document and share experiment results broadly
• Reward Resilience: Recognize teams that build resilient systems