chaos-engineering

Experiment: Database failover
Steady state:
  - Error rate: < 0.1%
  - P95 latency: < 300ms
  - Successful orders per minute: > 50

Hypothesis: When the primary database fails over to the replica,
  - Error rate will spike to < 2% for < 30 seconds
  - P95 latency will increase to < 1s for < 60 seconds
  - No orders will be permanently lost
  - The application will recover without manual intervention

Injection:
  Target: primary database (PostgreSQL)
  Method: block TCP port 5432 on the primary instance
  Scope: single database instance
  Duration: 60 seconds
  Blast radius: staging environment only (first run)

  Abort conditions:
    - Error rate > 10% for > 2 minutes
    - Any data corruption detected
    - Manual abort by experiment owner

Observation assignments:
  - Engineer A: application error rate and latency dashboard
  - Engineer B: database metrics (connections, replication lag, failover status)
  - Engineer C: application logs (search for database connection errors)
  - Engineer D: business metrics (order count, payment processing)

Analysis checklist:
  - Did the system behave as hypothesized? (Y/N, with details)
  - How long was the impact? (Expected vs. actual duration)
  - Were any errors visible to users?
  - Was any data lost or corrupted?
  - Did monitoring and alerting detect the problem correctly?
  - How long before alerts fired?
  - What was the recovery time?

Findings document:
  Experiment: Database failover (2026-03-20)
  Hypothesis: Confirmed / Partially confirmed / Disproved
  Recovery time: 45s (expected vs actual: expected <10s, actual 45s)
  Data integrity: no rows lost; 3 writes returned 500 instead of queueing

  Findings:
    - Connection pool did not detect stale connections for 45 seconds (expected: <10s)
    - Retry logic worked correctly for read operations
    - Write operations returned 500 errors for 38 seconds (expected: queued)

  Action items (every one has an owner and a due date — no item is deferred):
    - [ ] Configure connection pool health checks — assigned to @maria, due 2026-03-31
    - [ ] Implement write queue with 5-minute buffer — assigned to @dan, due 2026-04-02
    - [ ] Re-run experiment after fixes deployed (re-run scheduled 2026-04-03);
          specific metrics to check on re-run: stale-connection detection <10s,
          zero write 500s, error rate <2%

Decision tree:
  Running on Kubernetes?
    → Cloud-managed AWS workloads: AWS FIS (cloud-native, IAM-integrated)
    → On K8s with sidecar tolerance: Chaos Mesh (eBPF, fine-grained)
    → On K8s wanting workflows + UI: LitmusChaos (ChaosCenter, Workflows)
    → On K8s lightweight: kube-monkey

  Running on plain VMs / Docker?
    → Docker only: Pumba
    → Linux: tc + stress-ng (manual)

  Need network fault injection in integration tests?
    → toxiproxy (lightweight, programmatic API)

  Need to combine load testing with chaos?
    → k6 with xk6-disruptor extension

  Need managed platform with UI and compliance?
    → Gremlin or Steadybit (both commercial)

2 weeks before:
  - [ ] Define 2-3 experiments to run (don't overload the schedule)
  - [ ] Write hypotheses for each experiment
  - [ ] Get approval from engineering leadership and affected teams
  - [ ] Notify support team and stakeholders
  - [ ] Verify monitoring and alerting are working
  - [ ] Identify rollback procedures for each experiment
  - [ ] Schedule 3-4 hour block (experiments + analysis + retro)

1 day before:
  - [ ] Confirm all participants and their roles
  - [ ] Test that fault injection tools work in the target environment
  - [ ] Verify rollback procedures work (dry run)
  - [ ] Prepare dashboards and observation assignments
  - [ ] Brief the on-call team
  - [ ] Confirm abort criteria for each experiment

Hypothesis: When database latency increases by 500ms, the application
will remain functional with response times under 3 seconds.

Injection: Add 500ms latency to the database connection using toxiproxy.
Duration: 5 minutes.
Environment: staging.

What to observe:
  - Application response times (should increase by ~500ms, not 10x)
  - Connection pool behavior (should not exhaust connections)
  - Timeout handling (requests should not hang indefinitely)
  - Circuit breaker activation (if implemented)
  - Cache effectiveness (cached reads should be unaffected)

Hypothesis: When the payment provider returns 500 errors, the
application will show a user-friendly error message and allow
retry without duplicate charges.

Injection: Configure mock/proxy to return 500 for payment API calls.
Duration: 10 minutes.
Environment: staging.

What to observe:
  - Error message quality (user-friendly, not stack traces)
  - Retry behavior (does the application retry? How many times?)
  - Idempotency (retries don't create duplicate transactions)
  - Fallback (is there an alternative payment path?)
  - Monitoring (does the payment failure show up in alerts?)

Hypothesis: When Redis becomes unavailable, the application will fall
back to direct database queries with degraded but functional performance.

Injection: Block Redis port using toxiproxy or iptables.
Duration: 5 minutes.
Environment: staging.

What to observe:
  - Database query volume (should increase but not overwhelm)
  - Response times (should increase but remain under 5 seconds)
  - Error rate (cache miss should not cause errors)
  - Connection pool (database connections should not exhaust)
  - Recovery (when cache returns, does the application resume normal behavior?)