Gamma Incident Runbook

Overview

Prerequisites

• Access to monitoring dashboards
• Access to application logs
• On-call responsibilities defined
• Communication channels established

Incident Severity Levels

Quick Diagnostics

Step 1: Check Gamma Status

# Check Gamma status page
curl -s https://status.gamma.app/api/v2/status.json | jq '.status'

# Check our integration health
curl -s https://your-app.com/health/gamma | jq '.'

# Quick connectivity test
curl -w "\nTime: %{time_total}s\n" \
  -H "Authorization: Bearer $GAMMA_API_KEY" \
  https://api.gamma.app/v1/ping

Step 2: Review Key Metrics

# Check error rate (Prometheus)
curl -s 'http://prometheus:9090/api/v1/query?query=rate(gamma_requests_total{status=~"5.."}[5m])' | jq '.data.result'

# Check latency P95
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,rate(gamma_request_duration_seconds_bucket[5m]))' | jq '.data.result'

# Check rate limit
curl -s 'http://prometheus:9090/api/v1/query?query=gamma_rate_limit_remaining' | jq '.data.result'

Step 3: Review Recent Logs

# Last 100 error logs
grep -i "gamma.*error" /var/log/app/gamma-*.log | tail -100

# Rate limit hits
grep "429" /var/log/app/gamma-*.log | wc -l

# Timeout errors
grep -i "timeout" /var/log/app/gamma-*.log | tail -50

Incident Response Procedures

Scenario 1: API Returning 5xx Errors

• High error rate in monitoring
• Users reporting failed presentations
• 500/502/503 responses from Gamma

1. Verify Gamma status: https://status.gamma.app
2. If Gamma outage confirmed:
   • Enable degraded mode / show maintenance message
   • Monitor status page for updates
   • No action needed on our side
3. If Gamma is operational:
   bash

   # Check our request patterns
   grep "5[0-9][0-9]" /var/log/app/gamma-*.log | \
     awk '{print $1}' | sort | uniq -c | sort -rn
   
   # Look for malformed requests
   grep -B5 "500" /var/log/app/gamma-*.log | grep "request"

4. Rollback recent deployments if issue correlates

Scenario 2: Rate Limit Exceeded (429)

• 429 responses in logs
• Rate limit metrics at zero
• Slow or queued requests

1. Immediate mitigation:
   bash

   # Enable request throttling
   curl -X POST http://localhost:8080/admin/throttle \
     -d '{"gamma": {"rps": 10}}'

2. Check for runaway processes:
   bash

   # Find high-volume clients
   grep "gamma" /var/log/app/*.log | \
     awk '{print $5}' | sort | uniq -c | sort -rn | head -20

3. Enable circuit breaker:
   bash

   curl -X POST http://localhost:8080/admin/circuit-breaker \
     -d '{"service": "gamma", "state": "open"}'

4. Long-term: Review rate limit tier with Gamma

Scenario 3: High Latency

• Slow presentation creation
• Timeouts in logs
• P95 latency > 10s

1. Check Gamma latency vs our latency:
   bash

   # Direct Gamma latency
   for i in {1..5}; do
     curl -w "%{time_total}\n" -o /dev/null -s \
       -H "Authorization: Bearer $GAMMA_API_KEY" \
       https://api.gamma.app/v1/ping
   done

2. If Gamma is slow:
   • Increase timeouts temporarily
   • Enable async mode for non-critical operations
   • Queue heavy operations
3. If our infrastructure is slow:
   • Check CPU/memory on app servers
   • Review connection pool settings
   • Check network connectivity

Scenario 4: Authentication Failures (401/403)

• All requests failing with 401
• "Invalid API key" errors
• Sudden authentication failures

1. Verify API key:
   bash

   # Test key directly
   curl -H "Authorization: Bearer $GAMMA_API_KEY" \
     https://api.gamma.app/v1/ping
   
   # Check key format
   echo $GAMMA_API_KEY | head -c 20

2. If key is invalid:
   • Check if key was rotated
   • Deploy backup key:

     GAMMA_API_KEY_SECONDARY

   • Generate new key in Gamma dashboard
3. Notify team and update secrets

Communication Templates

Internal Notification

INCIDENT: Gamma Integration Issue

Severity: P[X]
Status: Investigating / Identified / Mitigating / Resolved
Impact: [Description of user impact]
Start Time: [ISO timestamp]

Summary: [Brief description]

Current Actions:
- [Action 1]
- [Action 2]

Next Update: [Time]

User-Facing Message

We're currently experiencing issues with presentation generation.
Our team is actively working to resolve this.

Workaround: [If available]
Status updates: [Link to status page]
ETA: [If known]

Post-Incident Checklist

• Incident timeline documented
• Root cause identified
• User impact quantified
• Fix verified in production
• Monitoring gaps identified
• Preventive measures documented
• Post-mortem scheduled (for P1/P2)

Resources

• Gamma Status
• Gamma Support
• Internal Runbook Wiki
• On-Call Schedule

Next Steps

gamma-data-handling