Resilience Patterns Skill

Production-grade resilience patterns for distributed systems and LLM-based workflows. Covers circuit breakers, bulkheads, retry strategies, and LLM-specific resilience techniques.

Overview

Building fault-tolerant multi-agent systems
Implementing LLM API integrations with proper error handling
Designing distributed workflows that need graceful degradation
Adding observability to failure scenarios
Protecting systems from cascade failures

Core Patterns

1. Circuit Breaker Pattern (reference: circuit-breaker.md)

Prevents cascade failures by "tripping" when a service exceeds failure thresholds.

+-------------------------------------------------------------------+
|                    Circuit Breaker States                         |
+-------------------------------------------------------------------+
|                                                                   |
|    +----------+     failures >= threshold    +----------+         |
|    |  CLOSED  | ----------------------------> |   OPEN   |        |
|    | (normal) |                              | (reject) |         |
|    +----+-----+                              +----+-----+         |
|         |                                         |               |
|         | success                    timeout      |               |
|         |                            expires      |               |
|         |         +------------+                  |               |
|         |         | HALF_OPEN  |<-----------------+               |
|         +---------+  (probe)   |                                  |
|                   +------------+                                  |
|                                                                   |
|   CLOSED:    Allow requests, count failures                       |
|   OPEN:      Reject immediately, return fallback                  |
|   HALF_OPEN: Allow probe request to test recovery                 |
|                                                                   |
+-------------------------------------------------------------------+

Key Configuration:

```
failure_threshold
```
: Failures before opening (default: 5)
```
recovery_timeout
```
: Seconds before attempting recovery (default: 30)
```
half_open_requests
```
: Probes to allow in half-open (default: 1)

2. Bulkhead Pattern (reference: bulkhead-pattern.md)

Isolates failures by partitioning resources into independent pools.

+-------------------------------------------------------------------+
|                      Bulkhead Isolation                           |
+-------------------------------------------------------------------+
|                                                                   |
|   +------------------+  +------------------+                      |
|   | TIER 1: Critical |  | TIER 2: Standard |                      |
|   |  (5 workers)     |  |  (3 workers)     |                      |
|   |  +-+ +-+ +-+     |  |  +-+ +-+ +-+     |                      |
|   |  |#| |#| | |     |  |  |#| | | | |     |                      |
|   |  +-+ +-+ +-+     |  |  +-+ +-+ +-+     |                      |
|   |  +-+ +-+         |  |                  |                      |
|   |  | | | |         |  |  Queue: 2        |                      |
|   |  +-+ +-+         |  |                  |                      |
|   |  Queue: 0        |  +------------------+                      |
|   +------------------+                                            |
|                                                                   |
|   +------------------+                                            |
|   | TIER 3: Optional |   # = Active request                       |
|   |  (2 workers)     |     = Available slot                       |
|   |  +-+ +-+         |                                            |
|   |  |#| |#| FULL!   |   Tier 1: synthesis, quality_gate          |
|   |  +-+ +-+         |   Tier 2: analysis agents                  |
|   |  Queue: 5        |   Tier 3: enrichment, optional features    |
|   +------------------+                                            |
|                                                                   |
+-------------------------------------------------------------------+

Tier Configuration (OrchestKit):

Tier	Workers	Queue	Timeout	Use Case
1 (Critical)	5	10	300s	Synthesis, quality gate
2 (Standard)	3	5	120s	Content analysis agents
3 (Optional)	2	3	60s	Enrichment, caching

3. Retry Strategies (reference: retry-strategies.md)

Intelligent retry logic with exponential backoff and jitter.

+-------------------------------------------------------------------+
|                   Exponential Backoff + Jitter                    |
+-------------------------------------------------------------------+
|                                                                   |
|   Attempt 1:  --> X (fail)                                        |
|               wait: 1s +/- 0.5s                                   |
|                                                                   |
|   Attempt 2:  --> X (fail)                                        |
|               wait: 2s +/- 1s                                     |
|                                                                   |
|   Attempt 3:  --> X (fail)                                        |
|               wait: 4s +/- 2s                                     |
|                                                                   |
|   Attempt 4:  --> OK (success)                                    |
|                                                                   |
|   Formula: delay = min(base * 2^attempt, max_delay) * jitter      |
|   Jitter:  random(0.5, 1.5) to prevent thundering herd            |
|                                                                   |
+-------------------------------------------------------------------+

Error Classification for Retries:

python

RETRYABLE_ERRORS = {
    # HTTP/Network
    408, 429, 500, 502, 503, 504,  # HTTP status codes
    ConnectionError, TimeoutError,  # Network errors

    # LLM-specific
    "rate_limit_exceeded",
    "model_overloaded",
    "context_length_exceeded",  # Retry with truncation
}

NON_RETRYABLE_ERRORS = {
    400, 401, 403, 404,  # Client errors
    "invalid_api_key",
    "content_policy_violation",
    "invalid_request_error",
}

4. LLM-Specific Resilience (reference: llm-resilience.md)

Patterns specific to LLM API integrations.

+-------------------------------------------------------------------+
|                    LLM Fallback Chain                             |
+-------------------------------------------------------------------+
|                                                                   |
|   Request --> [Primary Model] --success--> Response               |
|                     |                                             |
|                   fail                                            |
|                     v                                             |
|               [Fallback Model] --success--> Response              |
|                     |                                             |
|                   fail                                            |
|                     v                                             |
|               [Cached Response] --hit--> Response                 |
|                     |                                             |
|                   miss                                            |
|                     v                                             |
|               [Default Response] --> Graceful Degradation         |
|                                                                   |
|   Example Chain:                                                  |
|   1. claude-sonnet-4-5-20251101 (primary)                         |
|   2. gpt-5.2-mini (fallback)                                      |
|   3. Semantic cache lookup                                        |
|   4. "Analysis unavailable" + partial results                     |
|                                                                   |
+-------------------------------------------------------------------+

Token Budget Management:

+-------------------------------------------------------------------+
|                     Token Budget Guard                            |
+-------------------------------------------------------------------+
|                                                                   |
|   Input: 8,000 tokens                                             |
|   +---------------------------------------------+                 |
|   |#################################            |                 |
|   +---------------------------------------------+                 |
|                                          ^                        |
|                                          |                        |
|                                    Context Limit (16K)            |
|                                                                   |
|   Strategy when approaching limit:                                |
|   1. Summarize earlier context (compress 4:1)                     |
|   2. Drop low-priority content (optional fields)                  |
|   3. Split into multiple requests                                 |
|   4. Fail fast with "content too large" error                     |
|                                                                   |
+-------------------------------------------------------------------+

Quick Reference

Pattern	When to Use	Key Benefit
Circuit Breaker	External service calls	Prevent cascade failures
Bulkhead	Multi-tenant/multi-agent	Isolate failures
Retry + Backoff	Transient failures	Automatic recovery
Fallback Chain	Critical operations	Graceful degradation
Token Budget	LLM calls	Cost control, prevent failures

OrchestKit Integration Points

Workflow Agents: Each agent wrapped with circuit breaker + bulkhead tier
LLM Calls: All model invocations use fallback chain + retry logic
External APIs: Circuit breaker on YouTube, arXiv, GitHub APIs
Database Ops: Bulkhead isolation for read vs write operations

Files in This Skill

References (Conceptual Guides)

```
references/circuit-breaker.md
```
- Deep dive on circuit breaker pattern
```
references/bulkhead-pattern.md
```
- Bulkhead isolation strategies
```
references/retry-strategies.md
```
- Retry algorithms and error classification
```
references/llm-resilience.md
```
- LLM-specific patterns
```
references/error-classification.md
```
- How to categorize errors

Templates (Code Patterns)

```
scripts/circuit-breaker.py
```
- Ready-to-use circuit breaker class
```
scripts/bulkhead.py
```
- Semaphore-based bulkhead implementation
```
scripts/retry-handler.py
```
- Configurable retry decorator
```
scripts/llm-fallback-chain.py
```
- Multi-model fallback pattern
```
scripts/token-budget.py
```
- Token budget guard implementation

Examples

examples/orchestkit-workflow-resilience.md

- Full OrchestKit integration example

Checklists

```
checklists/pre-deployment-resilience.md
```
- Production readiness checklist
```
checklists/circuit-breaker-setup.md
```
- Circuit breaker configuration guide

2026 Best Practices

Adaptive Thresholds: Use sliding windows, not fixed counters
Observability First: Every circuit trip = alert + metric + trace
Graceful Degradation: Always have a fallback, even if partial
Health Endpoints: Separate health check from circuit state
Chaos Testing: Regularly test failure scenarios in staging

Related Skills

```
observability-monitoring
```
- Metrics and alerting for circuit breaker state changes
```
caching-strategies
```
- Cache as fallback layer in degradation scenarios
```
error-handling-rfc9457
```
- Structured error responses for resilience failures
```
background-jobs
```
- Async processing with retry and failure handling

Key Decisions

Decision	Choice	Rationale
Circuit breaker recovery	Half-open probe	Gradual recovery, prevents immediate re-failure
Retry algorithm	Exponential backoff + jitter	Prevents thundering herd, respects rate limits
Bulkhead isolation	Semaphore-based tiers	Simple, efficient, prioritizes critical operations
LLM fallback	Model chain with cache	Graceful degradation, cost optimization, availability

Capability Details

circuit-breaker

Keywords: circuit breaker, failure threshold, cascade failure, trip, half-open Solves:

Prevent cascade failures when external services fail
Automatically recover when services come back online
Fail fast instead of waiting for timeouts

bulkhead

Keywords: bulkhead, isolation, semaphore, thread pool, resource pool, tier Solves:

Isolate failures to prevent entire system crashes
Prioritize critical operations over optional ones
Limit concurrent requests to protect resources

retry-strategies

Keywords: retry, backoff, exponential, jitter, thundering herd Solves:

Handle transient failures automatically
Avoid overwhelming recovering services
Classify errors as retryable vs non-retryable

llm-resilience

Keywords: LLM, fallback, model, token budget, rate limit, context length Solves:

Handle LLM API rate limits gracefully
Fall back to alternative models when primary fails
Manage token budgets to prevent context overflow

error-classification

Keywords: error, retryable, transient, permanent, classification Solves:

Determine which errors should be retried
Categorize errors by severity and recoverability
Map HTTP status codes to resilience actions

resilience-patterns

NPX Install

Tags

SKILL.md Content