Resilience Patterns Skill

Overview

This skill provides guidance on implementing resilience patterns in .NET applications. It covers both synchronous resilience (HTTP clients, service calls) using Polly and asynchronous resilience (message handlers) using Brighter.

Key Principle: Design for failure. Systems should gracefully handle transient faults, prevent cascade failures, and provide meaningful fallback behavior.

When to Use This Skill

Keywords: resilience, circuit breaker, retry, polly, brighter, fault tolerance, transient failure, DLQ, dead letter queue, timeout, bulkhead, fallback, http client resilience

Use this skill when:

Implementing HTTP client resilience
Configuring retry policies for transient failures
Setting up circuit breakers to prevent cascade failures
Designing message handler error handling
Implementing dead letter queue patterns
Adding timeout policies to service calls
Configuring bulkhead isolation

Resilience Strategy Overview

Synchronous Resilience (Polly)

For HTTP calls and synchronous service communication:

Pattern	Purpose	When to Use
Retry	Retry failed operations	Transient failures (network, 503, timeouts)
Circuit Breaker	Stop calling failing services	Repeated failures indicate service is down
Timeout	Bound operation time	Prevent indefinite waits
Bulkhead	Isolate failures	Prevent one caller from exhausting resources
Fallback	Provide alternative	Graceful degradation

Asynchronous Resilience (Brighter)

For message-based and async operations:

Pattern	Purpose	When to Use
Retry	Redeliver failed messages	Transient processing failures
Dead Letter Queue	Park unprocessable messages	Poison messages, business rule failures
Circuit Breaker	Stop processing temporarily	Downstream service unavailable
Timeout	Bound handler execution	Prevent handler blocking

Quick Start: Polly v8 with HttpClient

Basic Setup

csharp

// Program.cs or Startup.cs
builder.Services.AddHttpClient<IOrderService, OrderService>()
    .AddStandardResilienceHandler();

The

AddStandardResilienceHandler()

adds a preconfigured pipeline with:

Rate limiter
Total request timeout
Retry (exponential backoff)
Circuit breaker
Attempt timeout

Custom Configuration

csharp

builder.Services.AddHttpClient<IOrderService, OrderService>()
    .AddResilienceHandler("custom-pipeline", builder =>
    {
        // Retry with exponential backoff
        builder.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromSeconds(1),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>()
                .HandleResult(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
        });

        // Circuit breaker
        builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            MinimumThroughput = 10,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(30)
        });

        // Timeout per attempt
        builder.AddTimeout(TimeSpan.FromSeconds(10));
    });

Detailed Polly patterns: See

references/polly-patterns.md

Quick Start: Brighter Message Handler

Basic Retry Policy

csharp

public class OrderCreatedHandler : RequestHandler<OrderCreated>
{
    [UsePolicy("retry-policy", step: 1)]
    public override OrderCreated Handle(OrderCreated command)
    {
        // Process order
        return base.Handle(command);
    }
}

Policy Registry Setup

csharp

var policyRegistry = new PolicyRegistry
{
    {
        "retry-policy",
        Policy
            .Handle<Exception>()
            .WaitAndRetry(
                retryCount: 3,
                sleepDurationProvider: attempt =>
                    TimeSpan.FromSeconds(Math.Pow(2, attempt)))
    }
};

services.AddBrighter()
    .UseExternalBus(/* config */)
    .UsePolicyRegistry(policyRegistry);

Detailed Brighter patterns: See

references/brighter-resilience.md

Pattern Decision Tree

When to Use Retry

Use retry when:

Failure is likely transient (network blip, temporary 503)
Operation is idempotent
Delay between retries is acceptable

Don't use retry when:

Failure is business logic (validation error, 400 Bad Request)
Operation is not idempotent (unless with idempotency key)
Immediate response required

When to Use Circuit Breaker

Use circuit breaker when:

Calling external services that might be down
Need to fail fast instead of waiting
Want to prevent cascade failures
Service recovery needs time

Configuration guidance: See

references/circuit-breaker-config.md

When to Use DLQ

Use DLQ when:

Message cannot be processed after max retries
Business rule prevents processing
Manual intervention needed
Audit trail required for failures

DLQ patterns: See

references/dlq-patterns.md

Retry Strategy Patterns

Immediate Retry

For very transient failures:

csharp

.AddRetry(new RetryStrategyOptions
{
    MaxRetryAttempts = 2,
    Delay = TimeSpan.Zero  // Immediate retry
});

Exponential Backoff

For transient failures that need time:

csharp

.AddRetry(new RetryStrategyOptions
{
    MaxRetryAttempts = 4,
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Exponential,
    UseJitter = true  // Prevents thundering herd
});

Delays: 1s → 2s → 4s → 8s (with jitter)

Linear Backoff

For rate-limited services:

csharp

.AddRetry(new RetryStrategyOptions
{
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromSeconds(2),
    BackoffType = DelayBackoffType.Linear
});

Delays: 2s → 4s → 6s

Full retry strategies: See

references/retry-strategies.md

Circuit Breaker Configuration

Conservative (Sensitive Service)

csharp

.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.25,        // Open after 25% failures
    MinimumThroughput = 5,       // Need at least 5 calls to evaluate
    SamplingDuration = TimeSpan.FromSeconds(10),
    BreakDuration = TimeSpan.FromSeconds(60)  // Stay open 60s
});

Aggressive (High Availability)

csharp

.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,          // Open after 50% failures
    MinimumThroughput = 20,      // Need 20 calls before evaluation
    SamplingDuration = TimeSpan.FromSeconds(30),
    BreakDuration = TimeSpan.FromSeconds(15)  // Quick recovery attempt
});

Detailed configuration: See

references/circuit-breaker-config.md

Dead Letter Queue Pattern

When Message Processing Fails

text

1. Message received
2. Handler attempts processing
3. Failure occurs
4. Retry policy applied (1...N attempts)
5. All retries exhausted
6. Message moved to DLQ
7. Alert/monitoring triggered
8. Manual investigation

Brighter DLQ Setup

csharp

services.AddBrighter()
    .UseExternalBus(config =>
    {
        config.Publication.RequeueDelayInMs = 500;
        config.Publication.RequeueCount = 3;
        // After 3 requeues, message goes to DLQ
    });

Full DLQ patterns: See

references/dlq-patterns.md

Combined Patterns

HTTP Client with Full Resilience

csharp

builder.Services.AddHttpClient<IPaymentGateway, PaymentGateway>()
    .AddResilienceHandler("payment-gateway", builder =>
    {
        // Order matters: outer to inner

        // 1. Total timeout (outer boundary)
        builder.AddTimeout(TimeSpan.FromSeconds(30));

        // 2. Retry (with circuit breaker inside)
        builder.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromMilliseconds(500),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true
        });

        // 3. Circuit breaker
        builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            MinimumThroughput = 10,
            BreakDuration = TimeSpan.FromSeconds(30)
        });

        // 4. Per-attempt timeout (inner)
        builder.AddTimeout(TimeSpan.FromSeconds(5));
    });

Message Handler with Fallback

csharp

public class ProcessPaymentHandler : RequestHandler<ProcessPayment>
{
    [UsePolicy("circuit-breaker", step: 1)]
    [UsePolicy("retry", step: 2)]
    [UsePolicy("fallback", step: 3)]
    public override ProcessPayment Handle(ProcessPayment command)
    {
        _paymentService.Process(command);
        return base.Handle(command);
    }
}

Observability

Polly Telemetry

csharp

services.AddResiliencePipeline("my-pipeline", builder =>
{
    builder.AddRetry(/* options */)
        .ConfigureTelemetry(LoggerFactory.Create(b => b.AddConsole()));
});

Key Metrics to Monitor

Metric	Purpose	Alert Threshold
Retry count	Track transient failures	> 3 per minute
Circuit state	Track service health	State = Open
DLQ depth	Track processing failures	> 0
Timeout rate	Track slow services	> 5%

Anti-Patterns

Over-Retrying

Problem: Retrying too many times, too quickly.

csharp

// BAD: 10 immediate retries
.AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 10 });

Fix: Use exponential backoff, limit retries:

csharp

// GOOD: 3 retries with backoff
.AddRetry(new RetryStrategyOptions
{
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Exponential
});

Retrying Non-Transient Failures

Problem: Retrying business logic failures.

csharp

// BAD: Retrying 400 Bad Request
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
    .HandleResult(r => !r.IsSuccessStatusCode)

Fix: Only retry transient failures:

csharp

// GOOD: Only retry transient HTTP codes
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
    .Handle<HttpRequestException>()
    .HandleResult(r => r.StatusCode is
        HttpStatusCode.ServiceUnavailable or
        HttpStatusCode.GatewayTimeout or
        HttpStatusCode.RequestTimeout)

Missing Circuit Breaker

Problem: Retrying endlessly when service is down.

Fix: Always pair retry with circuit breaker for external calls.

DLQ as Black Hole

Problem: Messages go to DLQ and are never processed.

Fix:

Monitor DLQ depth
Set up alerts
Implement replay mechanism
Document investigation procedures

References

```
references/polly-patterns.md
```
- Comprehensive Polly v8 patterns
```
references/circuit-breaker-config.md
```
- Circuit breaker configuration guide
```
references/retry-strategies.md
```
- Retry strategy patterns
```
references/brighter-resilience.md
```
- Brighter message handler resilience
```
references/dlq-patterns.md
```
- Dead letter queue patterns

Related Skills

```
fitness-functions
```
- Test resilience with performance fitness functions
```
modular-architecture
```
- Isolate resilience concerns by module
```
adr-management
```
- Document resilience decisions

Last Updated: 2025-12-22

Version History

v1.0.0 (2025-12-26): Initial release

resilience-patterns

NPX Install

Tags

SKILL.md Content

Resilience Patterns Skill

Overview

When to Use This Skill

Resilience Strategy Overview

Synchronous Resilience (Polly)

Asynchronous Resilience (Brighter)

Quick Start: Polly v8 with HttpClient

Basic Setup

Custom Configuration

Quick Start: Brighter Message Handler

Basic Retry Policy

Policy Registry Setup

Pattern Decision Tree

When to Use Retry

When to Use Circuit Breaker

When to Use DLQ

Retry Strategy Patterns

Immediate Retry

Exponential Backoff

Linear Backoff

Circuit Breaker Configuration

Conservative (Sensitive Service)

Aggressive (High Availability)

Dead Letter Queue Pattern

When Message Processing Fails

Brighter DLQ Setup

Combined Patterns

HTTP Client with Full Resilience

Message Handler with Fallback

Observability

Polly Telemetry

Key Metrics to Monitor

Anti-Patterns

Over-Retrying

Retrying Non-Transient Failures

Missing Circuit Breaker

DLQ as Black Hole

References

Related Skills

Version History