LLM Architect

Purpose

When to Use

• Designing end-to-end LLM systems from requirements to production
• Selecting models and serving infrastructure for specific use cases
• Implementing RAG (Retrieval Augmented Generation) pipelines
• Optimizing LLM costs while maintaining quality thresholds
• Building safety guardrails and compliance mechanisms
• Planning fine-tuning vs RAG vs prompt engineering strategies
• Scaling LLM inference for high-throughput applications

Quick Start

• Designing end-to-end LLM systems from requirements to production
• Selecting models and serving infrastructure for specific use cases
• Implementing RAG (Retrieval Augmented Generation) pipelines
• Optimizing LLM costs while maintaining quality thresholds
• Building safety guardrails and compliance mechanisms

• Simple API integration exists (use backend-developer instead)
• Only prompt engineering needed without architecture decisions
• Training foundation models from scratch (almost always wrong approach)
• Generic ML tasks unrelated to language models (use ml-engineer)

Decision Framework

Model Selection Quick Guide

RAG vs Fine-Tuning Decision Tree

Need to customize LLM behavior?
│
├─ Need domain-specific knowledge?
│  ├─ Knowledge changes frequently?
│  │  └─ RAG (Retrieval Augmented Generation)
│  └─ Knowledge is static?
│     └─ Fine-tuning OR RAG (test both)
│
├─ Need specific output format/style?
│  ├─ Can describe in prompt?
│  │  └─ Prompt engineering (try first)
│  └─ Format too complex for prompt?
│     └─ Fine-tuning
│
└─ Need latency <100ms?
   └─ Fine-tuned small model (7B-13B)

Architecture Pattern

[Client] → [API Gateway + Rate Limiting]
              ↓
         [Request Router]
          (Route by intent/complexity)
              ↓
    ┌────────┴────────┐
    ↓                 ↓
[Fast Model]    [Powerful Model]
(Haiku/Small)   (Sonnet/Large)
    ↓                 ↓
[Cache Layer] ← [Response Aggregator]
    ↓
[Logging & Monitoring]
    ↓
[Response to Client]

Core Workflow: Design LLM System

1. Requirements Gathering

• Latency: What's the P95 response time requirement?
• Scale: Expected requests/day and growth trajectory?
• Accuracy: What's the minimum acceptable quality? (measurable metric)
• Cost: Budget constraints? ($/request or $/month)
• Data: Existing datasets for evaluation? Sensitivity level?
• Compliance: Regulatory requirements? (HIPAA, GDPR, SOC2, etc.)

2. Model Selection

def select_model(requirements):
    if requirements.latency_p95 < 100:  # milliseconds
        if requirements.task_complexity == "simple":
            return "llama2-7b-finetune"
        else:
            return "mistral-7b-quantized"
    
    elif requirements.latency_p95 < 2000:
        if requirements.budget == "unlimited":
            return "claude-3-opus"
        elif requirements.domain_specific:
            return "claude-3-sonnet-finetuned"
        else:
            return "claude-3-haiku"
    
    else:  # Batch/async acceptable
        if requirements.accuracy_critical:
            return "gpt-4-with-ensemble"
        else:
            return "batch-api-cheapest-tier"

3. Prototype & Evaluate

# Run benchmark on eval dataset
python scripts/evaluate_model.py \
  --model claude-3-sonnet \
  --dataset data/eval_1000_examples.jsonl \
  --metrics accuracy,latency,cost

# Expected output:
# Accuracy: 94.3%
# P95 Latency: 1,245ms
# Cost per 1K requests: $2.15

4. Iteration Checklist

• Latency P95 meets requirement? If no → optimize serving (quantization, caching)
• Accuracy meets threshold? If no → improve prompts, fine-tune, or upgrade model
• Cost within budget? If no → aggressive caching, smaller model routing, batching
• Safety guardrails tested? If no → add content filters, PII detection
• Monitoring dashboards live? If no → set up Prometheus + Grafana
• Runbook documented? If no → document common failures and fixes

Cost Optimization Strategies

Safety Checklist

• Content filtering tested against adversarial examples
• PII detection and redaction validated
• Prompt injection defenses in place
• Output validation rules implemented
• Audit logging configured for all requests
• Compliance requirements documented and validated

Red Flags - When to Escalate

Quick Reference: Performance Targets

Additional Resources

• Detailed Technical Reference: See REFERENCE.md
  • RAG implementation workflow
  • Semantic caching patterns
  • Deployment configurations
• Code Examples & Patterns: See EXAMPLES.md
  • Anti-patterns (fine-tuning when prompting suffices, no fallback)
  • Quality checklist for LLM systems
  • Resilient LLM call patterns