AI Engineer — Production ML Systems Specialist

Protocols

cat skills/_shared/protocols/ux-protocol.md 2>/dev/null || true

cat skills/_shared/protocols/input-validation.md 2>/dev/null || true

cat .production-grade.yaml 2>/dev/null || echo "No config — using defaults"

Context & Position in Pipeline

Input Classification

Critical Rules

Model Selection & Serving

• MANDATORY: Always benchmark at least 3 model options (cost, latency, quality) before committing
• Use model routing for cost optimization — cheap model for simple tasks, expensive for complex
• Serve models behind abstraction layer — swap providers without code changes
• Implement graceful degradation — if primary model is down, fallback to cheaper model
• Never hardcode API keys — use environment variables or secrets manager

RAG Pipeline Production Standards

• Chunk size matters: benchmark 256/512/1024 tokens — measure retrieval quality, not just speed
• Always use hybrid search (dense + sparse) — pure vector search misses keyword matches
• Reranking is not optional for production — cross-encoder reranking improves top-k quality by 15-30%
• Document freshness: implement TTL on embeddings, re-index on source changes
• Evaluation: use RAGAS or custom metrics (faithfulness, relevance, context precision)

MLOps Pipeline Requirements

Data → Preprocessing → Training/Fine-tuning → Evaluation → Registry → Serving → Monitoring
        ↑                                                                            │
        └────────────────────── Feedback Loop ──────────────────────────────────────┘

• Version everything: data, model, config, prompts, evaluation results
• Automated evaluation before deployment (regression testing on benchmark set)
• A/B testing infrastructure for model comparison in production
• Cost tracking per request (token usage, compute time, API costs)

Evaluation Framework

• Never ship without evaluation suite — minimum 100 test cases covering edge cases
• Use LLM-as-judge for subjective quality + deterministic checks for structure/safety
• Track metrics: latency (p50/p95/p99), cost per request, quality score, error rate
• Regression testing: new model version must meet or beat existing on evaluation suite
• Human evaluation sampling: 5% of production requests reviewed weekly

Anti-Pattern Watchlist

• ❌ No evaluation framework ("it works on my examples")
• ❌ Single model provider with no fallback
• ❌ RAG without reranking (poor retrieval quality)
• ❌ No cost tracking (surprise $10K bills)
• ❌ Synchronous LLM calls blocking user requests (use streaming/async)

Phases

Phase 1 — AI Architecture & Model Selection

• Benchmark model options: compare cost/latency/quality on representative samples
• Design model routing strategy (simple → cheap model, complex → premium model)
• Design RAG architecture if applicable (chunking strategy, embedding model, vector DB, reranker)
• Set up provider abstraction layer:
  python

  # Example: LiteLLM provider abstraction
  from litellm import completion
  response = completion(model="gpt-4", messages=[{"role": "user", "content": "Hello"}])
  # Swap to: model="claude-3-opus" — zero code changes

• Define evaluation metrics and acceptance criteria
• Gate: Do not proceed until model benchmarks show ≥1 candidate meeting acceptance criteria.

Phase 2 — ML Pipeline & Fine-Tuning

• Data pipeline: collection, cleaning, formatting (JSONL, Parquet)
• Fine-tuning setup: LoRA/QLoRA for efficiency, full fine-tune for critical models
• Training infrastructure: cloud GPUs (RunPod, Lambda, together.ai) or managed (OpenAI, Vertex)
• Hyperparameter optimization: learning rate sweep, epoch tuning, data mix ratios
• Model registry: version, tag, promote (staging → production)
• Gate: Do not proceed until evaluation on benchmark set shows fine-tuned model meets acceptance criteria from Phase 1.

Phase 3 — Serving & Integration

• Model serving: API endpoints with streaming support
• Caching layer: semantic cache for repeated/similar queries (save 30-60% costs)
• Rate limiting and quota management per user/tenant
• Streaming responses for real-time UX
• Error handling: timeout → retry → fallback model → graceful error message

Phase 4 — Evaluation & Monitoring

• Automated evaluation suite (100+ test cases)
• Production monitoring: latency, error rate, cost, quality drift
• A/B testing framework for model comparison
• Feedback loop: user feedback → evaluation → model improvement
• Alerting: cost spike, latency spike, quality degradation, error rate increase

Output Structure

.forgewright/ai-engineer/
├── model-selection.md               # Model benchmarks and selection rationale
├── architecture.md                  # AI system architecture
├── rag-pipeline.md                  # RAG design (if applicable)
├── evaluation/
│   ├── eval-suite.md                # Evaluation framework design
│   ├── test-cases/                  # Test case datasets
│   └── results/                     # Benchmark results
├── mlops/
│   ├── pipeline.md                  # Training/deployment pipeline
│   ├── monitoring.md                # Production monitoring setup
│   └── cost-analysis.md             # Cost tracking and optimization
└── integration.md                   # API contracts and integration guide

Execution Checklist

• Model options benchmarked (min 3, with cost/latency/quality comparison)
• Provider abstraction layer (swap models without code changes)
• Fallback model configured for degraded mode
• RAG pipeline with hybrid search + reranking (if applicable)
• Evaluation suite with 100+ test cases
• LLM-as-judge + deterministic checks configured
• Model versioning and registry
• Streaming response support
• Semantic caching for cost optimization
• Cost tracking per request
• Production monitoring (latency, errors, quality drift)
• A/B testing infrastructure
• Rate limiting and quota management
• Automated regression testing before deployment