Cost Verification Auditor

When to Use

• Validating token estimation systems after implementation
• Pre-deployment cost accuracy checks
• Debugging unexpected API bills
• Periodic estimation drift detection

• Looking up model pricing (use pricing docs)
• Budget planning or forecasting
• Cost optimization strategies
• Comparing models by price

Core Audit Process

Decision Tree

Has estimator? ──No──→ Build estimator first (see Calibration Guidelines)
      │
     Yes
      ↓
Define 3+ test cases (simple/medium/complex)
      ↓
Estimate BEFORE execution (no peeking!)
      ↓
Execute against real API
      ↓
Calculate variance: (actual - estimated) / estimated
      ↓
Variance ≤ ±20%? ──Yes──→ PASS ✓
      │
     No
      ↓
Apply fixes from Anti-Patterns section
      ↓
Re-run verification

Variance Formula

const inputVariance = (actual.inputTokens - estimate.inputTokens) / estimate.inputTokens;
const outputVariance = (actual.outputTokens - estimate.outputTokens) / estimate.outputTokens;
const costVariance = (actual.totalCost - estimate.totalCost) / estimate.totalCost;

// PASS if both input AND output within ±20%
const passed = Math.abs(inputVariance) <= 0.20 && Math.abs(outputVariance) <= 0.20;

Common Anti-Patterns

Anti-Pattern: The 500-Token Overhead Myth

• Pre-2025: Many tutorials used 500+ token estimates
• 2025+: Direct API overhead is minimal (~10 tokens)

Anti-Pattern: Per-Node Accuracy Obsession

• Focus on total DAG cost variance (should be ±20%)
• Accept per-node output variance up to ±40%
• Use constrained prompts ("list exactly 3") to reduce variance

Anti-Pattern: Peeking Before Estimating

1. Estimate based on prompt length and heuristics
2. Execute API call
3. Compare variance
4. Adjust heuristics if needed

Calibration Guidelines

Input Token Estimation

// Calibrated 2026-01-30
const inputTokens = Math.ceil(prompt.length / CHARS_PER_TOKEN) + OVERHEAD;

Output Token Estimation

Model Behavior

Quick Fixes

Verification Checklist

• 3+ test cases (simple, medium, complex)
• Estimates run BEFORE API calls
• Variance formula:

  (actual - estimated) / estimated

• Target: ±20% for input AND output
• Report includes actionable recommendations

References

/references/calibration-data.md