Prompt Cache Audit

When to use

• cached_tokens=0

  ,

  cache_read_input_tokens=0

  , or cache writes without reads.
• Cache hit rate, TTFT, prefill latency, or input-token cost regressed.
• User says LLM cost or speed regressed around repeated long prompts, long-context agents, or shared static context.
• LLM request shape changed where repeated long prompts, TTFT, cached-token telemetry, or LLM cost matter.
• Prompt text, message order, request builders, tools, schemas,

  response_format

  , provider API surface, model/router settings, agent loop structure, context compaction, or inference deployment changed.
• The request uses long system prompts, tool catalogs, schemas, static documents, few-shot examples, or repeated RAG/CAG context.
• The app uses OpenAI

  prompt_cache_key

  , Anthropic

  cache_control

  , Bedrock

  cachePoint

  , OpenRouter routing, Gemini/Qwen/DeepSeek cache fields, or Azure OpenAI cached-token telemetry.
• An agent changes tools, compacts history, mutates early messages, or switches modes across steps.
• vLLM/SGLang/self-hosted deployments have multi-replica routing, KV pressure, tokenizer/chat-template drift, or cache-aware routing questions.

When not to use

• generic prompt writing or prompt-quality editing without a caching concern
• ordinary short prompt edits where no repeated long prefix, TTFT, cache telemetry, or LLM cost concern exists
• generic RAG design unless repeated context placement/cacheability is part of the task
• token counting or context-window sizing only
• response caching only, unless comparing it with prompt prefix caching
• non-LLM frontend/backend performance or non-inference Kubernetes routing
• speculative savings claims without usage data or clearly stated assumptions

Modes

• Code audit: inspect prompt construction, tool/schema serialization, history management, provider calls, routing, and engine config. Propose focused diffs and verify them.
• Advisory: if no codebase is available, ask targeted diagnostic questions and give provider-checked recommendations.
• Agent audit: when tools, tool routing, MCP, agent loops, compaction, or multi-step trajectories are present, always run the agent-specific checks.
• Deployment audit: when vLLM/SGLang, Kubernetes, Docker Compose, gateways, or multiple inference replicas are present, inspect routing and KV-cache capacity as first-class causes.

Default Project Audit Workflow

1. Scan the repo for provider calls, cache controls, routing hints, prompt builders, tool/schema registries, and self-hosted engine signals. Use

   scripts/extract_llm_calls.py

   when useful.
2. Inspect the request path in code: prompt rendering, system/developer messages, tool ordering, structured-output schemas, history management, compaction, and provider SDK parameters.
3. Inspect config and deployment files: environment defaults, feature flags, gateway/router settings, Docker Compose, Kubernetes, Helm, vLLM/SGLang flags, and replica topology.
4. Load only the relevant provider and scenario references, then apply the audit flow and anti-pattern checks.
5. Ask for usage logs, rendered request payloads, traces, or billing exports only when code/config review needs telemetry evidence, prefix comparison, ROI math, or incident correlation.

Audit Inputs

Project Context Gate

1. Map prompt families, request builders, model/provider routes, agent loops, deployment paths, and usage frequency.
2. Separate hot repeated paths from rare jobs, one-off prompts, admin flows, experiments, and prompt families that cannot share a stable prefix.
3. Mark each finding as applicable, conditionally applicable, or not applicable to the project path under review.
4. Ask for telemetry only after code/config context shows the evidence needed next.

Applicability Gate

1. Reusable prefix: Is there a static or semi-static prefix above the provider/model threshold or large enough to matter for self-hosted KV reuse?
2. Repeat cadence: Is the same prefix reused often enough before cache expiry or eviction?
3. Exact stability: Are tools, schemas, system/developer instructions, examples, images, and early messages byte/token stable across target requests?
4. Telemetry: Are cache-read/write fields, input/output tokens, TTFT/prefill timing, model/route, and prompt version available?
5. Cost shape: Is input prefill/input-token cost meaningful, or do output tokens/decode/tool latency dominate?
6. Safety boundary: Would broader cache reuse violate tenant, privacy, data residency, ZDR, or side-channel requirements?

Language Match Rule

cached_tokens

cache_control

cachePoint

TTFT

prompt_cache_key

response_format

Agent-First Output Contracts

• Quick triage: use when artifacts are incomplete. Answer with provider/engine guess, most likely cache blocker, evidence needed next, and one safe next command or artifact request.
• Code audit findings: use when code is available. Lead with a decision summary, then file-line findings in the report format, then clean checks, then verification commands.
• Provider migration risk: use when moving between OpenAI, Anthropic, Bedrock, OpenRouter, Azure OpenAI, Gemini, Qwen, DeepSeek, or self-hosted engines. Compare cache semantics, usage fields, prefix layout risk, routing risk, and cost assumptions before recommending edits.
• Agent loop audit: use for coding assistants, MCP clients, tool-using agents, compaction, mode switching, or long multi-step workflows. Always inspect stable tools, early messages, per-step prefix hashes, cache fields, output tokens, and compaction events.
• Deployment audit: use for vLLM, SGLang, Kubernetes, Docker Compose, gateways, autoscaling, or multi-replica inference. Treat routing locality and KV budget as first-class causes, not secondary deployment details.
• Not worth caching: use when the Applicability Gate fails or the evidence shows output decode, external tool latency, rate limits, or privacy isolation dominate. Say what should change instead and what evidence would reopen prompt-cache work.

Measurement change:
Prompt behavior change:
Provider/routing change:
Confidence:
Do first:
Do not do yet:

Measurement change: yes

Prompt behavior change: pilot only after telemetry

Provider/routing change: no, not yet

Change needed: yes

Change needed: no

Change needed: unknown until <specific evidence>

Evidence-Bearing Findings

source | severity | provider/engine | issue | evidence | evidence_type | confidence | impact_condition | cache impact | safe_first_action | fix | validation | do_not_do_yet

confirmed from code

confirmed from telemetry

provider-doc hypothesis

needs validation

1. Confirmed findings: issues supported by code/config/telemetry and applicable to the project path.
2. Hypotheses: plausible cache risks that need usage logs, rendered payloads, route metrics, or provider docs before severity can rise.
3. Not applicable: generic cache advice that the project context rules out, such as prefix-cache work for a once-daily prompt family with no meaningful repeated stable prefix.

Explicit Review Default

Use-Case Map

references/use-cases.md

cached_tokens=0

tools

response_format

references/mechanics.md

openrouter/auto

provider

docker-compose.yml

Scenario References

• Cost or migration:

  references/economics.md

  for effective-cost variables, output-share checks, TTL/write-premium break-even, and migration cache risk.
• Mechanics, latency, or self-hosted compute:

  references/mechanics.md

  for prefill vs decode, KV reuse, and what cache hits can and cannot improve.
• Release, incident, deploy, or monitoring:

  references/predeploy-checklist.md

  for blocking checks, triage order, and observability dimensions.
• OpenRouter or managed provider routing:

  references/openrouter.md

  for sticky routing, provider fallback/order, cache usage fields, and provider-specific cache controls through OpenRouter.
• Agents, coding assistants, MCP, or dynamic tools:

  references/agent-tools.md

  for tool strategy selection, mode switching, and context compaction.
• Self-hosted SGLang:

  references/sglang.md

  for RadixAttention, SGLang router, HiCache, tokenizer/chat-template drift, and cache-aware deployment checks.
• Full audit deliverable:

  references/report-template.md

  when the user asks for a written report or when findings need a reusable handoff artifact.
• Machine-readable rules:

  references/rules.json

  when scripting, rendering, or validating findings by anti-pattern ID.

Bundled Scripts

• scripts/prefix_stability_check.py

  : compare two rendered prompts or JSON request payloads as raw bytes by default and find the first divergent prefix location; use

  --canonical-json

  only when sorted-key normalization is intentional.

• scripts/layout_linter.py

  : inspect JSON request payloads, including Chat-style

  messages

  and Responses-style

  input

  /

  instructions

  , for volatile early content, unsorted tools, and dynamic schema fields before doing deeper manual layout review.

• scripts/analyze_usage_logs.py

  : summarize JSON/JSONL/CSV usage logs across OpenAI, Anthropic-compatible, Bedrock-style, and OpenAI-compatible cache fields; use

  --jsonl-normalized

  when a downstream report or dashboard needs per-record canonical events.

• scripts/estimate_cache_roi.py

  : estimate input-only and total-cost impact from static/dynamic/output tokens, hit rate, request count, and explicit pricing assumptions.

• scripts/extract_llm_calls.py

  : scan a repository for likely LLM provider calls, cache-control fields, routing signals, and self-hosted engine hints before choosing provider references.

• scripts/render_audit_report.py

  : combine usage-log summaries and one-line findings into a reusable Markdown or JSON audit report.

• scripts/validate_skill_package.py

  : validate frontmatter, referenced files, JSON evals, and Python helper syntax before sharing or publishing the skill.

• scripts/run_trigger_eval.py

  : summarize positive and negative trigger-eval coverage from

  evals/trigger_eval.json

  .

Script Transparency Rule

Freshness Gate

• pricing, cache discounts, storage charges, or write premiums
• current model names, support matrices, and availability by region
• minimum cacheable tokens, cache granularity, TTL, and retention
• usage field names and API parameters
• tool-search, allowed-tools, defer-loading, or cache-control semantics

Provider Detection

docker-compose.yml

Dockerfile

Deployment

Service

Ingress

openrouter

openrouter.ai/api/v1

OPENROUTER_API_KEY

@openrouter/sdk

OpenRouter

openrouter/auto

references/openrouter.md

AzureOpenAI

AZURE_OPENAI_ENDPOINT

azure.ai.openai

api-version

references/azure-openai.md

openai

responses.create

chat.completions

prompt_cache_key

prompt_cache_retention

references/openai.md

bedrock-runtime

BedrockRuntime

boto3.client("bedrock-runtime")

client.converse

converse_stream

InvokeModelCommand

ConverseCommand

invoke_model

cachePoint

CacheReadInputTokens

CacheWriteInputTokens

references/bedrock.md

anthropic

messages.create

cache_control

references/anthropic.md

vllm

--enable-prefix-caching

AsyncLLMEngine

LLM(

references/vllm.md

sglang

sglang_router

RadixAttention

--disable-radix-cache

HiCache

references/sglang.md

deepseek

api.deepseek.com

prompt_cache_hit_tokens

references/deepseek.md

google.genai

google.generativeai

vertexai

CachedContent

references/gemini.md

dashscope

qwen

bailian

aliyun

references/qwen.md

yandexgpt

foundationModels

llm.api.cloud.yandex.net

references/yandexgpt.md

z.ai

zai

glm-

api.z.ai

references/zai.md

Audit Flow

1. Detect mode, provider, and use-case scenario.
2. Load the relevant scenario reference and provider reference; do not load unrelated references.
3. Apply the Freshness Gate for provider-specific facts.
4. Run the Applicability Gate.
5. Map prompt structure in order: tools, structured-output schemas, system/developer instructions, few-shot examples, static documents/context, retrieved context, conversation history, user-specific data, volatile values.
6. Mark each segment as static, semi-static, dynamic, or volatile.
7. Measure the symptom: cache ratio, TTFT/prefill latency, output/decode time, cache writes vs reads, and whether the drop correlates with deploys, SDK changes, prompt changes, replica count, or agent steps.
8. Scan universal anti-patterns below.
9. If an agent loop or tools are present, run the Agent Tool Stability checks.
10. Apply provider-specific checks from the loaded reference.
11. Report findings with evidence type, confidence, impact condition, safe first action, concrete fix, validation metric, and any premature action to avoid.
12. When making code changes, verify prefix stability before claiming success.

Audit Playbooks

• OpenAI cached_tokens=0: check prompt length/threshold, first-prefix drift,

  responses.create

  vs Chat usage fields,

  prompt_cache_key

  granularity,

  prompt_cache_retention

  , output-token dominance, and whether an OpenAI-compatible wrapper is actually in use.
• Claude/Bedrock/OpenRouter writes without reads: distinguish cache creation/write fields from read/hit fields, then inspect cache breakpoint placement, dynamic content before the breakpoint, TTL/retention, model/region/API support, fallback routing, and actual routed provider/model.
• Dynamic tools in long agent loops: compare

  tools_count

  , sorted tool-name hash,

  prefix_hash

  , mode state, and cache fields per step. Prefer stable route-level tool bundles, sorted schemas, provider-supported allowed tools/tool search/deferred loading, or self-hosted masking after checking current docs.
• High hit rate but no savings: separate input savings from total cost and final latency. Check output-token share, decode time, external tool time, TPM/rate-limit behavior, and cache read/write pricing assumptions before changing prompt layout.
• OpenAI-compatible wrapper ambiguity: if

  base_url

  , Azure, OpenRouter, Bedrock, DashScope/Qwen, or another gateway wraps an OpenAI SDK, load the wrapper reference first and do not recommend direct OpenAI-only parameters until the wrapper docs support them.
• Self-hosted multi-replica miss: inspect gateway/service routing, prefix-aware hashing, tokenizer/chat-template drift,

  max_model_len

  , KV block pressure, eviction metrics, and route/replica-level hit metrics.
• New provider docs project-change audit: compare the new provider facts against current code, references, evals, and tests. Recommend no code change when the project already encodes the behavior or when the fact is not applicable to this provider path.

Rule Categories

cache_control

cachePoint

Severity

Applicability Before Severity

medium

low

needs validation

• Critical: confirmed metric drop or cache miss on a large shared prefix, expensive model, high traffic, long agent trajectory, or multi-replica production path.
• High: likely cache killer found in a hot path, or telemetry/traffic/token evidence shows meaningful cache/cost/TTFT impact but metrics are still incomplete.
• Medium: pattern can fragment cache but impact depends on traffic shape.
• Low: defensive cleanup, documentation, or monitoring improvement.

medium

high

Universal Anti-Patterns

AP-1: Volatile Data In Prefix

datetime

time

uuid

session_id

request_id

trace_id

run_id

user.name

user_id

tenant

company

cwd

git status

platform

# bad
system = f"Today: {datetime.now()}. You help {user.name}. {BASE_PROMPT}"

# good
system = BASE_PROMPT
user_msg = f"[ctx: time={datetime.now()}, user={user.name}] {query}"

AP-2: Non-Deterministic Tools, Schemas, Or Serialization

json.dumps

sort_keys=True

requestId

response_format

tools = sorted(registry.values(), key=lambda t: t["function"]["name"])
schema_text = json.dumps(response_format, ensure_ascii=False, sort_keys=True)

request_id

AP-3: Template, Whitespace, Media, Or SDK Drift

.strip()

detail

AP-4: Dynamic Tool Set Inside Agent Loop

tools

get_active_tools

select_tools

• Keep compact tools stable for the session and sort by name.
• Use masking/constrained decoding in self-hosted inference.
• Use provider mechanisms such as allowed-tools, tool search, or deferred loading only after checking current docs.
• Route to fixed tool bundles before the agent loop for multi-domain apps.

AP-5: History Mutation Instead Of Append-Only Growth

summarize

compaction

truncate

messages.pop

del messages

def manage_context(messages, max_tokens):
    anchor = messages[:2]  # system + first stable turn
    tail = messages[2:]
    while tail and count_tokens(anchor + tail) > max_tokens:
        tail.pop(0)
    if count_tokens(anchor + tail) > max_tokens:
        raise ValueError("stable prefix anchor exceeds context budget")
    return anchor + tail

AP-6: Mode Switching Or Framework Injection Mutates The Prefix

run_id

trace_id

cwd

AP-7: Cache-Blind Routing Across Replicas

• Managed API: use provider-supported cache key or routing hint only after checking docs.
• OpenRouter: inspect sticky routing,

  provider.order

  ,

  provider.only

  ,

  provider.ignore

  , fallback/model routing, and first-message conversation identity.
• vLLM/SGLang/self-hosted: use prefix-aware routing, consistent hashing, or a gateway that hashes the stable prefix.
• Minimum viable: route by stable prefix family while monitoring hot spots.

AP-8: Parallel Fan-Out On A Cold Prefix

asyncio.gather

Promise.all

ThreadPoolExecutor

await warm_cache(shared_prefix, max_tokens=1, tools=[])
results = await asyncio.gather(*[call(shared_prefix, q) for q in batch])

AP-9: Cache Lifetime, KV Budget, Or Eviction Mismatch

max_model_len

AP-9b: Over-Isolation Fragments Shared Prefixes

cache_salt

user_id

AP-10: Experiment Or Config Fragmentation

variant

experiment

feature_flag

openrouter/auto

models

provider.order

Agent Tool Stability Checks

• Log

  cached_tokens

  /

  cache_read_input_tokens

  on each step.
• Log

  prefix_hash

  for canonical

  system + tools + stable early messages

  .
• Log

  tools_count

  and a sorted list/hash of tool names.
• Log output tokens and streaming timestamps when latency is the symptom.
• Check whether cache drops exactly when the tool list changes.
• Confirm tool descriptions are fixed at session start or loaded via provider-supported deferred mechanisms.
• Confirm compaction does not rewrite

  system + tools + first messages

  .
• Confirm framework metadata is not injected before the cacheable prefix.
• For managed routers such as OpenRouter, log actual model/provider route when available.

import hashlib
import json


def prefix_fingerprint(system, tools=None, response_format=None, early_messages=None):
    payload = {
        "system": system,
        "tools": tools or [],
        "response_format": response_format,
        "early_messages": early_messages or [],
    }
    text = json.dumps(payload, ensure_ascii=False, sort_keys=True)
    return hashlib.sha256(text.encode()).hexdigest()[:12]

Report Format

source | severity | provider/engine | issue | evidence | evidence_type | confidence | impact_condition | cache impact | safe_first_action | fix | validation | do_not_do_yet

file:line | severity | provider/engine | issue | cache impact | fix | validation

## Prefix Cache Audit Report

Provider/engine: ...
Mode: code audit / advisory / agent audit
Provider facts: verified on YYYY-MM-DD / unverified
Measurement change: yes / no / unknown
Prompt behavior change: yes / no / pilot only / unknown
Provider/routing change: yes / no / not yet / unknown
Confidence: high / medium / low
Do first: ...
Do not do yet: ...

### Findings

path/to/file.py:42 | medium | OpenAI | tool schema order changes between calls | registry iteration is unsorted in the request builder | confirmed from code | medium-high | matters on hot repeated routes with long tool schemas | every order change can invalidate downstream prefix reuse | add tool/schema hash logging first | sort tools by stable name and serialize schemas with sorted keys | compare prefix fingerprints across three requests and confirm cached-token fields increase | do not change provider routing yet

### Clean Checks

- AP-1 volatile prefix data: clean
- AP-7 routing: not applicable, single managed endpoint

### Monitoring

- cache ratio definition for this provider
- prefill/TTFT vs decode/output split
- prefix hash dimensions
- deploy/change correlation to watch

Agent-First Quality Bar

• Answer the decision the user asked for: change needed, no change, or evidence missing.
• Prefer wrapper/router references over generic provider references when both signals exist.
• Do not make exact provider claims without loading the relevant reference and applying the Freshness Gate.
• Distinguish cache miss, cache write-without-read, uneconomic cache hit, decode-bound latency, rate-limit pressure, and privacy-driven isolation.
• Include validation that can falsify the recommendation: prefix fingerprints, provider usage fields, route/replica metrics, or cost/latency split.
• Do not propose cache controls, cache keys, or routing hints when the Not worth caching contract applies.

Verification

• Prefix-stability fixes: rendered cacheable prefix fingerprint is unchanged across different users/timestamps/queries.
• Provider fixes: repeated calls show cache-read/cached-token fields increasing according to the provider reference.
• Routing fixes: repeated prefix families land on the intended route and cache metrics improve by route.
• vLLM/self-hosted fixes: prefix cache hit metrics and KV block pressure metrics improve under a representative workload.

Advisory Questions

1. Which provider or inference engine?
2. Is this a cost/migration, prompt/code, agent, deployment, or observability/CI audit?
3. What artifacts are available: request code, rendered prompts, usage logs, deployment config, dashboards, evals?
4. What are median/p95 input tokens, static prefix tokens, output tokens, and agent steps?
5. What cache usage fields are visible in responses?
6. Are there multiple replicas or gateways?
7. Are tools/schemas stable across requests and agent steps?
8. Is history append-only, compacted, or summarized?
9. Are cache keys, salts, or routing hints per-user/per-request or shared by prefix family?
10. What changed before the cache hit rate or TTFT regressed?