llm-eval-harness

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

LLM Eval Harness

Overview

概述

Give this skill an endpoint (

base_url

model

+ an API key in an env var) and it measures whether the endpoint actually works and whether the model is fast, stable, protocol-correct, and good enough — instead of trusting the vendor's headline numbers. Six dimensions, usually scattered across ad-hoc scripts that get rewritten (with the same bugs) every time:

Dimension	Script	Answers
Availability	`scripts/availability_probe.py`	which model IDs work here, with 3-state error classification
Request fidelity	`scripts/fidelity_probe.py`	do system prompt / tools / history actually REACH the model?
Speed	`scripts/speed_probe.py`	TTFT + sustained decode tok/s, thinking-aware
Concurrency / stability	`scripts/concurrency_probe.py`	success rate, p50/p90 latency, where it breaks
Protocol compliance	`scripts/protocol_probe.py`	does the Anthropic `thinking` block actually fire when requested, AND does the endpoint accept one already sitting in history on a later turn (N≥10, both are separate checks — see Dimension 3)?
Quality / use-case regression	`scripts/usecase_runner.py` + blind judges	does it pass your accumulated cases?

向本技能提供端点信息（

base_url

model

+ 环境变量中的API密钥），它会测量该端点是否真正可用，以及模型是否快速、稳定、符合协议规范且性能达标——而非轻信厂商的宣传数据。本工具涵盖六个评估维度，这些维度通常分散在临时脚本中，而这些脚本每次重写都会出现相同的bug：

维度	脚本	解决的问题
可用性	`scripts/availability_probe.py`	哪些模型ID在此端点可用，包含三种状态的错误分类
请求保真度	`scripts/fidelity_probe.py`	系统提示词/工具/对话历史是否真正送达模型？
速度	`scripts/speed_probe.py`	TTFT + 持续解码tok/s，支持思考过程感知
并发/稳定性	`scripts/concurrency_probe.py`	成功率、p50/p90延迟、性能崩溃阈值
协议合规性	`scripts/protocol_probe.py`	当请求Anthropic的 `thinking` 块时，它是否真正触发？并且在后续轮次中，端点是否接受已存在于对话历史中的 `thinking` 块（N≥10，这是两个独立的检查——参见维度3）？
质量/用例退化	`scripts/usecase_runner.py` + 盲评法官	模型是否能通过您积累的测试用例？

Which dimensions to run

选择要运行的维度

Route from what the user is actually doing:

"接入新模型/新网关先测一下" / onboarding a provider → Availability → Fidelity, then speed/concurrency if adoption looks likely
"这个 ID 能不能用 / 写个支持列表" → Availability (and read disciplines §9 before writing any "unavailable" verdict)
"模型不听 system prompt / agent 行为怪但没报错" → Fidelity (system canary)
"快不快 / tok/s 是真的吗" → Speed; "扛不扛得住" → Concurrency
"这个兼容端点是真兼容吗" → Protocol + Fidelity's tools/auth checks
"这个模型/渠道用着用着就 400 了 / session 断了 / continue 不下去了" (extended-thinking client, multi-turn session) → Protocol
--check history-replay
specifically — this symptom is the generation check passing while history-replay silently doesn't; don't stop at generation
"换模型质量会不会掉" → Quality regression
"要给厂商报 bug" → run the relevant probe with
```
--output
```
, then follow references/vendor_evidence_protocol.md
"部署闸门 / 常驻监控 / 告警老误报" → references/production_testing_patterns.md

Don't run all six ritually — pick what answers the user's question.

Key handling (non-negotiable): every script takes the API key by env-var name (

--key-env MY_KEY

), never the key value on the command line — so it stays out of

ps

, shell history, and any saved report. Never hardcode a key into a use-case file or a wrapper. Read references/evaluation_disciplines.md for the full reasoning behind this and the other disciplines.

Your private data lives outside this bundle. Use-case libraries, model rosters, and keys belong in

~/.llm-eval/

(or wherever you keep secrets), NOT in this skill directory — the skill is generic and public; your test suite is yours. See "Use-case library" below.

根据用户的实际需求选择：

"接入新模型/新网关先测一下" / 接入新供应商 → 可用性 → 保真度，如果可能采用该模型，再测试速度/并发
"这个ID能不能用 / 写个支持列表" → 可用性（在编写任何“不可用”结论之前，请先阅读第9条准则）
"模型不听system prompt / agent行为怪异但无报错" → 保真度（系统金丝雀测试）
"快不快 / tok/s是真的吗" → 速度；"扛不扛得住" → 并发
"这个兼容端点是真兼容吗" → 协议合规性 + 保真度的工具/认证检查
"这个模型/渠道用着用着就400了 / session断了 / continue不下去了"（支持思考过程的客户端、多轮会话） → 专门运行协议合规性的
--check history-replay
——这种症状表现为生成检查通过，但历史重放悄悄失败；不要仅停留在生成测试
"换模型质量会不会掉" → 质量退化
"要给厂商报bug" → 运行相关探针并添加
```
--output
```
参数，然后遵循references/vendor_evidence_protocol.md
"部署闸门 / 常驻监控 / 告警老误报" → 参考references/production_testing_patterns.md

不要机械地运行全部六个维度——选择能解答用户问题的维度。

密钥处理（不可协商）：每个脚本通过环境变量名称（

--key-env MY_KEY

）获取API密钥，绝对不要在命令行中直接输入密钥值——这样可以避免密钥出现在

ps

进程列表、shell历史记录和任何保存的报告中。绝对不要将密钥硬编码到用例文件或包装器中。阅读references/evaluation_disciplines.md了解此规则及其他准则的完整理由。

您的私有数据需存放在本工具包之外。用例库、模型列表和密钥应存放在

~/.llm-eval/

（或您存放机密的其他位置），而不是本技能目录中——本技能是通用且公开的；您的测试套件是私有的。请参阅下方的“用例库”部分。

Quick start

快速开始

Detect what you have, then run the dimensions that apply. For an OpenAI-compatible model:

bash

export MY_KEY=sk-...                       # the key never appears in a command below

先检测您的环境，然后运行适用的维度。对于兼容OpenAI的模型：

bash

export MY_KEY=sk-...                       # 密钥绝不会出现在下方的命令中

Speed: real-task throughput + sustained decode ceiling

速度：真实任务吞吐量 + 持续解码上限

uv run --with openai python scripts/speed_probe.py
--base-url https://api.example.com/v1 --model some-model --key-env MY_KEY --mode both

uv run --with openai python scripts/speed_probe.py \ --base-url https://api.example.com/v1 --model some-model --key-env MY_KEY --mode both

Concurrency: ramp until it breaks

并发：逐步增加负载直到性能崩溃

uv run --with aiohttp python scripts/concurrency_probe.py
--url https://api.example.com/v1/chat/completions --model some-model --key-env MY_KEY
--format openai --concurrency 10 20 40 60


If the endpoint is Anthropic-Messages-shaped (`/v1/messages`), also run the protocol probe
(below). Pick dimensions by what the user actually asked — don't run all six if they only
asked "is it fast?".

uv run --with aiohttp python scripts/concurrency_probe.py \ --url https://api.example.com/v1/chat/completions --model some-model --key-env MY_KEY \ --format openai --concurrency 10 20 40 60


如果端点是Anthropic-Messages格式（`/v1/messages`），还需运行协议探针（如下所示）。根据用户的实际问题选择维度——如果用户只问“它快吗？”，不要运行全部六个维度。

Dimension 0 — Availability (which model IDs actually work)

维度0 — 可用性（哪些模型ID真正可用）

bash

uv run --with aiohttp python scripts/availability_probe.py \
  --base-url <base> --key-env <ENV> --format both \
  --models model-a vendor/model-b @models.txt --output /tmp/avail.json

Verdicts are 3-state per failure, not pass/fail:
```
no-channel
```
(no route for this ID),
```
upstream-error
```
(route exists, upstream failing — retry later),
```
empty-content
```
(usually a max_tokens artifact on reasoning models, NOT a broken model). The probe keeps
```
--max-tokens
```
at 8192 by default precisely so thinking models don't read as dead — do not "optimize" it downward.
Before writing any "unavailable" list: verify each failing ID exists in official docs. Current-generation model names postdate your training data — WebSearch the vendor's / model developer's model page instead of enumerating guesses; suffixes (
```
-preview
```
, dated variants, tier names) decide routability. Full traps: disciplines §9.
Never probe a client-side marker as if it were a real model ID. Bracket-suffixed strings like
```
some-model[1m]
```
are frequently a CLI client's own local convention (Claude Code parses and strips
```
[1m]
```
before the request is ever built — see claude-switch-models-setup's "Configuring Context Window Size" section for the full mechanism) — they never appear on the wire. The probe warns when it sees one, but the discipline is yours: probe the bare model ID the vendor actually documents, not a string copied out of a Claude Code
```
ANTHROPIC_MODEL
```
env var.
The gateway's
```
/v1/models
```
listing is one input, never the verdict — real gateways route models the listing omits.

bash

uv run --with aiohttp python scripts/availability_probe.py \\
  --base-url <base> --key-env <ENV> --format both \\
  --models model-a vendor/model-b @models.txt --output /tmp/avail.json

每个失败案例的 verdict 分为三种状态，而非简单的通过/失败：
```
no-channel
```
（该ID无路由）、
```
upstream-error
```
（存在路由，但上游服务故障——稍后重试）、
```
empty-content
```
（通常是推理模型的max_tokens问题，而非模型损坏）。探针默认将
```
--max-tokens
```
设置为8192，正是为了避免推理模型被误判为不可用——不要“优化”降低此值。
在编写任何“不可用”列表之前：验证每个失败的ID是否存在于官方文档中。当前代模型名称晚于您的训练数据——请搜索供应商/模型开发者的模型页面，而非枚举猜测；后缀（
```
-preview
```
、日期变体、层级名称）决定了路由能力。完整的陷阱说明：准则第9条。
永远不要将客户端标记当作真实模型ID进行探测。带括号后缀的字符串如
```
some-model[1m]
```
通常是CLI客户端的本地约定（Claude Code会在构建请求前解析并去除
```
[1m]
```
——请参阅claude-switch-models-setup的“配置上下文窗口大小”部分了解完整机制）——它们绝不会出现在网络请求中。探针看到此类字符串时会发出警告，但您需遵守准则：探测供应商实际文档中列出的原始模型ID，而非从Claude Code的
```
ANTHROPIC_MODEL
```
环境变量中复制的字符串。
网关的
```
/v1/models
```
列表只是一个输入，绝非最终结论——实际网关会路由列表中未列出的模型。

Dimension 0.5 — Request fidelity (does your payload reach the model?)

维度0.5 — 请求保真度（您的 payload 是否送达模型？）

bash

uv run --with aiohttp python scripts/fidelity_probe.py \
  --base-url <base> --model <model> --key-env <ENV> \
  --format anthropic --check all --repeat 10 --output /tmp/fidelity.json

Four checks:
```
system
```
(canary code planted in the system prompt — can the model echo it back?),
```
tools
```
(full round-trip including returning a tool result and verifying the final answer uses it),
```
multiturn
```
(plant facts across turns, recall them all),
```
auth
```
(x-api-key vs Bearer on both protocol endpoints — gateways commonly accept both on one path and only one on the other, and the wrong-header 401 reads exactly like a dead key).
This is the dimension that catches the nastiest gateway failure: everything returns 200 and chats fluently, but the system prompt never reached the model — so any tool whose rules live in the system prompt silently misbehaves with no error anywhere. On one real gateway, one model alias never delivered the system prompt in 100+ samples across every legal way of sending it, while sibling aliases delivered intermittently with rates that changed by the hour.
Delivery through routed gateways is probabilistic and time-varying — hence
```
--repeat
```
(default 10) and a three-state verdict (delivered / intermittent k-of-N / not-delivered). A clean single-window result is still just that window: re-sample in another time slice before publishing a number (disciplines §12).
The probe reports token-accounting corroboration but never trusts it alone — behavioral evidence rules; see disciplines §10 (calibrate the meter) and §11 (why the canary is a neutral external fact, not an identity or obedience test).

bash

uv run --with aiohttp python scripts/fidelity_probe.py \\
  --base-url <base> --model <model> --key-env <ENV> \\
  --format anthropic --check all --repeat 10 --output /tmp/fidelity.json

四项检查：
```
system
```
（在系统提示词中植入金丝雀代码——模型能否将其回显？）、
```
tools
```
（完整往返测试，包括返回工具结果并验证最终答案是否使用该结果）、
```
multiturn
```
（在多轮对话中植入事实，验证模型能否全部回忆）、
```
auth
```
（在两个协议端点上测试x-api-key与Bearer认证——网关通常在一个路径上接受两种认证，而在另一个路径上只接受一种，错误的头部会导致401，与密钥无效的表现完全相同）。
此维度会捕获最棘手的网关故障：所有请求都返回200且对话流畅，但系统提示词从未送达模型——因此任何依赖系统提示词规则的工具都会静默失效，且无任何错误提示。在某个真实网关中，某个模型别名在100+次抽样中，无论以何种合法方式发送系统提示词，都从未成功送达，而其他别名的送达率则随时间变化。
通过路由网关的送达是概率性且随时间变化的——因此需要
```
--repeat
```
参数（默认10次）和三种状态的verdict（已送达 / 间歇性送达k-of-N / 未送达）。单次窗口的干净结果仅代表该窗口的情况：在发布结论前，请在另一个时间窗口重新抽样（准则第12条）。
探针会报告令牌计数的佐证，但绝不会单独依赖它——行为证据才是关键；请参阅准则第10条（校准计量器）和第11条（为何金丝雀是中立的外部事实，而非身份或服从测试）。

Dimension 1 — Speed (thinking-aware)

维度1 — 速度（支持思考过程感知）

bash

uv run --with openai python scripts/speed_probe.py \
  --base-url <…/v1> --model <model> --key-env <ENV> --mode both --output /tmp/speed.json

```
mixed
```
runs representative tasks (what real usage feels like);
```
decode
```
forces one long output to find the sustained ceiling (the number to compare against a vendor's claim);
```
both
```
does both.
The trap this script exists to avoid: reasoning models stream thinking in a separate
```
reasoning_content
```
field, but
```
completion_tokens
```
counts it. Collecting only
```
content
```
while dividing by
```
completion_tokens
```
produces wildly inflated numbers — a real ~750 tok/s model once measured as 4700 tok/s this way. The script captures both, takes TTFT as the first token of either kind, and reports
```
completion_tokens / (total − TTFT)
```
.
Read the output correctly: real-task throughput is lower than the decode ceiling because short outputs never reach steady state — that's expected, not a bug. Report both numbers, and note when the model emits thinking (its end-to-end latency includes reasoning time, not just typing).

bash

uv run --with openai python scripts/speed_probe.py \\
  --base-url <…/v1> --model <model> --key-env <ENV> --mode both --output /tmp/speed.json

```
mixed
```
模式运行代表性任务（真实使用场景的感受）；
```
decode
```
模式强制生成长输出以找到持续解码上限（与厂商宣称对比的数值）；
```
both
```
模式同时运行两者。
本脚本旨在避免的陷阱：推理模型在单独的
```
reasoning_content
```
字段中流式传输思考过程，但
```
completion_tokens
```
会将其计入。仅收集
```
content
```
并除以
```
completion_tokens
```
会产生严重虚高的数值——某个实际约750 tok/s的模型曾被这种方式测出4700 tok/s的结果。本脚本会同时捕获两者，将TTFT定义为两种类型的第一个令牌的时间，并报告
```
completion_tokens / (total − TTFT)
```
。
正确解读输出：真实任务吞吐量低于解码上限，因为短输出永远无法达到稳定状态——这是预期情况，而非bug。请同时报告两个数值，并注意模型是否输出思考过程（其端到端延迟包括推理时间，而非仅生成时间）。

Dimension 2 — Concurrency / stability

维度2 — 并发/稳定性

bash

uv run --with aiohttp python scripts/concurrency_probe.py \
  --url <full endpoint URL> --model <model> --key-env <ENV> \
  --format openai|anthropic --concurrency 10 20 40 60 --output /tmp/conc.json

Pass several
```
--concurrency
```
levels to ramp and find the ceiling — the level where success rate drops or latency explodes. A model that's fast single-threaded can still collapse at modest concurrency (real example: one provider held 50 concurrent at 0.4s while another dropped requests at just 5 concurrent).
The script isolates from any ambient proxy (
```
trust_env=False
```
) and disables keep-alive pooling (
```
force_close
```
) — otherwise you measure the proxy's limit or one pinned upstream replica, not the model. It prints a "concurrency proof" (overlapping request pairs) so you can confirm requests really ran in parallel.
Distinguish failure modes from the output: HTTP 429 (clean throttle, retriable) vs a TCP drop that hangs to timeout (much worse for UX) vs 5xx. They imply very different fixes.

bash

uv run --with aiohttp python scripts/concurrency_probe.py \\
  --url <完整端点URL> --model <model> --key-env <ENV> \\
  --format openai|anthropic --concurrency 10 20 40 60 --output /tmp/conc.json

传入多个
```
--concurrency
```
级别以逐步增加负载并找到上限——即成功率下降或延迟骤增的级别。单线程速度快的模型在适度并发下仍可能崩溃（真实案例：某供应商在50并发下保持0.4s延迟，而另一个供应商在仅5并发时就开始丢弃请求）。
脚本会隔离环境代理（
```
trust_env=False
```
）并禁用长连接池（
```
force_close
```
）——否则您测试的是代理的限制或固定的上游副本，而非模型本身。它会打印“并发证明”（重叠请求对），以便您确认请求确实是并行运行的。
从输出中区分故障模式：HTTP 429（正常限流，可重试）、TCP断开导致超时（对用户体验更糟）、5xx错误。它们意味着完全不同的修复方案。

Dimension 3 — Protocol compliance (Anthropic thinking block: generation AND history-replay)

维度3 — 协议合规性（Anthropic思考块：生成与历史重放）

bash

uv run python scripts/protocol_probe.py \
  --url <…/v1/messages> --model <model> --key-env <ENV> --check all --repeat 10 --output /tmp/proto.json

Only relevant for endpoints claiming Anthropic
```
/v1/messages
```
compatibility.
```
--check all
```
(default) runs BOTH sub-checks — they test different code paths and a vendor can pass one while hard-failing the other:
- generation: does
```
thinking: {type: enabled}
```
  actually produce
```
thinking_delta
```
  /
```
signature_delta
```
  SSE events when you request it? (the original check)
- history-replay: does the endpoint ACCEPT a
```
type: "thinking"
```
  block that's already sitting in a prior assistant turn, when that history is replayed back on a later turn — exactly what Claude Code and every other agentic client does on every continuation? (
```
--check history-replay
```
  to run just this one)
Real incident (2026-07-21) that motivated the history-replay check: a Kimi/Moonshot model via a China reseller passed generation fine (emits thinking correctly when asked) but hard-rejected history-replay — 400 "invalid part type: thinking" the instant a prior thinking block came back as input, killing the session on every subsequent turn. Passing generation told us nothing about this; they're orthogonal failure modes (response generation vs. request validation).
Don't conclude "vendor/reseller X is broken" from one cross-axis comparison — and don't stop at the first single-axis comparison that confirms a vendor-documented parameter either. This took three rounds to get right, kept in disciplines §§17-19 as the canonical cautionary tale: round 1 compared a different model AND a different reseller at once and (wrongly) blamed the reseller. Round 2 fixed that — same reseller, only the model varied — and the result matched a documented, named vendor parameter (Moonshot's own
```
preserve_thinking
```
), which looked like confirmation. Round 2 was STILL wrong: the probe never left that one reseller, so it couldn't see that the vendor's own direct/native endpoint handled the "rejected" model fine — real production traffic proved it. Read §19 before treating any single-reseller-confirmed result as final; check the vendor's own endpoint or real traffic for the actual channel in question before writing an "X doesn't support thinking" conclusion into anything.
Compliance is often probabilistic, not binary, for BOTH checks. One real vendor honored the thinking block on only ~13% of generation requests (vs 100% for two competitors). That's why
```
--repeat
```
defaults to 10; generation's verdict has three states (
```
fully-implemented
```
,
```
intermittent (k/N)
```
,
```
not-implemented
```
), history-replay's has its own three-plus states (
```
accepts-thinking-in-history
```
,
```
rejects-thinking-in-history
```
,
```
inconsistent
```
, or
```
inconclusive
```
when errors look unrelated to thinking at all). Never conclude from a single sample.
It forces
```
Connection: close
```
per request so a load balancer can't pin all samples to one replica and hide the real distribution (a real probe saw 0/10 with keep-alive vs 17/90 with close on the same endpoint).

bash

uv run python scripts/protocol_probe.py \\
  --url <…/v1/messages> --model <model> --key-env <ENV> --check all --repeat 10 --output /tmp/proto.json

仅适用于宣称兼容Anthropic
```
/v1/messages
```
协议的端点。
```
--check all
```
（默认）会运行两项子检查——它们测试不同的代码路径，厂商可能通过其中一项而完全失败另一项：
- 生成：当请求
```
thinking: {type: enabled}
```
  时，是否真正产生
```
thinking_delta
```
  /
```
signature_delta
```
  SSE事件？（原始检查）
- 历史重放：当后续轮次重放对话历史时，端点是否接受已存在于先前助手轮次中的
```
type: "thinking"
```
  块——这正是Claude Code和所有其他智能客户端在每次续接时的操作？（使用
```
--check history-replay
```
  仅运行此项）
推动历史重放检查的真实事件（2026-07-21）：某中国经销商提供的Kimi/Moonshot模型通过了生成检查（请求时正确输出思考过程），但完全拒绝历史重放——一旦先前的思考块作为输入返回，立即返回400“invalid part type: thinking”，导致每次后续轮次会话终止。通过生成检查无法发现此问题；它们是正交的故障模式（响应生成 vs 请求验证）。
不要从单次跨轴比较中得出“厂商/经销商X存在问题”的结论——也不要在第一次单轴比较确认厂商文档参数后就停止测试。这花了三轮才正确完成，在准则第17-19条中作为典型警示案例：第一轮同时比较了不同模型和不同经销商，错误地归咎于经销商。第二轮修正了问题——同一经销商，仅模型不同——结果与厂商文档中命名的参数（Moonshot的
```
preserve_thinking
```
）匹配，看似得到了确认。但第二轮仍然错误：探针从未离开该经销商，因此无法看到厂商自己的直接/原生端点可以正常处理“被拒绝”的模型——真实生产流量证明了这一点。在将任何“X不支持思考”的结论写入文档之前，请阅读第19条；在得出最终结论前，请检查厂商自己的端点或相关渠道的真实流量。
两项检查的合规性通常都是概率性的，而非二元的。某个真实厂商仅在约13%的生成请求中遵守思考块（而竞争对手为100%）。这就是
```
--repeat
```
默认值为10的原因；生成的verdict有三种状态（
```
fully-implemented
```
、
```
intermittent (k/N)
```
、
```
not-implemented
```
），历史重放的verdict有三种以上状态（
```
accepts-thinking-in-history
```
、
```
rejects-thinking-in-history
```
、
```
inconsistent
```
，或当错误与思考无关时的
```
inconclusive
```
）。绝不要从单次抽样得出结论。
它会强制每个请求使用
```
Connection: close
```
，以便负载均衡器无法将所有样本固定到一个副本，从而隐藏真实的分布情况（某个真实探针在长连接下得到0/10的结果，而在关闭长连接下得到17/90的结果）。

Dimension 4 — Quality / use-case regression (blind judge)

维度4 — 质量/用例退化（盲评法官）

This is two halves on purpose: collect, then judge independently.

Step 1 — collect the model's answers to your use-case library:

bash

uv run --with openai python scripts/usecase_runner.py \
  --base-url <…/v1> --model <model> --key-env <ENV> \
  --usecases ~/.llm-eval/usecases.json --output-dir ~/.llm-eval/runs/<model>

Step 2 — judge with independent blind judges (orchestrate inline — do NOT let the model grade itself). For each answer in the run directory, spawn 3 independent Task agents (or fewer for a quick pass). Each judge gets ONLY: the prompt, the answer, and the case's

rubric

— and is explicitly told it is judging in isolation, with no knowledge of other judges' scores or any prior evaluation (this prevents anchoring). Then aggregate:

A case passes only on majority agreement among judges.
Compute precision per category (using each case's
```
tags
```
): a category where judges systematically disagree with the rubric is a real weakness — on one real eval, a whole category scored 12.5% precision and exposed a systematic misclassification that a single grader would have missed.
Count only explicit judgments. A judge that didn't return a verdict is not a pass — silence ≠ consent. This guards against automation bias.

For the rubric-scoring mechanics (LLM-as-judge thresholds,

llm-rubric

), you can also compose with the promptfoo-evaluation skill — point its

providers

at the same endpoint. This harness's blind-judge method and promptfoo's rubric assertions are complementary: use promptfoo for fast per-case pass/fail gating, blind judges for precision on a category you suspect is weak. Full method: references/quality_blind_judge.md.

这特意分为两部分：收集，然后独立评判。

步骤1 — 收集模型对您的用例库的回答：

bash

uv run --with openai python scripts/usecase_runner.py \\
  --base-url <…/v1> --model <model> --key-env <ENV> \\
  --usecases ~/.llm-eval/usecases.json --output-dir ~/.llm-eval/runs/<model>

步骤2 — 使用独立盲评法官进行评判（内联编排——绝对不要让模型自我评分）。对于运行目录中的每个答案，生成3个独立的Task代理（快速检查可减少数量）。每个法官仅会收到：提示词、答案、案例的

rubric

——并明确告知其处于隔离状态，不知道其他法官的评分或任何先前的评估（这可防止锚定效应）。然后汇总结果：

仅当法官达成多数一致时，案例才通过。
计算每个类别的精度（使用每个案例的
```
tags
```
）：法官系统地与rubric不一致的类别是真正的弱点——在某次真实评估中，某个类别的精度仅为12.5%，暴露了系统分类错误，而单一评分者会错过此问题。
仅计算明确的评判结果。未返回verdict的法官不算通过——沉默≠同意。这可防止自动化偏见。

对于rubric评分机制（LLM作为法官的阈值、

llm-rubric

），您也可以与promptfoo-evaluation技能组合使用——将其

providers

指向同一个端点。本工具的盲评方法与promptfoo的rubric断言互补：使用promptfoo进行快速的单案例通过/失败筛选，使用盲评法官对您怀疑存在弱点的类别进行精度评估。完整方法：references/quality_blind_judge.md。

Use-case library

用例库

Keep it OUTSIDE this bundle (e.g.

~/.llm-eval/usecases.json

) so it survives skill updates and never lands in a public repo. It's a plain JSON list — version it in a private repo to accumulate a regression suite over time:

json

[
  {"id": "refund-window", "prompt": "A customer asks for a refund 20 days after purchase. Reply as support.",
   "rubric": "1.0 if it correctly cites the 30-day refund window; 0.0 if it refuses or invents a different window.",
   "tags": ["support", "policy"]},
  {"id": "lru-cache", "prompt": "Implement an LRU cache in Python with O(1) get/put.",
   "rubric": "1.0 if get and put are both O(1) via dict + doubly linked list and the self-test passes.",
   "tags": ["code"]}
]

assets/example_usecases.json

is a starter you can copy. Only

id

and

prompt

are required;

rubric

expected

, and

tags

make judging sharper.

将其存放在本工具包之外（例如

~/.llm-eval/usecases.json

），以便在技能更新后仍能保留，且不会进入公共仓库。它是一个普通的JSON列表——在私有仓库中进行版本控制，随时间积累退化测试套件：

json

[
  {"id": "refund-window", "prompt": "A customer asks for a refund 20 days after purchase. Reply as support.",
   "rubric": "1.0 if it correctly cites the 30-day refund window; 0.0 if it refuses or invents a different window.",
   "tags": ["support", "policy"]},
  {"id": "lru-cache", "prompt": "Implement an LRU cache in Python with O(1) get/put.",
   "rubric": "1.0 if get and put are both O(1) via dict + doubly linked list and the self-test passes.",
   "tags": ["code"]}
]

assets/example_usecases.json

是一个入门示例，您可以复制使用。仅

id

和

prompt

是必填项；

rubric

、

expected

和

tags

可使评判更精准。

Running a full evaluation

运行完整评估

When the user says "evaluate / benchmark this model", the typical flow is:

Identify the shape — OpenAI-compatible (
```
/v1/chat/completions
```
) or Anthropic-Messages (
```
/v1/messages
```
)? Hit
```
GET /v1/models
```
or read the vendor docs; don't assume. This decides which probes apply (protocol probe is Anthropic-only). Remember the listing is incomplete evidence either way (disciplines §9).
For a new endpoint, availability and fidelity come first — speed numbers for a model whose system prompt never arrives are answering the wrong question.
Run the dimensions the user cares about — speed and concurrency for "is it fast/stable", add protocol for an Anthropic vendor, add quality when they have a use-case suite. Write each probe's
```
--output
```
JSON to a run directory.
Report honestly, separating measured from inferred. Lead with the headline the user asked about (e.g. "sustained decode ceiling exceeds the vendor's claimed tok/s, while real-task throughput runs lower"). If a number looks impossible (e.g. throughput far above the vendor claim, or a single-sample protocol verdict), treat it as a measurement artifact to investigate, not a result — that skepticism is the whole point of this harness. Rates on routed gateways are additionally time-varying: re-sample another window before freezing any number into a document (disciplines §12).
Comparing two models? Run the identical probes against each with the same flags, and put the two JSON outputs side by side. Keep the test conditions identical (same concurrency levels, same use cases) or the comparison is meaningless.
Found a vendor bug worth reporting? Don't paste raw probe output at their support channel — build the evidence package per references/vendor_evidence_protocol.md (self-audit first, observation wording, request-id table, pre-registered thresholds, adversarial counter-review).

For tests that will run repeatedly against a live system — deployment gates, resident canaries, fault-injection mocks, and the monitoring statistics that lie — see references/production_testing_patterns.md.

当用户说“evaluate / benchmark this model”时，典型流程如下：

识别端点类型——是兼容OpenAI（
```
/v1/chat/completions
```
）还是Anthropic-Messages（
```
/v1/messages
```
）？调用
```
GET /v1/models
```
或阅读厂商文档；不要假设。这决定了适用的探针（协议探针仅适用于Anthropic）。请记住，列表是不完整的证据（准则第9条）。
对于新端点，先测试可用性和保真度——系统提示词从未送达的模型，其速度数值毫无意义。
运行用户关心的维度——速度和并发用于“它快吗/稳定吗”，Anthropic厂商需添加协议测试，有测试用例套件时添加质量测试。将每个探针的
```
--output
```
JSON写入运行目录。
如实报告，区分测量结果与推断。以用户关心的核心结论开头（例如“持续解码上限超过厂商宣称的tok/s，而真实任务吞吐量较低”）。如果某个数值看起来不合理（例如吞吐量远高于厂商宣称，或单次抽样的协议verdict），请将其视为需要调查的测量 artifact，而非结果——这种怀疑正是本工具的意义所在。路由网关的速率还会随时间变化：在将任何数值写入文档之前，请在另一个窗口重新抽样（准则第12条）。
**比较两个模型？**对每个模型运行完全相同的探针和参数，并将两个JSON输出并排对比。保持测试条件完全相同（相同的并发级别、相同的用例），否则比较毫无意义。
**发现值得报告的厂商bug？**不要将原始探针输出粘贴到其支持渠道——请按照references/vendor_evidence_protocol.md构建证据包（先自我审核、观察措辞、请求ID表、预先注册的阈值、对抗性复核）。

对于需要重复运行在实时系统上的测试——部署闸门、常驻金丝雀测试、故障注入模拟、监控统计——请参阅references/production_testing_patterns.md。

Next step

下一步

After a run, offer the natural follow-ups:

Evaluation complete for <model>.

Options:
A) Render an HTML dashboard of the results — compose with a visualization skill (Recommended if sharing)
B) Compare against another model — same probes, side-by-side
C) Add the failing cases to ~/.llm-eval/usecases.json as a permanent regression guard
D) Done — the numbers answer the question

运行完成后，提供自然的后续选项：

<model>评估完成。

选项：
A) 渲染结果的HTML仪表盘——与可视化技能组合使用（推荐用于分享）
B) 与另一个模型对比——使用相同探针，并排展示
C) 将失败案例添加到~/.llm-eval/usecases.json中，作为永久的退化防护
D) 完成——数值已解答问题
```",