eval-triage

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Triage Failing Experiments and Evaluations

排查失败的实验和评估

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

从“运行失败”一路定位到具体的相关数据行、evaluator 和输入，进而查明根因。请按顺序执行以下步骤。

Prerequisites

前置条件

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Every list command takes

--limit <n>

to cap the rows and

--jq <expr>

to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (

--jq '.traces | length'

), and read how many there are in all at

.pagination.total

. Bare

--jq length

counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx --yes langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without

--user-approved

, and

--dry-run

prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with

--email

. With

--session

, always run

--dry-run

first and let the user read the payload, because a transcript carries content they never reviewed.

npx --yes langwatch report --help

explains the options.

使用

langwatch docs <path>

以 Markdown 格式阅读文档。以下是一些有用的入口：

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

使用

langwatch --help

和

langwatch <subcommand> --help

发现可用命令。List 和 get 命令接受

--format json

，可获得机器可读的输出。每个 list 命令都接受

--limit <n>

来限制返回的行数，并接受

--jq <expr>

来读取响应的一部分。分页 list 响应带有 envelope，因此应通过行数组统计行数（

--jq '.traces | length'

），并通过

.pagination.total

查看总行数。仅使用

--jq length

统计的是 envelope 的字段数，而不是行数。请先阅读文档，不要猜测 SDK API 或 CLI flag。

如果当前没有 shell 可用，可以通过普通 HTTP 获取相同的 Markdown 内容。为任意文档路径追加

.md

（例如 https://langwatch.ai/docs/integration/python/guide.md）。索引：https://langwatch.ai/docs/llms.txt。场景索引：https://langwatch.ai/scenario/llms.txt

如果执行此 skill 期间遇到任何失败或令人困惑的情况（命令失效、文档与实际情况不符、不得不临时绕过错误等），请征得用户许可，然后运行

npx --yes langwatch report --user-approved

，并附上

--title

和

--summary

（或

--session <transcript.jsonl>

），将报告发送给 LangWatch 团队；这些报告会直接影响后续修复的内容。无需登录或提供 API key。未指定

--user-approved

时不会发送任何内容；

--dry-run

会在不发送的情况下打印准确的 payload。标题、摘要和 transcript 会先在本地按模式脱敏，包括 secrets、API keys、email addresses、phone numbers、card numbers 和 public IPv4 addresses。未匹配任何模式的内容将按原样发送，包括通过

--email

传入的 contact address。使用

--session

时，务必先运行

--dry-run

并让用户阅读 payload，因为 transcript 会包含用户可能从未审阅过的内容。

npx --yes langwatch report --help

介绍了这些选项。

Step 1: Find the Failing Run

第 1 步：查找失败的运行

bash

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

A run can fail two ways, and they triage differently:

Execution failure: the run errored out or stalled.
```
status
```
shows the error; jump to Step 4.
Score regression: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

bash

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

运行可能以两种方式失败，而两者的排查方式不同：

执行失败（Execution failure）：运行报错中断或停滞。
```
status
```
会显示错误；跳至第 4 步。
分数回退（Score regression）：运行已完成，但 evaluator 分数下降或数据行失败。继续执行第 2 步。

Step 2: Isolate the Failing Rows

第 2 步：隔离失败的数据行

bash

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

```
--filter failed
```
keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
```
--evaluator <name>
```
shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

bash

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

```
--filter failed
```
仅保留至少在一个 evaluator 上失败的数据行。应从这里入手，而不是从完整结果集开始。
```
--evaluator <name>
```
在运行了多个 evaluator 时显示某一个 evaluator 的列：判断回退是否集中于某个 evaluator（评分问题），还是分散在所有 evaluator 中（真实的行为回退）？

对于每个失败的数据行，记录输入、预期输出（来自 dataset）和实际输出。以相同方式失败的行指向同一个根因；以不同方式失败的行则表明可能存在 flakiness 或有噪声的 evaluator。

Step 3: Inspect the Evaluators

第 3 步：检查 evaluator

bash

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

Before blaming the agent, rule out the scorer:

LLM-judge evaluators: check the model in
```
settings
```
. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
Thresholds: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
Deterministic evaluators (exact match, JSON validity): these don't drift; failures here are real.

bash

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

在将问题归咎于 Agent 之前，先排除 scorer 的问题：

LLM-judge evaluator：检查
```
settings
```
中的 model。如果 judge model 发生变更、被限流，或相对于 rubric 来说能力太弱，就会产生与 Agent 无关的分数波动。
Thresholds：分数为 0.49、通过阈值为 0.5，属于临界 judge，并非回退。查看所有数据行的分数分布，而不要只看 pass/fail。
Deterministic evaluator（exact match、JSON validity）：这类 evaluator 不会漂移；这里的失败是真实问题。

Step 4: Root Cause

第 4 步：根因

Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)?
```
git log
```
on prompts and agent code usually answers this directly.
If rows fail on retrieval or context: inspect a production trace of the same path (
```
langwatch trace search
```
/
```
langwatch trace get
```
; see the
```
debug-with-langwatch
```
recipe).
If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use
```
langwatch dataset get <slugOrId>
```
to inspect it.
Apply the fix and re-run:

bash

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

将失败运行与最近一次通过的运行进行比较：发生了什么变化（prompt version、model、dataset、code）？通常可以直接对 prompt 和 Agent code 执行
```
git log
```
来回答。
如果数据行在 retrieval 或 context 上失败：检查同一路径的 production trace（
```
langwatch trace search
```
/
```
langwatch trace get
```
；参见
```
debug-with-langwatch
```
recipe）。
如果 dataset 本身看起来有问题（expected outputs 过期、存在错误数据行），请修复 dataset。使用
```
langwatch dataset get <slugOrId>
```
进行检查。
应用修复并重新运行：

bash

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

Step 5: Prevent the Recurrence

第 5 步：防止再次发生

If the failure mode wasn't covered by any evaluator, add one (
```
langwatch evaluator create
```
) and wire it into the experiment.
If it only shows up in production, set up a monitor (
```
langwatch monitor create
```
) so online evaluation catches it before the next experiment does.

如果任何 evaluator 都未覆盖该失败模式，请添加一个（
```
langwatch evaluator create
```
），并将其接入 experiment。
如果问题仅在 production 中出现，请设置 monitor（
```
langwatch monitor create
```
），让 online evaluation 在下一次 experiment 再次出现问题之前捕获它。