eval-triage

Original：🇺🇸 English

Translated

Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.

10installs

Sourcelangwatch/skills

Added on2026-09-27

NPX Install

npx skill4agent add langwatch/skills eval-triage

SKILL.md Content

View Translation Comparison →

Triage Failing Experiments and Evaluations

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

Prerequisites

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Every list command takes

--limit <n>

to cap the rows and

--jq <expr>

to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (

--jq '.traces | length'

), and read how many there are in all at

.pagination.total

. Bare

--jq length

counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx --yes langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without

--user-approved

, and

--dry-run

prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with

--email

. With

--session

, always run

--dry-run

first and let the user read the payload, because a transcript carries content they never reviewed.

npx --yes langwatch report --help

explains the options.

Step 1: Find the Failing Run

bash

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

A run can fail two ways, and they triage differently:

Execution failure: the run errored out or stalled.
```
status
```
shows the error; jump to Step 4.
Score regression: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

Step 2: Isolate the Failing Rows

bash

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

```
--filter failed
```
keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
```
--evaluator <name>
```
shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

Step 3: Inspect the Evaluators

bash

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

Before blaming the agent, rule out the scorer:

LLM-judge evaluators: check the model in
```
settings
```
. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
Thresholds: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
Deterministic evaluators (exact match, JSON validity): these don't drift; failures here are real.

Step 4: Root Cause

Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)?
```
git log
```
on prompts and agent code usually answers this directly.
If rows fail on retrieval or context: inspect a production trace of the same path (
```
langwatch trace search
```
/
```
langwatch trace get
```
; see the
```
debug-with-langwatch
```
recipe).
If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use
```
langwatch dataset get <slugOrId>
```
to inspect it.
Apply the fix and re-run:

bash

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

Step 5: Prevent the Recurrence

If the failure mode wasn't covered by any evaluator, add one (
```
langwatch evaluator create
```
) and wire it into the experiment.
If it only shows up in production, set up a monitor (
```
langwatch monitor create
```
) so online evaluation catches it before the next experiment does.

eval-triage

NPX Install

Tags

SKILL.md Content

Triage Failing Experiments and Evaluations

Prerequisites

Step 1: Find the Failing Run

Step 2: Isolate the Failing Rows

Step 3: Inspect the Evaluators

Step 4: Root Cause

Step 5: Prevent the Recurrence