Evidence-Driven Testing

Record annotated proof of behavior, then attach it to the PR and tracker issue.

The recording is the capture of you testing the app via computer use: start the recorder, then drive the app yourself — click, type, navigate — through each test target. Every action in the video is the test being performed live; the recording has no value as evidence unless it shows that interactive session. If the harness has no computer-use tools but a GUI exists, drive the app with

cua-driver

instead (see below) — it is still your live session.

The bundled recorder,

scripts/evidence.py

, captures the display with FFmpeg on Linux, macOS, and Windows, timestamps each annotation you add while testing, burns them into the video on stop, and verifies the result with ffprobe. It writes

evidence.mp4

report.md

, and

manifest.json

into the session folder.

Inputs

Test targets (required): The behaviors/flows to verify, phrased as testable statements.
PR / issue (optional): Where to post the evidence. If omitted, deliver to the requester only.

The recorder

EVIDENCE

below means the path to

scripts/evidence.py

inside this skill's folder (wherever the skill is installed, e.g.

~/.claude/skills/evidence-driven-testing/scripts/evidence.py

). It needs only Python 3 and FFmpeg.

Check first:
```
python3 $EVIDENCE doctor
```
— verifies
```
ffmpeg
```
,
```
ffprobe
```
,
```
libx264
```
, and the
```
ass
```
filter (
```
ready
```
), then which screen-capture source works on this machine (
```
capture_ready
```
and the source
```
auto
```
will pick). Exits non-zero only when the toolchain is missing; read
```
capture_ready
```
before recording.

Platforms (

--source auto

picks the first available):

OS	Source	Needs
Linux X11 / XWayland	`x11` (x11grab)	`DISPLAY` set
Linux Wayland	`wayland` (wf-recorder)	`WAYLAND_DISPLAY` set, `wf-recorder` on PATH, and a compositor confirmed to support wlr-screencopy — either a known wlroots one (Sway, Hyprland, river, Wayfire, labwc, dwl, niri) or verified via `wayland-info` . GNOME and KDE Wayland are not capturable this way; `doctor` says so. Use `x11` through XWayland for X11 apps, or a fallback recorder. `--source wayland` still forces it
macOS	`avfoundation`	Screen Recording permission granted to the terminal / agent host app; `doctor` lists screen indexes for `--screen-index`
Windows	`gdigrab`	any standard ffmpeg build; `powershell` for process checks

Capture is the full screen by default;

--geometry WxH

and

--offset X,Y

crop a region on x11, wayland, and gdigrab. XWayland only sees X11 windows, so prefer the

wayland

source when the app under test is Wayland-native.

Crash-safe raw capture: the raw recording is MPEG-TS (
```
raw.ts
```
), so if the recorder is killed hard or crashes, what was captured still probes and renders.
```
stop
```
remuxes or re-encodes it into a standard
```
evidence.mp4
```
.
How stopping works: on Linux the recorder is signalled through a pidfd. On macOS and Windows
```
start
```
launches a small supervisor process that owns the ffmpeg child, and
```
stop
```
asks it (via
```
stop.request
```
in the session folder) to interrupt, then terminate, then kill; it writes
```
recorder-exit.json
```
when done. Nothing ever signals a bare PID, so a recycled PID can't be hit. If the supervisor dies while the recorder is still running,
```
stop
```
refuses and tells you which PID to stop by hand. If it died before it could even record which process it started,
```
stop
```
stays blocked until you have checked for a stray recorder yourself and rerun it with
```
--accept-untracked-recorder
```
; the report then carries that caveat.
Fallback recorders when
```
doctor
```
reports no capture source (for example Wayland without wf-recorder):
```
cua-driver recording start <dir>
```
/
```
stop
```
(see the cua-driver section), or the OS recorder (macOS:
```
screencapture -v out.mov
```
). On these paths there is no annotation overlay, so keep the annotation protocol as files — an
```
assertions.md
```
listing each
```
setup
```
/
```
test_start
```
/
```
assertion
```
with its result and the approximate video timestamp, exactly as in the headless path.
Never present
```
--source test
```
(the synthetic pattern generator) as UI evidence. It exists to smoke-test the toolchain; the repo's
```
tests/test_evidence.py
```
exercises it.

Instructions

1. Prepare the screen

Maximize the browser/app window; close popups, notifications, and extra panels.
Navigate to the starting state (logged in, correct page) BEFORE recording, unless setup itself is under test.
Note the exact revision under test:
```
git rev-parse HEAD
```
and
```
git branch --show-current
```
(or the deployment URL) — the recorder stamps them into the report.

2. Start recording

Begin the screen recording before the first meaningful action:

bash

python3 $EVIDENCE start \
  --output .artifacts/<task-name> \
  --title "<what is being verified>" \
  --commit "$(git rev-parse HEAD)" --branch "$(git branch --show-current)" \
  --environment "<OS / browser / display / deployment>"

It prints JSON with a

session

path and the chosen

source

; keep the path (

SESSION=...

) for every later command. The source is auto-detected and the whole screen is captured; pass

--source

--geometry

--offset

--display

--xauthority

(X11),

--screen-index

(macOS), or

--output-name

(Wayland) only when

doctor

or the situation calls for it.

Add a

setup

annotation describing the starting context:

bash

python3 $EVIDENCE annotate "$SESSION" --type setup \
  --message "Logged in, navigating to connectors page"

3. Test via computer use, annotating as you go

Perform every interaction through computer use on the live app — the recording captures your session, so the testing and the evidence are the same act. Work at a watchable pace: let the UI settle after each action so state changes are visible on video.

At each named test's start, add a

test_start

annotation in Jest style:

bash

python3 $EVIDENCE annotate "$SESSION" --type test_start \
  --message "It should execute the tool directly when permission is 'always'"

After each check, add an

assertion

annotation with

--result passed

failed

, or

untested

bash

python3 $EVIDENCE annotate "$SESSION" --type assertion --result passed \
  --message "Tool ran without a permission prompt"

Rules for assertions:
- One assertion per meaningful state change — consolidate, don't annotate per UI label.
- Use "Precondition: ..." assertions to establish starting state.
- Keep under 80 characters, high-signal (the recorder rejects longer messages).
- If a test cannot run (missing prerequisite, expired auth window), mark it
```
untested
```
  with the reason — never skip silently.
- The timestamp records when you asserted, not whether it was true — look at the screen before choosing
```
passed
```
  .

4. Stop and review

Stop recording after the final assertion:
bash
```
python3 $EVIDENCE stop "$SESSION"
```
This stops the capture (gracefully, so the recorder flushes; escalating only if it ignores the request), burns the annotations into
```
evidence.mp4
```
, probes the result, and writes
```
report.md
```
and
```
manifest.json
```
next to it. It prints
```
"verified": true
```
on success; if rendering fails the session is marked
```
finalization_failed
```
— fix the reported cause and run
```
stop
```
again (a retry does not signal the recorder twice). If the recorder process can no longer be signalled safely (it died, or its PID now belongs to another process), the session is marked
```
recorder_lost
```
. Running
```
stop
```
again finalizes whatever video was captured, but only once that recorder process is confirmed gone — if it is still alive, stop it first, or the video would be rendered while still being written.
Confirm the recording captured the key moments before sharing: extract a frame at each assertion timestamp (
```
ffmpeg -ss <t> -i evidence.mp4 -frames:v 1 frame.png
```
) and check the state and the label are visible.
Fill in the Caveats section of
```
report.md
```
; never leave the placeholder.

5. Post the evidence

```
report.md
```
is the report: what was tested, environment + exact commit, pass/fail per test, caveats. Extend it rather than rewriting from scratch.
Post the video + summary as a PR comment (embed in the PR description if it's your PR).
```
gh pr comment
```
cannot attach a local video — upload
```
evidence.mp4
```
through the PR's comment box in an authenticated browser, or upload it to a host and link it (for example the
```
before-and-after
```
upload adapters). Reopen the comment and confirm the video plays before claiming it is posted.
Attach the same video to the tracker issue (Linear/Jira) with a one-line result.
Send the report + recording to the requester.

Guardrails

The video must show the actual test session being driven live. Never present scripted playback, stitched clips, or synthetic footage as a recording; if the harness lacks computer-use tools but a GUI exists, drive via
```
cua-driver
```
; with no GUI at all, use the headless path instead.
Never record a half-covered or tiled window — maximize first.
Never record a screen showing secrets, tokens, customer data, or payment details; if a flow requires them, mark it
```
untested
```
and say why.
When verifying a fix, show or reference the old failure alongside the new success.
Always state the exact commit/branch/deployment tested against.

No computer-use tools? Drive with cua-driver (GUI available)

When a display exists but the agent has no built-in computer-use capability, use cua-driver (macOS / Windows / Linux) as the actuator. It is still you testing the app live — the recording rule holds unchanged; only the input mechanism differs.

Verify the setup with
```
cua-driver doctor
```
before recording. If a
```
cua-driver
```
skill is installed, read it and follow its protocol — the snapshot-before-action invariant is mandatory.
Loop per interaction:
```
launch_app
```
→
```
get_window_state
```
(accessibility tree
- screenshot) → act via
```
element_token
```
  (
```
click
```
  ,
```
type_text
```
  ,
```
press_key
```
  ) →
```
verify_state
```
  for the expected postcondition. Each
```
verify_state
```
  check maps 1:1 onto an
```
assertion
```
  annotation.
Wherever
```
doctor
```
reports
```
capture_ready: yes
```
, keep using the bundled recorder above for the video and the annotations; cua-driver only supplies the input.
Otherwise,
```
cua-driver recording start <output-dir>
```
/
```
cua-driver recording stop
```
is the recorder (the output directory is required, and the daemon must be running:
```
cua-driver serve
```
). Video capture is on by default and is finalized to
```
<output-dir>/recording.mp4
```
on stop — but on Windows/Linux it shells out to ffmpeg, so a missing ffmpeg or display yields only the per-turn trajectory folders (before/after screenshots,
```
action.json
```
,
```
click.png
```
), no video. After stopping, verify
```
recording.mp4
```
exists before citing it; if it is absent, fix the recorder or present the per-turn before/after screenshots as numbered captures per the headless protocol.
If no annotation overlay is available on this path, keep the protocol as files: an
```
assertions.md
```
listing each
```
test_start
```
/
```
assertion
```
with its result, exactly as in the headless path.

Headless path (no GUI available)

When the agent has no desktop to record, keep the same assertion discipline; swap the recorder for scripted capture:

Save everything to
```
.artifacts/<task-name>/
```
(gitignore it — evidence gets uploaded, never committed). Keep the capture script beside the captures so the run is repeatable.
Screenshots: the
```
before-and-after
```
CLI (
```
@vercel/before-and-after
```
) captures URLs or elements and its pairs feed PR embeds directly. In containers/VMs where Chrome fails with "No usable sandbox", set
```
AGENT_BROWSER_ARGS="--no-sandbox"
```
.

Video / multi-step flows: a one-off Playwright script, run without adding playwright to the project's dependencies:

bash

npx --yes --package=playwright node record.mjs

(Plain

npx playwright node record.mjs

fails —

node

is not a Playwright CLI command;

--package=playwright

is what puts the module on the path.) Minimal

record.mjs

import { chromium } from "playwright";
const browser = await chromium.launch();
const context = await browser.newContext({
  recordVideo: { dir: ".artifacts/<task-name>/" },
});
const page = await context.newPage();
await page.goto("http://localhost:3000/path-under-test");
// ...drive the flow, one meaningful state change per step...
await context.close(); // finalizes the .webm
await browser.close();

Trim or compress with ffmpeg if the file is large.

The annotation protocol becomes files: number captures in test order with the assertion in the name —
```
01-precondition-signed-in.png
```
,
```
02-it-saves-on-blur-passed.png
```
— and keep an
```
assertions.md
```
in the artifacts folder listing each
```
test_start
```
/
```
assertion
```
with its result (
```
passed
```
/
```
failed
```
/
```
untested
```
+ reason).

Non-UI changes still need evidence

API / performance: a scripted probe with measured numbers — request counts per phase, latency before/after — captured to
```
probe-output.txt
```
.
Rendering / canvas / shader: rendered frames plus pixel assertions (diff values), reviewed by eye and saved as PNGs.
Agent behavior: the relevant transcript excerpt showing the tool call and response.
Bug fixes: reproduce and capture the failure before writing the fix — that capture is the "before" half of a before/after pair.

Capture hygiene

Confirm the server you're probing is running your code (right port, right process), especially when multiple agents share a machine:
```
lsof -i :<port>
```
— or where
```
lsof
```
isn't installed,
```
ss -ltnp "sport = :<port>"
```
to find the listener's PID, then
```
ps -p <pid> -o args=
```
to confirm it's yours.
Evidence complements the repo's checks (typecheck/build/tests); it never replaces them.
Hand before/after media pairs to a before/after tool for the PR embed (e.g.
```
before-and-after before.png after.png --markdown
```
).

evidence-driven-testing

NPX Install

Tags

SKILL.md Content

Evidence-Driven Testing

Inputs

The recorder

Instructions

1. Prepare the screen

2. Start recording

3. Test via computer use, annotating as you go

4. Stop and review

5. Post the evidence

Guardrails

No computer-use tools? Drive with cua-driver (GUI available)

Headless path (no GUI available)

Non-UI changes still need evidence

Capture hygiene