Verify

Every change ships with evidence from the real thing. Unit tests passing is not evidence that the CLI prints the right line, a locator finds the node on a real page, or an agent step replays. The testbed and the benchmarks are the app here: they consume the built packages the way a user does. Use this skill in the loop, not only at the end: change, build, run, look, repeat.

Setup

bash

pnpm install --frozen-lockfile
pnpm --filter @e2e-dev/web exec playwright install chromium
pnpm build                    # every consumer below runs dist; rebuild after each change

A stale

dist

is the most common false result. Rebuild before trusting any run, and after switching branches.

Pick the surface

Change	Run	Proves
Runner, collect, locator, config, sessions, CLI flags	the testbed suite, or the file covering it	The built CLI end to end on a deterministic app
Reporters (list, json, markdown, junit)	the testbed file plus `test:stress`	The printed and written output, hostile titles included
`@e2e-dev/web`	the testbed file, then the web benchmark's `tests/`	Real Chromium on plain and hard surfaces
Agent ( `src/agent/` ), prompts, trace cache	the web benchmark's `tests-agent/` replay, then `--no-cache --ai-trace`	Replay still hits, and the live agent still reaches the goal
`@e2e-dev/mobile`	the mobile benchmark on a simulator or emulator	Real devices; see `apps/mobile-benchmark`
`e2e mcp`	the `e2e` MCP server on the testbed	The tools as a coding agent sees them
`e2e init` , packaging, the skill	packed tarballs in a scratch project under `/tmp`	What a new user gets from npm
Docs pages	`pnpm docs:dev`	The rendered page
`@e2e-dev/github`	the PR's own `e2e-github` comments	The comment a user's PR gets

Commands (run from the app directory; call the CLI directly, since

pnpm run <script> -- --flag

forwards the

--

and the CLI reads the flag as a file):

bash

cd apps/testbed && node node_modules/e2e/dist/cli/bin.js run tests/todos.e2e.ts
cd apps/web-benchmark && pnpm run build     # the Next.js app, once per scenario change
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run tests/<scenario>.e2e.ts
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run --config e2e.agent.config.ts
pnpm test:testbed && pnpm test:web-benchmark          # the whole gate, as CI runs it

Add a deterministic test next to the behavior when it must stay proven: a testbed page plus test for a runner feature, a benchmark scenario for a hard surface (keep scenario diffs minimal, never fix a planted bug). Tests follow skills/e2e, the consumer skill; this repo is its first consumer.

Agentic runs need

AI_GATEWAY_API_KEY

. The benchmark agent suites replay their committed recordings and call the model only for a step with none, as CI does; a step that spends a model call there means its recording went stale. Go live with

--no-cache --ai-trace

only when the change is to the agent, and read the trace with unbox-ai (

compare

main against the branch), never by opening it. Re-recorded entries are committed in the same PR ("Committed recordings" in

AGENTS.md

The MCP server

.mcp.json

registers

e2e mcp

on the testbed.

open_session

, then

call {tool: "observe"}

locate

screenshot

, and the action verbs. Pass

config: "../web-benchmark/e2e.config.ts"

open_session

for a benchmark. The server process runs the

dist

it started with: after a rebuild, restart it (

/mcp

in Claude Code) before verifying MCP or engine changes through it. Several sessions can be open at once (

--max-sessions

, default 4); pass the session id to every call and

close_session

when done.

Use it to write locators (

locate

prints the

screen.*

call), to look at a surface before testing it, to verify changes to the MCP tools themselves, and to record a video of what you drove when the engine records video (both web and mobile do):

start_recording

once the screen is set up,

stop_recording

for the file paths.

Drive it like a user

Run the built CLI, not a unit test harness. Read what it prints and what it writes (
```
.e2e/report.json
```
,
```
--reporter list,markdown
```
for
```
.e2e/summary.md
```
and
```
.e2e/failures/
```
).
Exercise the failure path too: a change to an error code, a timeout, or a policy refusal is verified by triggering it and reading the message a user gets.
Check the side effects: artifacts land where the docs say, and a filled secret stays out of every file under
```
.e2e/
```
(
```
grep -r
```
the value).
```
APP_ALREADY_RUNNING
```
means another checkout's run holds the app port. Wait for it or ask; never stop a process you did not start.
Mobile: follow
```
apps/mobile-benchmark/README.md
```
for devices. One run per device at a time.

Evidence

The PR must show it working (

writing-pr

). Report the surface, what you ran, and what you saw, with media:

Terminal output: paste the relevant lines of the CLI output as a fenced
```
text
```
block (ANSI stripped, paths trimmed). This is the primary evidence for runner, CLI, and reporter changes.
Video:
```
--video
```
(add
```
--headed
```
to watch it) records every attempt to
```
.e2e/artifacts/<target>/<test>/.../attempt-<n>/video/
```
:
```
video.webm
```
from
```
@e2e-dev/web
```
,
```
video.mp4
```
from
```
@e2e-dev/mobile
```
. Attach it for engine, locator, and agent changes a viewer can see.
Stills: cut a frame from the video (
```
ffmpeg -ss <seconds> -i <video> -frames:v 1 after.png
```
), or screenshot the docs page.
Bug fixes, main vs branch: write the regression test first and run it while the checkout still matches main, so the failure is captured. Already changed the code? Build main beside it:
```
git worktree add /tmp/e2e-main origin/main
```
, then
```
pnpm install --frozen-lockfile && pnpm build
```
there and run the same command in both.
Upload:
```
gh pr create|edit --body-file body.md --attach ./after.png --attach ./video.webm
```
(
```
gh
```
2.99+) uploads each file, rewrites markdown image paths in the body, and appends videos as players. With an older
```
gh
```
, drag the files into the PR in the browser, or say the media is local and why.

If something could not be verified, say so and name the blocker. A confident claim without evidence is worse than "inconclusive".

Cleanup

e2e run

stops the apps it started.

close_session

stops the MCP session's. Remove a

/tmp/e2e-main

worktree with

git worktree remove

when done.

verify

NPX Install

Tags

SKILL.md Content