Back to Details

verify

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Verify

验证

Every change ships with evidence from the real thing. Unit tests passing is not evidence that the CLI prints the right line, a locator finds the node on a real page, or an agent step replays. The testbed and the benchmarks are the app here: they consume the built packages the way a user does. Use this skill in the loop, not only at the end: change, build, run, look, repeat.
每次更改都必须以真实运行结果作为证据。单元测试通过并不能证明 CLI 会打印 正确的输出行、locator 能在真实页面上找到目标节点，或 agent 步骤能够成功重放。 这里的 testbed 和 benchmarks 就相当于实际应用：它们会像真实用户一样使用已构建的包。 要在整个迭代过程中使用此 skill，而不只是到最后才用：更改、构建、运行、查看、 重复。

Setup

设置

bash
pnpm install --frozen-lockfile
pnpm --filter @e2e-dev/web exec playwright install chromium
pnpm build                    # every consumer below runs dist; rebuild after each change
A stale 
dist
is the most common false result. Rebuild before trusting any run, and after switching branches.
bash
pnpm install --frozen-lockfile
pnpm --filter @e2e-dev/web exec playwright install chromium
pnpm build                    # every consumer below runs dist; rebuild after each change
过期的 
dist
是最常见的“假结果”来源。在相信任何运行结果前先重新构建， 切换分支后也要重新构建。

Pick the surface

选择验证范围

ChangeRunProves
Runner, collect, locator, config, sessions, CLI flagsthe testbed suite, or the file covering itThe built CLI end to end on a deterministic app
Reporters (list, json, markdown, junit)the testbed file plus 
test:stress
The printed and written output, hostile titles included
@e2e-dev/web
the testbed file, then the web benchmark's 
tests/
Real Chromium on plain and hard surfaces
Agent (
src/agent/
), prompts, trace cache		the web benchmark's 
tests-agent/
replay, then 
--no-cache --ai-trace
Replay still hits, and the live agent still reaches the goal
@e2e-dev/mobile
the mobile benchmark on a simulator or emulatorReal devices; see 
apps/mobile-benchmark
e2e mcp
the 
e2e
MCP server on the testbed		The tools as a coding agent sees them
e2e init
, packaging, the skill		packed tarballs in a scratch project under 
/tmp
What a new user gets from npm
Docs pages
pnpm docs:dev
The rendered page
@e2e-dev/github
the PR's own 
e2e-github
comments		The comment a user's PR gets
Commands (run from the app directory; call the CLI directly, since 
pnpm run <script> -- --flag
forwards the 
--
and the CLI reads the flag as a file):
bash
cd apps/testbed && node node_modules/e2e/dist/cli/bin.js run tests/todos.e2e.ts
cd apps/web-benchmark && pnpm run build     # the Next.js app, once per scenario change
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run tests/<scenario>.e2e.ts
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run --config e2e.agent.config.ts
pnpm test:testbed && pnpm test:web-benchmark          # the whole gate, as CI runs it
Add a deterministic test next to the behavior when it must stay proven: a testbed page plus test for a runner feature, a benchmark scenario for a hard surface (keep scenario diffs minimal, never fix a planted bug). Tests follow skills/e2e, the consumer skill; this repo is its first consumer.
Agentic runs need 
AI_GATEWAY_API_KEY
. The benchmark agent suites replay their committed recordings and call the model only for a step with none, as CI does; a step that spends a model call there means its recording went stale. Go live with 
--no-cache --ai-trace
only when the change is to the agent, and read the trace with unbox-ai (
compare
main against the branch), never by opening it. Re-recorded entries are committed in the same PR ("Committed recordings" in 
AGENTS.md
).
更改运行证明
Runner、collect、locator、config、session、CLI flagtestbed 测试套件，或覆盖该功能的测试文件已构建 CLI 在确定性应用上完成端到端运行
Reporter（list、json、markdown、junit）testbed 测试文件以及 
test:stress
打印和写入的输出，包括包含恶意标题时的表现
@e2e-dev/web
testbed 测试文件，然后运行 web benchmark 的 
tests/
真实 Chromium 在普通界面和高难度界面上的表现
Agent（
src/agent/
）、prompt、trace cache		web benchmark 的 
tests-agent/
重放，然后使用 
--no-cache --ai-trace
重放仍能命中目标，实时 agent 仍能达到目标
@e2e-dev/mobile
在 simulator 或 emulator 上运行 mobile benchmark真实设备；参见 
apps/mobile-benchmark
e2e mcp
在 testbed 上运行 
e2e
MCP server		coding agent 所能看到的工具
e2e init
、打包、skill		在 
/tmp
下的临时项目中检查打包后的 tarball		新用户从 npm 获得的内容
Docs 页面
pnpm docs:dev
渲染后的页面
@e2e-dev/github
PR 自身的 
e2e-github
评论		用户 PR 会收到的评论
以下命令从 app 目录运行；请直接调用 CLI，因为 
pnpm run <script> -- --flag
会转发 
--
，而 CLI 会将该 flag 读取为文件：
bash
cd apps/testbed && node node_modules/e2e/dist/cli/bin.js run tests/todos.e2e.ts
cd apps/web-benchmark && pnpm run build     # the Next.js app, once per scenario change
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run tests/<scenario>.e2e.ts
cd apps/web-benchmark && node node_modules/e2e/dist/cli/bin.js run --config e2e.agent.config.ts
pnpm test:testbed && pnpm test:web-benchmark          # the whole gate, as CI runs it
如果某项行为必须持续得到证明，请在对应行为旁添加确定性测试：runner 功能需添加 testbed 页面及测试，复杂界面需添加 benchmark 场景（尽量减少场景差异，绝不要修复刻意植入的 bug）。测试应遵循 skills/e2e，即 consumer skill；本仓库是它的首个 consumer。
Agent 运行需要 
AI_GATEWAY_API_KEY
。benchmark agent 测试套件会重放仓库中已提交的录制内容；与 CI 一样，只有某一步没有录制内容时才会调用模型。如果某一步在这里消耗了模型调用，就说明其录制内容已过期。仅当更改涉及 agent 时，才使用 
--no-cache --ai-trace
进行实时运行，并通过 unbox-ai 读取 trace（将 main 与分支进行对比），不要直接打开 trace。重新录制的条目须在同一 PR 中提交（见 
AGENTS.md
中的“Committed recordings”）。

The MCP server

MCP server

.mcp.json
registers 
e2e mcp
on the testbed. 
open_session
, then 
call {tool: "observe"}
, 
locate
, 
screenshot
, and the action verbs. Pass 
config: "../web-benchmark/e2e.config.ts"
to 
open_session
for a benchmark. The server process runs the 
dist
it started with: after a rebuild, restart it (
/mcp
in Claude Code) before verifying MCP or engine changes through it. Several sessions can be open at once (
--max-sessions
, default 4); pass the session id to every call and 
close_session
when done.
Use it to write locators (
locate
prints the 
screen.*
call), to look at a surface before testing it, to verify changes to the MCP tools themselves, and to record a video of what you drove when the engine records video (both web and mobile do): 
start_recording
once the screen is set up, 
stop_recording
for the file paths.
.mcp.json
为 testbed 注册了 
e2e mcp
。先调用 
open_session
，再调用 
{tool: "observe"}
、
locate
、
screenshot
和各个 action verb。调用 
open_session
时传入 
config: "../web-benchmark/e2e.config.ts"
即可使用 benchmark。server 进程运行的是它启动时所使用的 
dist
：重新构建后，需要重启进程（在 Claude Code 中执行 
/mcp
），然后再通过它验证 MCP 或 engine 更改。
可以同时打开多个 session（
--max-sessions
，默认值为 4）；每次调用都要传入 session id，完成后调用 
close_session
。
可用它编写 locator（
locate
会打印 
screen.*
调用）、在测试界面前先查看该界面、验证 MCP 工具自身的更改，并在 engine 能够录制视频时录制实际操作过程的视频（web 和 mobile 均支持）：界面准备完毕后调用一次 
start_recording
，再调用 
stop_recording
获取文件路径。

Drive it like a user

像用户一样操作

  • Run the built CLI, not a unit test harness. Read what it prints and what it writes (
    .e2e/report.json
    , 
    --reporter list,markdown
    for 
    .e2e/summary.md
    and 
    .e2e/failures/
    ).
  • Exercise the failure path too: a change to an error code, a timeout, or a policy refusal is verified by triggering it and reading the message a user gets.
  • Check the side effects: artifacts land where the docs say, and a filled secret stays out of every file under 
    .e2e/
    (
    grep -r
    the value).
  • APP_ALREADY_RUNNING
    means another checkout's run holds the app port. Wait for it or ask; never stop a process you did not start.
  • Mobile: follow 
    apps/mobile-benchmark/README.md
    for devices. One run per device at a time.
  • 运行构建后的 CLI，而不是单元测试 harness。查看它打印和写入的内容（
    .e2e/report.json
    、使用 
    --reporter list,markdown
    生成 
    .e2e/summary.md
    和 
    .e2e/failures/
    ）。
  • 也要测试失败路径：对错误码、超时或策略拒绝逻辑的更改，应通过实际触发并阅读用户看到的消息来验证。
  • 检查副作用：产物是否写入文档指定位置，以及填入的 secret 是否未出现在 
    .e2e/
    下的任何文件中（使用 
    grep -r
    搜索该值）。
  • APP_ALREADY_RUNNING
    表示另一个 checkout 的运行任务占用了应用端口。等待它结束或询问相关人员；绝不停止并非由你启动的进程。
  • Mobile：按照 
    apps/mobile-benchmark/README.md
    中的设备说明操作。每台设备同一时间只能运行一次。

Evidence

证据

The PR must show it working (
writing-pr
). Report the surface, what you ran, and what you saw, with media:
  • Terminal output: paste the relevant lines of the CLI output as a fenced 
    text
    block (ANSI stripped, paths trimmed). This is the primary evidence for runner, CLI, and reporter changes.
  • Video: 
    --video
    (add 
    --headed
    to watch it) records every attempt to 
    .e2e/artifacts/<target>/<test>/.../attempt-<n>/video/
    : 
    video.webm
    from 
    @e2e-dev/web
    , 
    video.mp4
    from 
    @e2e-dev/mobile
    . Attach it for engine, locator, and agent changes a viewer can see.
  • Stills: cut a frame from the video (
    ffmpeg -ss <seconds> -i <video> -frames:v 1 after.png
    ), or screenshot the docs page.
  • Bug fixes, main vs branch: write the regression test first and run it while the checkout still matches main, so the failure is captured. Already changed the code? Build main beside it: 
    git worktree add /tmp/e2e-main origin/main
    , then 
    pnpm install --frozen-lockfile && pnpm build
    there and run the same command in both.
  • Upload: 
    gh pr create|edit --body-file body.md --attach ./after.png --attach ./video.webm
    (
    gh
    2.99+) uploads each file, rewrites markdown image paths in the body, and appends videos as players. With an older 
    gh
    , drag the files into the PR in the browser, or say the media is local and why.
If something could not be verified, say so and name the blocker. A confident claim without evidence is worse than "inconclusive".
PR 必须展示功能确实可用（
writing-pr
）。说明验证范围、运行了什么以及观察到了什么，并附上媒体：
  • 终端输出： 将 CLI 输出中的相关行粘贴到带 
    text
    围栏的代码块中（去除 ANSI 颜色代码并裁剪路径）。这是 runner、CLI 和 reporter 更改的主要证据。
  • 视频： 
    --video
    会将每次尝试录制到 
    .e2e/artifacts/<target>/<test>/.../attempt-<n>/video/
    （添加 
    --headed
    可观看运行过程）：
    @e2e-dev/web
    生成 
    video.webm
    ，
    @e2e-dev/mobile
    生成 
    video.mp4
    。对于用户可见的 engine、locator 和 agent 更改，应附上视频。
  • 静帧： 从视频中截取一帧（
    ffmpeg -ss <seconds> -i <video> -frames:v 1 after.png
    ），或截取 docs 页面。
  • Bug 修复，main 对比分支： 先编写回归测试，并在 checkout 仍与 main 一致时运行它，以捕获失败。已经修改了代码？如果是，则在旁边构建 main：执行 
    git worktree add /tmp/e2e-main origin/main
    ，然后在该目录中运行 
    pnpm install --frozen-lockfile && pnpm build
    ，并在两边运行同一条命令。
  • 上传： 
    gh pr create|edit --body-file body.md --attach ./after.png --attach ./video.webm
    （
    gh
    2.99+）会上传每个文件、重写正文中的 markdown 图片路径，并将视频作为播放器追加。对于较旧的 
    gh
    ，可在浏览器中将文件拖入 PR，或说明媒体仅保存在本地以及原因。
如果某项内容无法验证，请明确说明并指出阻碍因素。没有证据的笃定结论比“结果不确定”更糟糕。

Cleanup

清理

e2e run
stops the apps it started. 
close_session
stops the MCP session's. Remove a 
/tmp/e2e-main
worktree with 
git worktree remove
when done.
e2e run
会停止它启动的应用，
close_session
会停止 MCP session 对应的应用。完成后，使用 
git worktree remove
删除 
/tmp/e2e-main
worktree。