add-your-agent

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Add your agent to Veris Bench

将 Agent 添加到 Veris Bench

Take the agent in this repository to a registered Your image candidate on a Veris Bench bench, and prove it works with a smoke trial.

The agent is the constant. Add one server file, a Dockerfile and a candidate file beside the agent. Change the agent's own code only when there is no other way, and say so in your report.

A smoke attempt that is

completed

does not prove the integration works. An agent whose database calls all fail, or whose model calls the grader cannot see, still completes. Step 9 says what to check.

Read

reference/api.md

before step 1 and

reference/image.md

before step 2.

将本仓库中的 Agent 接入 Veris Bench，注册为 Your image 候选，并通过 smoke trial 证明其可用。

Agent 本身应保持不变。 在 Agent 旁边添加一个 server 文件、一个 Dockerfile 和一个 candidate 文件。仅在其他方法均不可行时才修改 Agent 自身的代码，并在报告中说明。

仅凭 smoke 尝试的状态为

completed

并不能证明集成成功。即使 Agent 的数据库调用全部失败，或 model 调用对评分器不可见，它仍可能显示为

completed

。第 9 步说明了需要检查的内容。

在执行第 1 步前阅读

reference/api.md

，在执行第 2 步前阅读

reference/image.md

。

Inputs

输入

Collect these before you change anything. Ask the user for what is missing.

Input	Where it comes from
`VERIS_API_KEY`	Environment variable, starts with `vbk_` . Only a person can issue one: console, Overview, API keys, Issue. Never print it or write it to a file.
`BENCH_API`	Default `https://bench.api.veris.ai/v1` . Use another only when the user gives one.
`BENCH_ID`	The skill's argument, if the user gave one. Otherwise ask, or list their benches and have them choose.
Registry	A repository prefix Docker can push to, such as `ghcr.io/<org>` .
Model keys	The secrets the agent reads, such as `OPENAI_API_KEY` . They go in the bench, never in the image or the repository. The user adds a missing one in the console, unless it is already in your environment.

在修改任何内容之前，先收集以下信息。缺少的信息应向用户询问。

输入	来源
`VERIS_API_KEY`	环境变量，以 `vbk_` 开头。只能由本人签发：console、Overview、API keys、Issue。绝不打印该密钥，也不要将其写入文件。
`BENCH_API`	默认为 `https://bench.api.veris.ai/v1` 。仅当用户提供其他地址时才使用其他值。
`BENCH_ID`	此 skill 的参数。如果用户提供了，则直接使用；否则应询问用户，或者列出其 benches 并让用户选择。
Registry	Docker 可以推送到的 repository 前缀，例如 `ghcr.io/<org>` 。
Model keys	Agent 读取的 secrets，例如 `OPENAI_API_KEY` 。它们应配置在 bench 中，绝不能放在 image 或 repository 中。缺失的密钥由用户在 console 中添加，除非当前环境中已经存在。

Steps

步骤

Check access. Read the bench.
```
access
```
must be
```
write
```
. If it is
```
read
```
, stop and tell the user why (
```
reference/api.md
```
, "Access").
Read the agent, without editing. Write down: language and entry point; the function that runs one conversation turn and how it keeps history; the model SDK and its installed version; every external service it calls and where each address or DSN comes from (variable, config file or a literal); every secret it reads; and files it opens by a relative path. Check the SDK against "Telemetry" in
```
reference/image.md
```
now: an unsupported version records no model calls and says nothing.
Read the bench's world. List its services (
```
reference/api.md
```
, "World"). Match each service the agent calls to a twin. Tell the user about any service the agent calls that has no twin: those calls reach the real vendor, unless Veris turned on the gateway for the bench.
Add the server file,
```
bench_server.py
```
or
```
bench_server.mjs
```
, from the pattern in
```
reference/image.md
```
. It wraps the agent's turn function, keeps one conversation per session id, serves
```
/health
```
, and listens on
```
0.0.0.0:8008
```
. Use http. Use ws only if the agent already speaks WebSocket.
Point the agent at the twins, in the server file. Before the agent's modules load, copy each twin's address from
```
VERIS_DS_SERVICE_URLS
```
into the setting the agent already reads. Apply the client fixes in
```
reference/image.md
```
, "Twins".
Write the Dockerfile and
.dockerignore
from
```
reference/image.md
```
. If the repository has a Dockerfile, reuse it when it can run the server.
Check locally. Build for
```
linux/amd64
```
, run it, and send
```
/health
```
and one turn (
```
reference/image.md
```
, "Local check"). Fix anything that fails here before you push.
Push and register. Push, pin by digest, run
```
check-image
```
, save a registry login if needed, make sure each secret is in the bench's candidate-env, write
```
bench/candidate.json
```
, and register the candidate (
```
reference/api.md
```
).
Run a smoke trial and verify it. Two tasks that need the agent's tools, no personas, one repeat. When nothing is pending, check every attempt against the table below. Each row that fails is an integration bug: find the cause in its artifacts, fix it, push a new digest, update the candidate, re-run it in the same trial once nothing is pending, and check again.
Report (format below).

检查访问权限。 读取该 bench。
```
access
```
必须为
```
write
```
。如果为
```
read
```
，则停止操作并告知用户原因（
```
reference/api.md
```
中的 “Access”）。
在不编辑的情况下阅读 Agent。 记录：使用的语言和入口点；执行一轮对话的函数及其维护对话历史的方式；model SDK 及其已安装版本；Agent 调用的每项外部服务，以及每个地址或 DSN 的来源（变量、配置文件或字面量）；Agent 读取的每个 secret；以及通过相对路径打开的所有文件。此时根据
```
reference/image.md
```
中的 “Telemetry” 检查 SDK：不支持的版本不会记录 model 调用，也不会给出任何提示。
阅读 bench 的 world。 列出其 services（
```
reference/api.md
```
中的 “World”）。将 Agent 调用的每项 service 与对应的 twin 匹配。告知用户 Agent 调用但没有 twin 的 service：除非 Veris 已为该 bench 启用 gateway，否则这些调用会到达真实 vendor。
添加 server 文件
```
bench_server.py
```
或
```
bench_server.mjs
```
，具体采用
```
reference/image.md
```
中的模式。它封装 Agent 的 turn 函数，按 session id 分别保存对话，提供
```
/health
```
，并监听
```
0.0.0.0:8008
```
。使用 http；仅当 Agent 已支持 WebSocket 时才使用 ws。
在 server 文件中让 Agent 指向 twins。 在 Agent 的 modules 加载之前，将
```
VERIS_DS_SERVICE_URLS
```
中每个 twin 的地址复制到 Agent 已经读取的配置项中。应用
```
reference/image.md
```
中 “Twins” 的 client fixes。
根据
reference/image.md
编写 Dockerfile 和
.dockerignore
。如果 repository 已有 Dockerfile 且可用于运行 server，则复用它。
在本地检查。 为
```
linux/amd64
```
构建，运行容器，并发送
```
/health
```
请求和一轮对话（
```
reference/image.md
```
中的 “Local check”）。在 push 前修复此处出现的所有失败。
推送并注册。 推送 image，按 digest 锁定版本，运行
```
check-image
```
，必要时保存 registry 登录凭据，确保每个 secret 都位于 bench 的 candidate-env 中，编写
```
bench/candidate.json
```
，然后注册 candidate（
```
reference/api.md
```
）。
运行 smoke trial 并验证。 设置两个需要 Agent 工具的任务，不配置 personas，重复一次。当没有待处理项时，根据下表检查每次尝试。任何一项检查失败都属于集成 bug：从 artifacts 中找出原因，修复，推送新 digest，更新 candidate，在同一 trial 中重新运行（仍需确保没有待处理项），然后再次检查。
报告（格式见下文）。

Done means every smoke attempt shows this

完成标准：每次 smoke 尝试都必须满足以下条件

Check	Where to read it ( `reference/api.md` , "Verify")	If it fails
It ran	`status` is `completed`	Read the end of `error` : it holds your container's last log lines.
The agent answered	`turns` is 2 or more, and the transcript has agent replies	Wrong path or field mapping, or the server crashed on a turn. Read `candidate.log` and `transcript.jsonl` .
The grader sees the model calls	`agent_trace_reason` is `null` , and `agent_trace.json` has `llm_calls` above 0	See "Telemetry" in `reference/image.md` . `agent_trace_reason` alone is not enough: it is `null` when the trace holds no model call. `judge_disabled_trace` means the bench's grader doesn't read traces: not a bug, skip this check.
The agent used its tools	`tool_calls` above 0 if the agent has tools, and the tool results in `agent_trace.json` are not errors	A tool error that names a connection or a certificate is step 5; read "Twins" in `reference/image.md` .
It reached the twins	each twin the agent uses received at least one request from it ( `reached` in the Verify command)	The agent is still pointed at its own default address. Redo step 5. Don't use `ledger_summary` counts for this: they include requests the platform makes.

passed

is not on this list. Whether the agent passes a task measures the agent, not the integration. Two attempts say little about quality; a full trial does.

检查项	阅读位置（ `reference/api.md` 中的 “Verify”）	失败时的处理方式
已运行	`status` 为 `completed`	读取 `error` 末尾：其中包含容器最后的日志行。
Agent 已回复	`turns` 为 2 或更多，且 transcript 中包含 Agent 回复	路径或字段映射错误，或 server 在处理某一轮对话时崩溃。读取 `candidate.log` 和 `transcript.jsonl` 。
评分器可以看到 model 调用	`agent_trace_reason` 为 `null` ，且 `agent_trace.json` 中的 `llm_calls` 大于 0	参见 `reference/image.md` 中的 “Telemetry”。仅凭 `agent_trace_reason` 不足以判断：当 trace 中没有 model 调用时，它也是 `null` 。 `judge_disabled_trace` 表示 bench 的评分器不读取 traces：这不是 bug，跳过此项检查。
Agent 使用了工具	如果 Agent 有工具， `tool_calls` 应大于 0，且 `agent_trace.json` 中的 tool results 不是错误	如果 tool error 指向连接或证书问题，执行第 5 步；阅读 `reference/image.md` 中的 “Twins”。
已到达 twins	Agent 使用的每个 twin 都至少收到一次来自它的请求（Verify 命令中的 `reached` ）	Agent 仍指向自身默认地址。重新执行第 5 步。不要使用 `ledger_summary` 计数进行判断：其中也包含平台发出的请求。

passed

不在此列表中。Agent 是否通过任务衡量的是 Agent 本身，而非集成效果。两次尝试不足以说明质量；完整 trial 才能说明。

Traps that look like success

看似成功的陷阱

Symptom	Cause	Fix
Node agent: every database tool fails, and the log has a `pg` SSL warning	node-postgres treats the twin DSN's `sslmode=require` as `verify-full` , and the twin's certificate is self-signed	Append `uselibpqcompat=true` to the DSN in the server file
Node agent: `agent_trace_reason` is `no_spans_received`	The trace hook writes into the working directory, which the run user can't write	The run user owns the app directory ( `chown` )
Node agent: `llm_calls` is 0, and `candidate.log` has no `veris-otel-node: instrumented OpenAI` line	The `openai` package is older than 5	Upgrade `openai` to 5 or later, and update the lockfile so the image build installs it
The attempt fails within a minute, `exec format error` or a pull error	An arm64 image built on Apple Silicon, or the bench can't pull it	`--platform linux/amd64` ; run `check-image`
Agent replies are empty	`image.path` or the response field doesn't match the server	Match `bench/candidate.json` to the server
Tools fail only on the bench	A literal address, or a variable the bench doesn't set	Step 5

症状	原因	修复方法
Node agent：所有数据库工具均失败，日志中有 `pg` SSL warning	node-postgres 会将 twin DSN 中的 `sslmode=require` 视为 `verify-full` ，而 twin 的证书是 self-signed	在 server 文件中的 DSN 末尾追加 `uselibpqcompat=true`
Node agent： `agent_trace_reason` 为 `no_spans_received`	trace hook 会写入 working directory，而运行用户无权写入	让运行用户拥有 app directory（ `chown` ）
Node agent： `llm_calls` 为 0，且 `candidate.log` 中没有 `veris-otel-node: instrumented OpenAI` 这一行	`openai` package 的版本低于 5	将 `openai` 升级到 5 或更高版本，并更新 lockfile，确保 image 构建时安装该版本
尝试在一分钟内失败，出现 `exec format error` 或拉取错误	在 Apple Silicon 上构建了 arm64 image，或 bench 无法拉取该 image	使用 `--platform linux/amd64` ；运行 `check-image`
Agent 回复为空	`image.path` 或 response field 与 server 不匹配	使 `bench/candidate.json` 与 server 匹配
工具仅在 bench 上失败	使用了字面量地址，或 bench 未设置某个变量	执行第 5 步

Report to the user

向用户报告

The files you added or changed, one line each.
The image reference, candidate id and trial id, and the console link
```
https://benchmark.veris.ai/benchmarks/<bench id>?trial=<trial id>
```
.
For each smoke attempt: the five checks, and
```
passed
```
.
Any change to the agent's own code or dependencies, and why.
Every fix you made after the first smoke run, and why.
Next: a full trial, as on https://docs.veris.ai/run-a-trial, and for a new version, a new candidate in the same trial.

你添加或修改的文件，每项一行。
Image reference、candidate id、trial id，以及 console 链接
```
https://benchmark.veris.ai/benchmarks/<bench id>?trial=<trial id>
```
。
每次 smoke attempt 的五项检查结果，以及
```
passed
```
状态。
对 Agent 自身代码或 dependencies 做出的任何更改及其原因。
首次 smoke run 之后做出的每项修复及其原因。
下一步：按照 https://docs.veris.ai/run-a-trial 运行完整 trial；如需新版本，则在同一 trial 中创建新 candidate。