evaluate-skill

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Evaluate a skill

评估 skill

A skill edit is only as good as the reports it produces. This workflow runs the candidate skill in sessions that know nothing except the skill file, a one-line business brief, and a locked-down local MCP endpoint, then compares what came back.

一次 skill 编辑的效果如何，取决于它生成的报告。此工作流会在会话中运行候选 skill；这些会话除 skill 文件、一行业务简介和受限制的本地 MCP endpoint 外，不掌握任何其他信息，随后再比较返回的结果。

Prerequisites

前置条件

Local server in no-auth mode:
```
AUTH_MODE=local_noauth
```
in
```
.env.local
```
, then
```
pnpm db:migrate:local
```
and
```
pnpm dev
```
(or
```
pnpm dev:agents
```
). Pass the URL it prints, with
```
/mcp
```
appended, as
```
--endpoint
```
. Default local D1, never remote bindings or remote Postgres. Keep it private; it has no login.
```
codex
```
CLI on the path. Runs use its configured default model at high reasoning; record the resolved model from the events log if you need to name it.
Research credits: the local server still calls DataForSEO. An audit run makes 20 to 40 tool calls. Do not run more sessions than the comparison needs.

本地 server 以 no-auth 模式运行：在
```
.env.local
```
中设置
```
AUTH_MODE=local_noauth
```
，然后运行
```
pnpm db:migrate:local
```
和
```
pnpm dev
```
（或
```
pnpm dev:agents
```
）。将它打印的 URL 追加
```
/mcp
```
后作为
```
--endpoint
```
传入。默认使用本地 D1，绝不使用远程 bindings 或远程 Postgres。保持其私有；它没有登录功能。
```
codex
```
CLI 位于 PATH 中。运行时使用其配置的默认模型，并启用 high reasoning；如需指明模型，请从 events log 中记录解析后的模型。
研究额度：本地 server 仍会调用 DataForSEO。一次 audit 运行会发起 20 到 40 次工具调用。运行的会话数不要超过比较所需的数量。

Run

运行

node .agents/skills/evaluate-skill/scripts/run.mjs \
  --skill seo-audit \
  --site example.com --site holdout.example \
  --brief "Evaluation brief: Acme at example.com sells <what>. The goal is relevant organic visitors who could become customers. No first-party analytics are connected." \
  --brief "Evaluation brief: <the holdout business>. ..." \
  --endpoint http://<local-dev-host>/mcp

Runs take about 10 to 14 minutes and execute in parallel. Launch the script detached, because a tool-call timeout that kills the shell kills the sessions too:

(nohup node .agents/skills/evaluate-skill/scripts/run.mjs ... > .logs/eval.log 2>&1 &)

What the script does, per site:

Copies the candidate skill and
```
seo-report
```
into a fresh temp folder. Edits during the run cannot change its instructions; the manifest records the skill hashes.
Creates a new local project seeded only with the brief.
Starts a gateway in front of the local MCP that allows a fixed tool list and rejects any call for another project id, then probes that rejection before starting.
Runs
```
codex exec
```
ephemeral, ignoring user config, memories, apps, plugins, and multi-agent, with the skill's self-review as the only reviewer.
Captures the saved report HTML, every MCP call and response, the event stream, and the final message under the output folder.

Read the reports with

node .agents/skills/evaluate-skill/scripts/report-text.mjs <out>/*-report.html

. Each session's working notes (for seo-audit,

opportunities.md

and

evidence/

) stay in its temp folder, named in

manifest.json

node .agents/skills/evaluate-skill/scripts/run.mjs \
  --skill seo-audit \
  --site example.com --site holdout.example \
  --brief "Evaluation brief: Acme at example.com sells <what>. The goal is relevant organic visitors who could become customers. No first-party analytics are connected." \
  --brief "Evaluation brief: <the holdout business>. ..." \
  --endpoint http://<local-dev-host>/mcp

每次运行约需 10 到 14 分钟，并行执行。以 detached 方式启动脚本，因为终止 shell 的工具调用超时也会终止会话：

(nohup node .agents/skills/evaluate-skill/scripts/run.mjs ... > .logs/eval.log 2>&1 &)

脚本针对每个站点执行以下操作：

将候选 skill 和
```
seo-report
```
复制到一个全新的临时目录。运行期间的编辑无法改变其指令；manifest 会记录 skill 的哈希值。
创建一个新的本地项目，其中仅初始化用户提供的简介。
在本地 MCP 前启动一个 gateway，只允许固定的工具列表，并拒绝任何针对其他 project ID 的调用；启动前会先探测该拒绝行为。
临时运行
```
codex exec
```
，忽略用户配置、memories、apps、plugins 和 multi-agent，并将 skill 的自评作为唯一的评审方式。
在输出目录下捕获已保存的报告 HTML、每次 MCP 调用及响应、event stream 和最终消息。

可使用

node .agents/skills/evaluate-skill/scripts/report-text.mjs <out>/*-report.html

读取报告。每个会话的工作笔记（对于 seo-audit，即

opportunities.md

和

evidence/

）保留在其临时目录中，并在

manifest.json

中记录名称。

Isolation rules

隔离规则

Instructions: frozen copies only. Never point a session at the working tree.
Conversation: a new
```
codex exec
```
per run. Never resume or fork.
Data: a new project per run, seeded with the user-provided brief and nothing else. Never write evaluator notes, prior reports, or a reference answer into any project the sessions can read.
Prompt: the same neutral prompt for every variant. Do not name expected findings, target keywords, or the rubric.
Evaluator material (rubric, reference audits, prior reports) lives outside the session folders and outside the repository. Raw artifacts belong in the ignored
```
.logs/
```
.

This is practical isolation, not a security boundary: sessions share the local account, provider caches, and organization limits.

**指令：**仅使用冻结副本。绝不能将会话指向工作树。
**对话：**每次运行都使用新的
```
codex exec
```
。绝不 resume 或 fork。
**数据：**每次运行都使用新项目，并且只初始化用户提供的简介。绝不能向会话可读取的任何项目中写入评估笔记、此前报告或参考答案。
**Prompt：**每个变体都使用相同的中性 prompt。不得提及预期发现、目标关键词或评分标准。
评估材料（评分标准、参考审计、此前报告）必须位于会话目录和仓库之外。原始产物应放在被忽略的
```
.logs/
```
中。

这是一种实用的隔离措施，而非安全边界：会话共享本地账户、provider 缓存和组织限额。

Compare

比较

Always run at least two sessions of the site you are tuning on and one holdout site with a different shape, so an instruction that fits one company's answer is caught. Keep every run, including failures; do not pick the best of several and call the skill fixed.

Score each report before opening its tool trace, then read the trace to classify any miss as discovery, tool, reasoning, or reporting. For an audit report:

Dimension	Weight	A strong report...
Discovery and coverage	25%	Reads commercial pages and their siblings, tools, and technical paths; broadens when a signal appears
Diagnosis and evidence	20%	Checks live pages and live results; separates observations from causes; records source, date, and settings
Business value and scale	20%	Ties each action to a real buyer; compares competing opportunities; never invents conversion or revenue
Prioritization	15%	Leads with a bounded change to a demonstrated problem for likely buyers; keeps maintenance and broad programs back
Calibration	10%	Makes useful calls without analytics; no penalty, causation, or trend claims the evidence cannot carry
Communication and actionability	10%	Names the page, the change, and the reason; material findings survive compression

A material factual error overrides the score. Ask a second model to recompute every live position from the run's own MCP log before trusting a report's numbers.

Coverage checks that generalize beyond any one site:

Sibling pages in an important family (comparison, pricing, template, location) were read side by side, and a page that only swaps a name into a shared answer was noticed.
A page already at the top for its query was protected, not rewritten.
Every serious candidate ended as a recommendation or as a table row with a real reason. A vague "later" is a miss.
Search volume, visits, and revenue were kept distinct; US demand was not multiplied into a global figure.

始终至少针对正在调优的站点运行两个会话，并针对一个形态不同的 holdout 站点运行一个会话，从而发现仅适用于某家公司答案的指令。保留每一次运行，包括失败；不要从多个结果中挑选最好的一个，就宣称 skill 已修复。

先对每份报告评分，再打开其工具调用轨迹；随后读取轨迹，将任何缺失归类为探索、工具、推理或报告问题。对于 audit 报告：

维度	权重	优秀报告应当……
发现与覆盖范围	25%	阅读商业页面及其同类页面、工具和技术路径；发现信号后扩大范围
诊断与证据	20%	检查实时页面和实时结果；区分观察与原因；记录来源、日期和设置
业务价值与规模	20%	将每项行动与真实买家关联；比较竞争机会；绝不捏造转化率或收入
优先级排序	15%	优先给出针对可能买家所面对且有证据支持的问题的有限改动；将维护工作和大型项目放在后面
校准	10%	在没有 analytics 的情况下作出有用判断；不提出证据无法支持的处罚、因果关系或趋势声明
沟通与可执行性	10%	指明页面、改动和原因；重要发现经压缩后仍能保留

重大事实错误的优先级高于评分。在信任报告中的数字之前，要求第二个模型根据本次运行自身的 MCP log 重新计算每个实时排名。

以下覆盖度检查可推广到任何单一站点之外：

已一并阅读重要页面家族中的同类页面（对比、定价、模板、位置），并注意到仅将名称替换进共用答案的页面。
对于已经在其 query 中排名靠前的页面，应予以保护，而不是重写。
每个重要候选项最终都成为建议，或成为带有真实理由的表格行。含糊的“稍后处理”属于一项缺失。
搜索量、访问量和收入应保持区分；不能将美国需求换算为全球数字。

What past evaluations taught

过往评估带来的经验

Instruction length hurts. A 3,600-word skill of cautions performed worse than a 2,200-word skill with a fixed process. Add a step, not a warning.
Force the comparison. Sessions that wrote a shortlist before drafting kept their diagnoses; sessions that drafted straight from research ranked by raw query volume.
Show the rejected candidates. A "What else we checked" table in the report is what stops material findings from disappearing when the report is shortened.
One snapshot lies. Positions around 8 to 12 change between two checks minutes apart; the skill re-runs decisive queries and reports both.
Labels beat notes. Readers rejected "10th of 17 organic rows" and arrows; tables now say "#10 (page 1)" with market and date in the header.

**指令过长有害。**满是注意事项的 3,600 词 skill，表现逊于采用固定流程的 2,200 词 skill。应增加一个步骤，而不是一条警告。
**强制进行比较。**起草前先列出 shortlist 的会话保留了诊断结论；直接根据研究起草的会话则按原始 query volume 排序。
**展示被否决的候选项。**报告中的“还检查了什么”表格，可以防止报告缩短时重要发现消失。
**单个快照并不可靠。**第 8 到第 12 名左右的位置，会在相隔几分钟的两次检查之间发生变化；skill 会重新运行决定性 query，并同时报告两次结果。
**明确标签胜过文字备注。**读者不接受“自然搜索 17 行中的第 10 行”和箭头这类表达；现在表格会写“#10（第 1 页）”，并在表头标出市场和日期。

Clean up

清理

Each run's project, report, and audit can be removed through MCP with

delete_report

delete_site_audit

, and

update_project_context

, or left in the local database. A new project per run is simpler than proving an old one is clean. Stop the local server when finished.

每次运行创建的项目、报告和 audit 都可以通过 MCP 使用

delete_report

、

delete_site_audit

和

update_project_context

删除，也可以留在本地数据库中。每次运行使用新项目，比证明旧项目已清理干净更简单。完成后停止本地 server。