kitaru-replay-experiment

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Kitaru replay experiment

Kitaru重放实验

Test one candidate condition against known cases and explain whether the available evidence improved, regressed, traded off, or stayed inconclusive. Do not make the deployment decision.

针对已知用例测试单一候选条件，并解释现有证据表明结果是提升、退化、权衡还是无定论。请勿做出部署决策。

Core contract

核心约定

Start from an accepted behavior, exact cohort version, exact evaluator versions and parameters, and one candidate change. Suggest one bounded candidate only when asked.
Replay starts a fresh agent task from each historical session's stored top-level inputs after applying the override. It does not restore an arbitrary checkpoint, conversation, process memory, adapter instance state, filesystem, or external world state.
Resolve adapter support and its construction path before asking to run the experiment. A shared replay schema does not prove that an adapter supports a requested override or tool source.
Require an explicit tool policy for every tool-using run. Omission resolves to live passthrough on the current server and is unsafe as an implicit default.
Carry exact IDs, versions, evaluator parameters, run-spec evidence, tool policy, failures, and missing results forward.
Explain remote writes, model and worker compute, cost uncertainty, and possible live effects before execution. One approval after this explanation covers experiment creation and the run start; any tool path with external effects needs separate approval.
Use established Kitaru product terms only. Do not coin labels for summaries or steps, such as “run card,” “result card,” “agent fingerprint,” or “execution checksum.” Do not replace a Kitaru object with a friendly-sounding alias such as “accepted baseline”; explain the official term when necessary, then use it consistently. In user-facing text, describe what will happen and what the user must decide in ordinary language.
Prefer native Kitaru MCP operations. Use the structured CLI for built-in waiting or another capability MCP does not expose. Verify installed schemas when they differ from the references.
Run every Kitaru CLI command and SDK script with
```
KITARU_ACTIVE_SKILL=kitaru-replay-experiment
```
set so the server attributes the resulting activity to this skill.
Start or restart a user-controlled worker with
```
--concurrency 10
```
. Use
```
KITARU_WORKER_CONCURRENCY=10
```
only when the launch surface exposes worker settings through environment variables instead of CLI options.
Never bypass a missing adapter, evidence, comparison, or product contract with direct REST calls or ad hoc local state.

从已验收行为、指定队列版本、指定评估器版本及参数，以及单一候选变更开始。仅在被询问时才提出单一受限候选方案。
重放会在应用覆盖配置后，从每个历史会话存储的顶层输入启动全新的Agent任务。不会恢复任意检查点、对话、进程内存、适配器实例状态、文件系统或外部世界状态。
在请求运行实验前，先确认适配器支持情况及其构建路径。共享重放架构并不代表适配器支持请求的覆盖配置或工具源。
每次涉及工具调用的运行都需要明确的工具策略。若未指定，则默认使用当前服务器的实时直通模式，这作为隐式默认设置是不安全的。
保留准确的ID、版本、评估器参数、运行规格证据、工具策略、失败记录和缺失结果。
在执行前解释远程写入、模型与 worker 计算、成本不确定性及可能的实时影响。本次解释后的单次批准涵盖实验创建和运行启动；任何涉及外部影响的工具路径都需要单独批准。
仅使用已确立的Kitaru产品术语。请勿为总结或步骤创造标签，如“运行卡片”“结果卡片”“Agent指纹”或“执行校验和”。请勿用听起来友好的别名替代Kitaru对象，如“已验收基线”；必要时先解释官方术语，然后持续使用该术语。在面向用户的文本中，用通俗语言描述将要发生的事情以及用户必须做出的决策。
优先使用原生Kitaru MCP操作。当MCP未暴露内置等待或其他功能时，使用结构化CLI。当已安装架构与参考架构不同时，进行验证。
运行所有Kitaru CLI命令和SDK脚本时，设置
```
KITARU_ACTIVE_SKILL=kitaru-replay-experiment
```
，以便服务器将产生的活动归因于本skill。
使用
```
--concurrency 10
```
启动或重启用户控制的worker。仅当启动界面通过环境变量而非CLI选项暴露worker设置时，才使用
```
KITARU_WORKER_CONCURRENCY=10
```
。
切勿通过直接REST调用或临时本地状态绕过缺失的适配器、证据、对比或产品约定。

Load references only when needed

仅在需要时加载参考文档

Read references/experiment-contract.md before any Kitaru operation or adapter capability claim.
Read references/experiment-method.md before choosing the comparison type, interpreting conflicting results, adding repeats, or evaluating a user-defined gate.

在进行任何Kitaru操作或适配器功能声明前，阅读references/experiment-contract.md。
在选择对比类型、解读冲突结果、添加重复测试或评估用户定义的准入条件前，阅读references/experiment-method.md。

Resolve the starting state

确定初始状态

Begin read-only. Reuse exact identifiers from an investigation checkpoint or the user's request, then re-read each object.

Resolve:

the accepted behavior and intended use: exploration, known-case regression, or an exact user-defined gate;
the exact cohort-version ID and ordered case count;
one exact candidate agent-version ID and the current public run spec and capabilities;
exact evaluator-version IDs, parameters, and factual evidence about their checks and limitations;
one proposed replay override and its expected effect;
the adapter and construction path;
every tool the candidate may invoke and one explicit policy covering all of them.

Route to

kitaru-investigation

when no behavior or cohort has been accepted. Use its evaluator route when no suitable evaluator exists or when the current evaluator is too weak for the intended claim. Offer a claim downgrade instead of forcing validation ceremony when the user only needs an exploratory result.

If an experiment or run already exists, re-read it and resume supervision or interpretation. Experiment configuration freezes after it has runs, so create a new experiment for a materially changed candidate, evaluator set, override, or tool policy.

先以只读模式开始。重用来自调查检查点或用户请求的准确标识符，然后重新读取每个对象。

需确定：

已验收行为及预期用途：探索性测试、已知用例回归测试或用户定义的明确准入条件；
准确的队列版本ID及有序用例数量；
单一准确的候选Agent版本ID，以及当前公开运行规格和功能；
准确的评估器版本ID、参数，以及关于其检查逻辑和局限性的事实证据；
一个提议的重放覆盖配置及其预期效果；
适配器及其构建路径；
候选对象可能调用的所有工具，以及覆盖所有工具的明确策略。

当尚无已验收行为或队列时，转至

kitaru-investigation

流程。当不存在合适的评估器或当前评估器无法满足预期声明要求时，使用其评估器路由。当用户仅需要探索性结果时，建议降低声明等级，而非强制要求验证流程。

若实验或运行已存在，重新读取并恢复监督或解读。实验配置在启动运行后即冻结，因此当候选方案、评估器集合、覆盖配置或工具策略发生实质性变更时，需创建新实验。

Preflight capability and safety

预检查功能与安全性

Reject unsupported adapter conditions before approval or paid execution. Return the exact blocker and smallest supported alternative. Do not let a user approve past an unresolved adapter or construction path.

Resolve the model provider and credentials required by the candidate process. Verify credential availability to the worker without reading, displaying, or fingerprinting secret values when the environment exposes a safe readiness signal. A credential visible to the coding agent or stored in a project file does not prove that an already-running worker inherited it. If worker readiness is unavailable or cannot be verified, ask the user to configure the credential and restart or provision the worker, then stop before experiment creation or run start.

For tool-using agents:

Choose the tool source that answers the hypothesis. Do not default to recorded history merely because a baseline is available.
Prefer explicit passthrough for verified deterministic, isolated local fixtures when the candidate may change tool choice or arguments. State that this still executes tool code, name its effects, and distinguish it from an external live system.
Use baseline-scoped recorded history when holding recorded tool results fixed is part of the intended comparison and the adapter supports it. Explain that lookup matches a prior call; it does not restore baseline state or permit new arguments.
Use
```
fail
```
or
```
error_result
```
on a history or static miss unless live execution is deliberate.
Treat
```
on_miss=passthrough
```
as live passthrough and gate it separately.
Warn that cohort-wide or agent-wide history can borrow stale or cross-case state. Matching uses tool name and canonical JSON arguments. Baseline scope replays repeated identical calls in recorded order, wider scopes select the latest match in scope.
Treat LLM-generated results as a synthetic condition, not restoration of the historical external world.
Keep tool execution failures separate from agent-quality failures.

Check whether repeated calls with identical arguments can legitimately return different values. Baseline scope replays each such call with its own recorded result in order, wider scopes answer every one with the latest match. Baseline scope is the narrowest history source, not proof that a stored result remains faithful.

When passthrough and history are both valid but would change what the experiment means, explain the concrete difference and ask the user which condition they want unless a verified project-specific contract already selects one.

在获得批准或付费执行前，拒绝不支持的适配器条件。返回准确的阻塞点及最小可行的替代方案。不得让用户在适配器或构建路径未解决的情况下批准执行。

确定候选流程所需的模型提供商及凭证。当环境提供安全就绪信号时，在不读取、显示或生成凭证指纹的前提下，验证worker是否具备凭证可见性。编码Agent可见的凭证或存储在项目文件中的凭证，并不代表已运行的worker已继承该凭证。若worker就绪状态不可用或无法验证，请用户配置凭证并重启或配置worker，然后在实验创建或运行启动前停止操作。

对于使用工具的Agent：

选择能验证假设的工具源。不要仅因为存在基线就默认使用历史记录。
当候选对象可能改变工具选择或参数时，优先为经过验证的确定性、隔离本地测试装置使用显式直通模式。说明这仍会执行工具代码，指明其影响，并将其与外部实时系统区分开。
当保持记录的工具结果固定是预期对比的一部分且适配器支持时，使用基线范围的历史记录。解释查找操作会匹配先前的调用；不会恢复基线状态或允许新参数。
除非故意进行实时执行，否则在历史记录或静态数据缺失时使用
```
fail
```
或
```
error_result
```
。
将
```
on_miss=passthrough
```
视为实时直通模式，并单独进行准入检查。
警告队列范围或Agent范围的历史记录可能借用过时或跨用例的状态。匹配使用工具名称和标准JSON参数。基线范围按记录顺序重放重复的相同调用，更广范围则选择范围内的最新匹配结果。
将LLM生成的结果视为合成条件，而非恢复历史外部世界状态。
将工具执行失败与Agent质量失败分开处理。

检查相同参数的重复调用是否合理返回不同值。基线范围按顺序重放每个此类调用及其对应的记录结果，更广范围则用最新匹配结果回答所有调用。基线范围是最窄的历史数据源，但不代表存储的结果仍然可靠。

当直通模式和历史记录均有效但会改变实验意义时，解释具体差异并询问用户需要哪种条件，除非已验证的项目特定约定已选定其中一种。

Confirm the run with the user

与用户确认运行细节

Before asking for approval, give the user a short summary in ordinary language. Prefer a few bullets over a table. Include only:

the change being tested and the question it should answer;
the cohort and evaluator versions, with the number of cases;
what each new task starts from and the important state it does not restore;
how tools will behave, including any possible external effects;
the expected model and worker work, with cost uncertainty; and
what the comparison can and cannot establish.

Include exact agent, experiment, or adapter details only when the user needs them to verify the decision or when they expose a material limitation. Keep complete IDs, versions, parameters, configuration hashes, and adapter evidence in the resumable technical record rather than making the user read them all before a routine exploratory run.

Hash the complete public run-spec and capabilities models internally using canonical serialization. Do not resolve or expose secrets. Immediately before experiment creation and again before starting execution, re-read both models and compare the hashes. If either changed, invalidate approval and explain which configuration changed without naming the hash as a product object. An exact agent-version ID identifies a mutable configuration, not an immutable reproducible artifact, and these checks cannot freeze or prove the configuration a worker later resolves. For a user-defined gate or live-effect run, require immutable runtime identity or narrow the use to exploration. Without that identity, report observed violations but return

cannot evaluate

for the gate. For an exploratory run, preserve the approved hashes, re-read the models after settlement, and return

inconclusive

if observable drift occurred.

Ask for one approval after the summary is visible. If passthrough can affect an external system, ask separately for approval of the named live tools and effects. Verified isolated local mock execution can be covered by the first approval.

在请求批准前，用通俗语言为用户提供简短摘要。优先使用项目符号而非表格。仅包含：

待测试的变更及其要回答的问题；
队列和评估器版本，以及用例数量；
每个新任务的启动来源，以及不会恢复的重要状态；
工具的行为方式，包括任何可能的外部影响；
预期的模型和worker工作量，以及成本不确定性；
该对比能证明和不能证明的内容。

仅当用户需要验证决策或内容暴露实质性限制时，才包含准确的Agent、实验或适配器细节。将完整的ID、版本、参数、配置哈希和适配器证据保存在可恢复的技术记录中，而非让用户在常规探索性运行前阅读所有内容。

使用标准序列化方式在内部对完整的公开运行规格和功能模型进行哈希处理。不要解析或暴露密钥。在创建实验前和启动执行前，重新读取这两个模型并对比哈希值。若任一模型发生变更，则作废批准并解释哪项配置发生了变更，无需将哈希值作为产品对象提及。准确的Agent版本ID标识的是可变配置，而非不可变的可复制品，这些检查无法冻结或证明worker后续解析的配置。对于用户定义的准入条件或有实时影响的运行，要求不可变的运行时标识或仅用于探索性测试。若无该标识，则报告观察到的违规情况，但对准入条件返回

无法评估

。对于探索性运行，保留批准的哈希值，在稳定后重新读取模型，若观察到漂移则返回

无定论

。

在摘要展示后请求一次批准。若直通模式可能影响外部系统，则单独请求对指定的实时工具及其影响进行批准。经过验证的隔离本地模拟执行可包含在首次批准中。

Create and supervise one run

创建并监督一次运行

After approval:

Create one experiment with the exact agent parent, replay override, explicit tool policy, and evaluator versions and parameters.
Re-read it and verify that the server resolved the intended configuration.
Recheck the candidate run-spec and capabilities hashes internally.
Start one run against the exact cohort version and candidate agent version. Set
```
evaluate_baselines=true
```
for comparative claims.
Watch the asynchronous run until it completes, fails, is canceled, reaches an agreed stop condition, or the local wait times out.
Read the run and all paginated replay jobs needed to account for completed, failed, canceled, and missing cases.
Re-read the candidate run spec and capabilities after settlement. Report any observed drift and the residual risk that a transient mutation could not be detected.

Run once by default. Add predeclared matched repetitions only when stochasticity could reverse a consequential or borderline conclusion, and approve the extra work and cost first.

Cancel only when the user asks or an approved stop condition is met. Cancellation preserves evidence. Deletion is destructive and is not cancellation.

获得批准后：

使用准确的Agent父项、重放覆盖配置、明确的工具策略、评估器版本及参数创建一个实验。
重新读取实验并验证服务器是否解析了预期配置。
再次在内部检查候选运行规格和功能的哈希值。
针对指定队列版本和候选Agent版本启动一次运行。对于对比声明，设置
```
evaluate_baselines=true
```
。
监控异步运行直至完成、失败、被取消、达到约定的停止条件或本地等待超时。
读取运行记录及所有分页重放任务，以统计已完成、失败、取消和缺失的用例。
在稳定后重新读取候选运行规格和功能。报告任何观察到的漂移，以及无法检测到瞬时突变的残留风险。

默认运行一次。仅当随机性可能颠覆重要或边缘结论时，才添加预先声明的匹配重复测试，并先批准额外的工作量和成本。

仅在用户要求或达到批准的停止条件时才取消运行。取消操作会保留证据。删除操作具有破坏性，不属于取消范畴。

Read exact evidence

读取准确证据

For each case, resolve evaluation results tied to the expected evaluator-version ID. Verify evaluator parameters or deterministic configuration hashes where available. Decline a decision-grade parameterized baseline comparison when the parameters of an existing result cannot be established.

Do not use the current UI aggregate as a decision-grade source. Read exact per-session evidence and show raw counts, denominators, failed, canceled, and missing cases beside every rate. Failed, canceled, and missing results remain outside the quality denominator.

Check candidate prompts, evaluator prompts, examples, tool fixtures, history scopes, and related trace families for leakage. Warn for exploration. Require resolution or narrow the claim only when the intended use is consequential.

针对每个用例，解析与预期评估器版本ID关联的评估结果。在可用时验证评估器参数或确定性配置哈希。当无法确定现有结果的参数时，拒绝决策级别的参数化基线对比。

不要将当前UI聚合数据作为决策级来源。读取准确的每会话证据，并在每个比率旁显示原始计数、分母、失败、取消和缺失的用例数量。失败、取消和缺失的结果不计入质量分母。

检查候选提示词、评估器提示词、示例、工具测试装置、历史范围及相关跟踪家族是否存在信息泄露。对于探索性测试发出警告。仅当预期用途重要时，才要求解决问题或缩小声明范围。

Explain the result

解释结果

Report:

exact experiment and run IDs, cohort version, candidate version, evaluator versions and parameters, tool policy, and run-spec evidence;
comparison type and the state that was and was not restored;
improved, regressed, unchanged, failed, canceled, and missing case counts, plus pending or evaluating counts for nonterminal runs;
important paired case changes and protection outcomes;
evaluator limitations, provenance gaps, and known execution differences;
one conclusion:
```
improved
```
,
```
regressed
```
,
```
trade-off
```
, or
```
inconclusive
```
;
only when the user supplied a fit-for-purpose gate:
```
meets
```
,
```
does not meet
```
, or
```
cannot evaluate
```
that exact gate;
one next useful action.

Do not say

ship

. The user owns deployment. Kitaru does not currently expose a public winner, release policy, statistical significance, or CI-exit verdict.

If a run stalls, fails, is canceled, or remains partial, return its exact current state, completed, failed, canceled, missing, pending, and evaluating counts, resumable identifiers, and the smallest safe next action. Keep canceled cases outside quality denominators rather than folding them into failed or missing cases. If replay exposes a genuinely new or ambiguous behavior, hand exact session evidence to a bounded follow-up

kitaru-investigation

instead of silently changing the evaluator.

报告：

准确的实验和运行ID、队列版本、候选版本、评估器版本及参数、工具策略和运行规格证据；
对比类型，以及已恢复和未恢复的状态；
提升、退化、无变化、失败、取消和缺失的用例数量，以及非终端运行的待处理或评估中数量；
重要的配对用例变更和防护结果；
评估器局限性、来源缺口及已知执行差异；
一个结论：
```
提升
```
、
```
退化
```
、
```
权衡
```
或
```
无定论
```
；
仅当用户提供适用的准入条件时：该准入条件的
```
符合
```
、
```
不符合
```
或
```
无法评估
```
结果；
一个有用的后续操作建议。

请勿提及“上线”。部署决策由用户负责。Kitaru目前不公开优胜者、发布策略、统计显著性或CI退出 verdict。

若运行停滞、失败、被取消或未完成，返回其准确的当前状态、已完成、失败、取消、缺失、待处理和评估中的数量、可恢复的标识符，以及最安全的最小后续操作。将取消的用例排除在质量分母之外，而非归入失败或缺失用例。若重放暴露了真正的新行为或模糊行为，将准确的会话证据交给受限的后续

kitaru-investigation

流程，而非默默更换评估器。

Suggest one candidate when asked

仅在被询问时提出单一候选方案

Prefer, in order:

clarify an ambiguous instruction or tool description;
add one small non-leaking example or deterministic protection;
change one pipeline component or model setting;
consider a larger structural or model change after simpler interventions fail.

Do not start an automatic prompt search, cascade, fine-tuning program, or multi-candidate tournament.

优先按以下顺序选择：

澄清模糊的指令或工具描述；
添加一个小型无泄露示例或确定性防护措施；
修改一个流水线组件或模型设置；
在简单干预失败后，考虑较大的结构或模型变更。

请勿启动自动提示词搜索、级联操作、微调程序或多候选方案竞赛。