Back to Details

checkpoint-promotion

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Checkpoint Promotion

Checkpoint 推广

The Phase 5 gate for the whole plugin: a checkpoint that trains cleanly and beats its task metric still doesn't ship without clearing all four stages below. 
eval-harness-first
built the suite re-run here — this skill is where that suite's baseline decides something.
Input: a trained checkpoint, 
eval/baseline-<model>.json
from 
eval-harness-first
, and the frozen 
eval/drift-suite.yaml
. Output format: 
promotion-report.md
— the four-stage evidence plus a terminal 
PROMOTE
or 
REJECT
verdict that 
/finetune
Phase 5 and 
/promote-checkpoint
consume directly.
这是整个插件的第5阶段门控：即使一个checkpoint训练过程顺畅且达到任务指标要求，也必须通过以下四个阶段才能发布。
eval-harness-first
构建了此处要重新运行的测试套件——本技能正是利用该套件的基线来做出决策。
输入： 训练好的checkpoint、来自
eval-harness-first
的
eval/baseline-<model>.json
，以及冻结的
eval/drift-suite.yaml
。 输出格式： 
promotion-report.md
——包含四个阶段的证据，以及最终的
PROMOTE
或
REJECT
结论，可供
/finetune
第5阶段和
/promote-checkpoint
直接调用。

The Four-Stage Gate

四阶段门控

Each stage gates the next — a failure at stage 2 means stage 3 doesn't run. Stages 2 and 3 share one expensive inference pass, so running them concurrently and applying gate order at verdict time is licensed on a deterministic arena (nothing saved by serializing); a judge-based arena should still wait for stage 2 first — that's where the real savings are.
  1. Data-quality gate. Before any eval touches the checkpoint: dedup the training set, check for eval-goldens leakage (the exact failure 
    trace-to-training-data
    's Hygiene section exists to prevent), and scan for label noise. A checkpoint trained on leaked goldens invalidates every later stage.
  2. Held-out + frozen capability-drift suite. Re-run 
    eval-harness-first
    's 
    eval/drift-suite.yaml
    — MMLU/GSM8K/IFEval plus 200–500 domain-adjacent items — against the checkpoint and diff against 
    baseline-<model>.json
    per benchmark against the Drift Budget table below.
  3. Paired arena vs. base. Position-randomized judge, checkpoint vs. base model, same prompts — or the deterministic paired-comparison variant in 
    references/gate-templates.md
    when every grader in the harness is deterministic (no LLM-judge; position randomization N/A there). A holdout win that loses the live arena does not ship — stage-2 numbers and stage-3 judgments must agree; a win on frozen goldens and a loss in paired comparison is a real signal, not a discrepancy to explain away.
  4. Canary. 5–10% stratified rollout with auto-rollback for any checkpoint reaching production traffic. Local-only users stop at stage 3 — skipping stage 4 for a local deployment is the correct stopping point, not a shortcut.
每个阶段作为下一阶段的前置条件——若第2阶段失败，则第3阶段无需运行。第2阶段和第3阶段共享一次高成本的推理过程，因此在确定性场景中（串行执行无任何收益），可允许并行运行这两个阶段，再在结论阶段应用门控顺序；而基于评判者的场景仍需先等待第2阶段完成——这才是真正能节省成本的地方。
  1. 数据质量门控。在对checkpoint进行任何评估之前：对训练集去重，检查是否存在评估黄金数据泄露（这正是
    trace-to-training-data
    的Hygiene部分要预防的问题），并扫描标签噪声。若checkpoint是在泄露的黄金数据上训练的，后续所有阶段的结果都将无效。
  2. 保留数据集+冻结能力漂移套件。重新运行
    eval-harness-first
    的
    eval/drift-suite.yaml
    ——包含MMLU/GSM8K/IFEval以及200–500个领域相关样本——将checkpoint的结果与
    baseline-<model>.json
    中的各基准进行对比，并参照下方的漂移预算表判断。
  3. 与基础模型的配对竞技场对比。采用位置随机化评判者，将checkpoint与基础模型在相同提示下对比；当测试套件中的所有评分器都是确定性的（无LLM评判者，位置随机化不适用）时，可使用
    references/gate-templates.md
    中的确定性配对对比变体。 若在保留数据集上表现优异但在实时竞技场中落败，则不能发布——第2阶段的数值和第3阶段的评判必须一致；在冻结黄金数据集上获胜但在配对对比中落败是真实信号，而非需要解释的差异。
  4. 金丝雀发布。对进入生产流量的checkpoint进行5–10%的分层滚动发布，并设置自动回滚机制。仅本地使用的用户可在第3阶段停止——对于本地部署而言，跳过第4阶段是正确的终止点，而非捷径。

Drift Budget

漂移预算

Drift (pts)Verdict
≤1Noise — proceed
2–5Rerun with seed variation before deciding
>5HARD FAIL — no exception for task gains
The >5pt row governs regardless of the others: a checkpoint that gained 8 points on the target task and lost 6 points of general capability still fails here — task improvement never buys back a drift-budget breach.
Item count derives from the budget, not convenience: the strict n for a half-width under half the 5pt hard-fail threshold is ~1,300 at typical accuracy (p≈0.7); n=200 is a pragmatic floor (±6pt half-width at that same p, n=50 ±13pt) — report the half-width with every verdict, and treat a margin smaller than it as 
REJECT (uncertain)
, not PASS/HARD FAIL. Full math and a 5-run cautionary example: 
references/gate-templates.md
.
RERUN is not a verdict. A 2–5pt drift only ever produces a 
PROMOTE
or 
REJECT
after the seed-variation rerun completes — 
PROMOTE
requires landing back at ≤1pt (noise); any rerun still
1pt — 2–5pt band or >5pt breach alike — resolves stage 2 to a hard 
REJECT
. No report may reach the Verdict section with stage 2 still showing 
RERUN
.
漂移值（分）结论
≤1噪声——继续推进
2–5调整种子后重新运行再做决定
>5硬性失败——即使任务指标提升也无例外
5分的规则优先级高于其他所有规则：若某个checkpoint在目标任务上提升了8分，但在通用能力上下降了6分，仍会在此处失败——任务指标的提升永远无法弥补漂移预算的超标。
样本数量由预算决定，而非便利性： 在典型准确率（p≈0.7）下，要使半宽小于5分硬性失败阈值的一半，所需样本数约为1300；200是务实的下限（相同p值下半宽±6分，n=50时半宽±13分）——需在每个结论中报告半宽，若差值小于半宽，则标记为
REJECT (uncertain)
，而非通过/硬性失败。完整的数学计算和包含5次运行的警示示例见：
references/gate-templates.md
。
重新运行不是结论。2–5分的漂移只有在调整种子重新运行完成后，才能得出
PROMOTE
或
REJECT
结论——
PROMOTE
要求漂移值回到≤1分（噪声范围）；任何重新运行后漂移值仍>1分——无论是2–5分区间还是>5分超标——第2阶段都将判定为硬性
REJECT
。所有报告的结论部分都不能显示第2阶段仍处于
RERUN
状态。

Catastrophic Forgetting

灾难性遗忘

Unmanaged LoRA fine-tuning loses real general capability, and stage 2 is what catches it:
  • ~43% knowledge loss unmanaged — no replay, no regularization.
  • ~10% with basic management — some replay or a conservative LR.
  • ~3% with replay + EWC — the disciplined case.
  • 10–30% general-data replay mix is the standard mitigation — blend general- domain data into training rather than target-task data alone.
If a checkpoint hits the >5pt hard fail in stage 2, work this escalation ladder in order — the one canonical order this skill and 
references/gate-templates.md
both point to:
  1. Adjust the replay-mix fraction — swap rows, don't add them (adding confounds fraction with total optimizer steps). Dose is not monotonic at small-run scale (<~100 steps) — re-check drift after any swap.
  2. Lower the learning rate.
  3. Fewer epochs.
  4. A smaller LoRA rank — the same rank/LR levers 
    lora-qlora-recipes
    and 
    preference-optimization
    tune for the training run, applied here in reverse.
This order is a default, not a law: remediation guidance from a single before/after run pair is a hypothesis — label it low-confidence once any lever produces a reversal, and prefer a seed-variation repeat over trusting the next rung blindly. A lever that clears the drift breach but drops a success-criterion metric below target is a two-sided tradeoff for a human, not a reason to keep descending the ladder. Full reasoning and the 5-run trajectory behind both caveats: 
references/gate-templates.md
.
Disclose drift-suite instruction reuse. A replay row copying the drift harness's exact instruction phrasing (not just disjoint source items) makes that benchmark's post-replay score an upper bound — flag it instruction-familiar, or re-probe with a paraphrase, before treating a near-budget pass as clean.
未经管理的LoRA微调会导致真实的通用能力下降，而第2阶段正是用于检测这一问题：
  • 约43%的知识损失（无管理）——无重放、无正则化。
  • 约10%的知识损失（基础管理）——有部分重放或保守的学习率（LR）。
  • 约3%的知识损失（重放+EWC）——规范管理的情况。
  • 10–30%的通用数据重放混合是标准缓解方案——将通用领域数据混入训练集，而非仅使用目标任务数据。
若checkpoint在第2阶段触发>5分的硬性失败，请按以下顺序尝试解决——本技能和
references/gate-templates.md
均指向这一标准顺序：
  1. 调整重放混合比例——替换而非新增行（新增行会混淆比例与总优化步数）。在小规模运行（<~100步）下，剂量并非单调变化——每次替换后需重新检查漂移情况。
  2. 降低学习率。
  3. 减少训练轮数。
  4. 使用更小的LoRA秩——与
    lora-qlora-recipes
    和
    preference-optimization
    在训练过程中调整的秩/LR控制杆相同，此处反向应用。
该顺序是默认方案，而非强制规则：单次前后运行对比得出的修复指导只是假设——一旦某个控制杆产生逆转效果，需标记为低置信度，并且优先选择调整种子重复运行，而非盲目尝试下一阶梯的方案。若某个控制杆解决了漂移超标问题，但将成功标准指标降至目标以下，则这是需要人工权衡的双向取舍，而非继续尝试下一阶梯的理由。完整的推理过程和支撑这两个注意事项的5次运行轨迹见：
references/gate-templates.md
。
披露漂移套件指令复用情况。若重放行复制了漂移测试套件的 exact 指令表述（而非仅使用不相交的源样本），则该基准的重放后分数将是上限——在将接近预算的结果视为合格之前，需标记为“指令熟悉”，或使用改写后的指令重新测试。

The Verdict

最终结论

promotion-report.md
covers all four stages as sections and must end with a terminal verdict: 
PROMOTE
or 
REJECT
, the evidence that produced it, and exactly one top remediation when the verdict is 
REJECT
. Template: 
references/gate-templates.md
. The terminal contract other skills parse:
undefined
promotion-report.md
需将四个阶段作为章节涵盖在内，并且必须以最终结论结尾：
PROMOTE
或
REJECT
，同时附上得出该结论的证据；若结论为
REJECT
，需明确列出一项最优修复方案。模板见：
references/gate-templates.md
。其他技能解析的最终约定如下：
undefined

Verdict

Verdict

REJECT
Evidence: domain-adjacent drift suite dropped 6.2pt (threshold:
5pt hard fail) despite +8pt on the target task.
Top remediation: swap the replay-mix fraction from 10% toward 20%, holding step count constant.

- **REJECT is a result, not an
  error.** A checkpoint that
  fails stage 2's drift budget or
  stage 3's arena comparison did
  its job. Don't treat a REJECT
  as a failed run needing a rerun
  of this skill; it's the correct
  output of a working gate.
- **One remediation, not a
  menu.** Evidence sections may
  list everything observed; the
  verdict section names the
  single highest-leverage fix per
  the escalation ladder above. A
  report that hedges across three
  possible fixes hasn't done the
  prioritization this skill
  exists to do.
- **No auto-retraining.** This
  skill produces a verdict and a
  report, not a re-triggered
  training run. A `REJECT` hands
  the remediation back to a human
  decision at
  `finetuning-method-selection` or
  the relevant training skill.
REJECT
Evidence: domain-adjacent drift suite dropped 6.2pt (threshold:
5pt hard fail) despite +8pt on the target task.
Top remediation: swap the replay-mix fraction from 10% toward 20%, holding step count constant.

- **REJECT是结果，而非错误**。若checkpoint未通过第2阶段的漂移预算或第3阶段的竞技场对比，这正是它的作用。不要将REJECT视为需要重新运行本技能的失败案例；这是正常工作的门控机制的正确输出。
- **仅提供一项修复方案，而非菜单选项**。证据部分可列出所有观察到的问题；结论部分需根据上述阶梯顺序，命名单一最高杠杆的修复方案。若一份报告在三个可能的修复方案之间摇摆不定，则未完成本技能旨在实现的优先级排序。
- **无自动重新训练**。本技能生成结论和报告，而非触发重新训练。`REJECT`会将修复方案交还给`finetuning-method-selection`或相关训练技能处的人工决策环节。

Related Skills

相关技能

  • eval-harness-first
    — owns the drift suite and baseline this skill re-runs and diffs against; no 
    baseline-<model>.json
    means nothing to gate against.
  • quantized-export
    — the only valid next step after a 
    PROMOTE
    verdict.
  • preference-optimization
    and 
    lora-qlora-recipes
    — own the LR and rank levers in the Catastrophic Forgetting escalation path; this skill diagnoses the breach, those skills own the config that caused it.
  • dataset-curation
    — owns the replay-mix construction recipe the escalation ladder's first rung applies.
Complete 
promotion-report.md
template with all four stages, the drift-suite scoring table, the paired-arena protocol (item count, position randomization, win-rate threshold), and a replay-mix configuration example: 
references/gate-templates.md
.
  • eval-harness-first
    ——负责本技能重新运行和对比的漂移套件与基线；若无
    baseline-<model>.json
    ，则无门控对比的依据。
  • quantized-export
    ——
    PROMOTE
    结论后的唯一有效后续步骤。
  • preference-optimization
    和
    lora-qlora-recipes
    ——负责灾难性遗忘解决阶梯中的LR和秩控制杆；本技能诊断超标问题，而这些技能负责导致问题的配置。
  • dataset-curation
    ——负责解决阶梯第一步所应用的重放混合构建方案。
包含所有四个阶段、漂移套件评分表、配对竞技场协议（样本数量、位置随机化、胜率阈值）以及重放混合配置示例的完整
promotion-report.md
模板见：
references/gate-templates.md
。