benchmark-checklist

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Benchmark checklist

基准测试检查清单

Use this when you produce a performance number: a PR's before and after, a regression claim, a hillclimb harness, or a library or config choice. Answer each question with evidence from a run, not a guess about the code.

For a quick ballpark the user asked for, one run is enough. Still check questions 4 and 7, and say that it is one run. Skip the rest unless that run looks wrong. A choice between options is never a ballpark.

当你产出性能数据时使用本清单：包括 PR 的前后对比、性能回归结论、hillclimb 测试工具（harness），或库与配置选型。每个问题都要以某次运行的证据来回答，而不是根据代码猜测。

如果用户只是要求一个大致结果，运行一次即可。但仍要检查第 4 和第 7 个问题，并说明结果仅来自一次运行。除非该次运行看起来有问题，否则可以跳过其余检查。在多个选项之间做选择绝不属于粗略估算。

Before you run anything

运行任何内容之前

Write down the claim you expect to make, in the words you would ship ("export is 30% faster at p50 on the 60k-row dataset"). The questions test that sentence.
Read the measurement script. Note what it times, counts, and ignores.
Check the load average with
```
uptime
```
and the core count with
```
nproc
```
. If the machine is busy, find out what is running. If you cannot stop it, interleave the sides so both see the same noise, and say so in the report.

用你最终会写出的措辞记录预期结论（“在 60k 行数据集上，export 的 p50 快 30%”）。这些问题正是用来检验这句话。
阅读测量脚本。记下它测量、计数和忽略了什么。
使用
```
uptime
```
检查负载平均值，使用
```
nproc
```
检查核心数。如果机器繁忙，查明正在运行什么。如果无法停止这些负载，就交替执行两个方案，使双方承受相同的噪声，并在报告中说明。

The questions

检查问题

Why not double? Name the limiter. Profile in a run you do not report, because profilers and tracers slow the work down. Use CPU per process (
```
top
```
,
```
pidstat
```
), a runtime profiler (
```
node --cpu-prof
```
,
```
py-spy
```
,
```
perf
```
), I/O wait, and syscall counts (
```
strace -c
```
on Linux), then map the hot spot to source. Watch the load generator too: if it saturates first, you measured the load generator. If a change did not move the number, the limiter explains why, so find it before you call the change useless.
Was it tuned? Run every side the way production runs it: release builds, production flags and env, batching and transaction settings, connection pools, caches as warm or cold as production sees them, and the same versions and data. If one side runs on defaults, you compared configurations, not implementations. A limiter that is a setting (a commit per row, a debug build, a missing index) means that side is untuned. Tune it and measure again before you pick a winner. If you cannot tune it, do not pick a winner from that run. When the user is choosing what to adopt, narrowing the claim to the code as it ships today does not fix this, because they adopt the option, not today's settings.
Did it break limits? Do the arithmetic. Compare bytes per second with disk and network bandwidth, operations per second times cost per operation with your cores, and the time saved with the time the changed piece took. Removing a piece that takes 10% of the run can make the run at most about 11% faster. A result past a limit measured something other than the work, such as a cache, a no-op, or a bug.
Did it error? Count failures and non-success responses, and check that outputs are correct, not just present. Rejections are often fast, and timeouts and retries are slow. If the script does not count errors, add the count.
Does it reproduce? Run each side at least 5 times, alternating (A, B, A, B, and so on) so warmup, lazy initialization, caches, and drift favor neither. Report the median and range. A gap smaller than the run-to-run variation is no measurable difference. When the call is close, use a rank-sum test or the harness's own statistics.
Does it matter? Next to any micro result, measure the end-to-end path a user waits on, with realistic data sizes and concurrency, and report the micro result as a share of the whole. A helper that takes 1% of a request can make the request at most 1% faster, however fast the helper gets.
Did it even happen? Confirm the work ran inside the timed region: the request reached the server, the rows were written, the bytes were read, and the code used the result. Lazy code (generators nobody iterates, promises nobody awaits, results the JIT can discard) and timeouts produce numbers for work that never happened.

为什么没有翻倍？ 指明限制因素。在一次不纳入报告的运行中进行 profiling，因为 profiler 和 tracer 会拖慢工作。使用
```
top
```
、
```
pidstat
```
查看各进程的 CPU 使用情况，使用 runtime profiler（
```
node --cpu-prof
```
、
```
py-spy
```
、
```
perf
```
）进行分析，检查 I/O wait 和 syscall 数量（Linux 上使用
```
strace -c
```
），再将热点定位到源码。也要观察负载生成器：如果它先达到饱和，那么测到的就是负载生成器本身。如果某项变更没有改变数值，限制因素可以解释原因；在断定该变更毫无用处之前，先找到它。
是否调优过？ 以生产环境中的运行方式执行每个方案：使用 release 构建、生产 flags 和 env、批处理与事务设置、连接池，cache 的 warm/cold 状态应与生产环境实际情况一致，并使用相同的版本和数据。如果某一方使用默认设置，你比较的就是配置，而不是实现。如果限制因素来自某项设置（每行提交一次、debug build、缺少索引），就说明该方案未经调优。完成调优并重新测量后，才能选出优胜方案。如果无法调优，就不要根据该次运行选出优胜方案。当用户是在选择要采用的方案时，把结论限定为“代码按当前发布配置下的表现”仍不能解决问题，因为用户采用的是这个选项，而不是它今天的设置。
是否突破了上限？ 做算术核对。将字节/秒与磁盘和网络带宽比较，将操作/秒乘以单次操作成本后与核心数比较，并将节省的时间与被修改部分原先所花的时间比较。移除一个占运行时间 10% 的部分，最多只能让运行快约 11%。如果结果超过了某个上限，说明测到的并不是目标工作，例如 cache、no-op 或 bug。
是否发生错误？ 统计失败次数和非成功响应，并检查输出是否正确，而不只是是否存在。拒绝请求往往很快，而超时和重试很慢。如果脚本未统计错误，就补充计数。
结果可复现吗？ 每个方案至少运行 5 次，并交替执行（A、B、A、B，依此类推），避免 warmup、lazy initialization、cache 和 drift 偏向任何一方。报告中位数和范围。如果差距小于运行间波动，则不存在可测量的差异。两者接近时，使用秩和检验（rank-sum test）或 harness 自身的统计结果。
它重要吗？ 除微基准结果外，还要测量用户实际等待的端到端路径，使用符合实际情况的数据规模和并发量，并报告微基准结果占整体的比例。一个占请求耗时 1% 的 helper，最多只能让请求快 1%，无论该 helper 本身变得多快。
工作真的发生了吗？ 确认实际工作发生在计时区间内：请求确实到达服务器，数据行确实被写入，字节确实被读取，代码确实使用了结果。惰性代码（无人迭代的 generator、无人 await 的 promise、可能被 JIT 丢弃的结果）和超时，都会为根本未执行的工作产生性能数据。

Report

报告

Lead with the verdict: faster, slower, no measurable difference, or inconclusive.
Give the number with its unit, the run count, the range, and the limiter. For example, "p50 41 ms → 33 ms, median of 7 runs per side, range 32 to 35 ms after, bound by JSON parsing on one core."
Call the verdict inconclusive when you claim a difference but cannot name the limiter, when a side ran untuned, or when you could not check questions 4 and 7. Name the gap.
Keep a PR body to one primary number, per the Opening a PR playbook. Put the runs, range, and limiter evidence in a linked artifact or notes file.

先给出结论：更快、更慢、无可测量差异或结论不确定。
给出数字、单位、运行次数、范围和限制因素。例如：“p50 41 ms → 33 ms，每侧 7 次运行的中位数，变更后范围为 32 至 35 ms，瓶颈为单核上的 JSON 解析。”
如果你声称存在差异，却无法指出限制因素；某个方案未经调优；或无法检查第 4 和第 7 个问题，则必须将结论标为不确定。明确指出缺失之处。
按照 Opening a PR 手册，PR 正文只保留一个主要数字。将运行数据、范围和限制因素证据放入链接的 artifact 或 notes file。

Other perf material

其他性能材料

Vet the Perf issue playbook's baseline before the playbook plans from it, and every number after that.
Vet the Hillclimb playbook's harness before it is frozen. The frozen harness then prints error and work counts, so each keep-or-revert checks questions 4 and 7 for free.

在 Perf issue 手册依据 baseline 制定方案之前，先审查该 baseline；此后手册产出的每个数字也要审查。
在冻结 Hillclimb 手册的 harness 之前，先对其进行审查。冻结后的 harness 会输出错误计数和工作计数，因此每次做 keep-or-revert 决策时，都能无需额外操作地检查第 4 和第 7 个问题。