Back to Details

mandela

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
Audit a validation for leakage: does outside ground-truth actually enter, or is everyone confirming a result no one independently produced?
对验证过程进行泄露审计：外部真实数据是否真正输入，还是所有人都在确认一个无人独立生成的结果？

Goal

目标

The name is the Mandela Effect — a whole population confidently remembers something that never independently happened; a leaky validation is the same shape. Walk the 8 patterns below. 
mandela
checks one thing: whether a validation is independent, or whether the designer, model, and scorer are only confirming each other.
这个方法被称为Mandela Effect——一群人“确信地记得”某件从未独立发生过的事情；存在泄露的验证也是同样的逻辑。请遵循以下8种模式。
mandela
只检查一件事：验证过程是否独立，还是设计者、模型和评分器只是在互相印证。

Workflow

工作流程

  1. Identify the validation (eval / metric / experiment / holdout / "how we'll know"). Name its components — what plays model, scorer, designer, dataset.
  2. Ask the core question: does external ground-truth enter independently?
  3. Test the validation against all 8 patterns below (some apply only to certain components — a human subject, a scorer); report only the ones that fire, each by name.
  4. Give the independent-ground-truth fix for each hit.
  1. 确定验证对象（评估/指标/实验/保留数据集/“验证成效的方式”）。明确其组成部分——即扮演模型、评分器、设计者、数据集的角色。
  2. 提出核心问题：外部真实数据是否独立输入？
  3. 将验证对象与以下所有8种模式进行比对（部分模式仅适用于特定组件——如人类受试者、评分器）；仅报告触发的模式，并注明每种模式的名称。
  4. 为每个触发的模式提供基于独立真实数据的修复方案。

The 8 leakage patterns

8种泄露模式

  1. Recall, not reason — a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
  2. Wrong null hypothesis — an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
  3. Shared hallucination — two components verifying each other; circularity reported as a number.
  4. Tautology — a scorer grading buckets it drew itself.
  5. Verifier = designer — a private, unreproducible recipe in a holdout's clothes.
  6. Shared-pool bias — train and holdout drawn from one labeler pool, so one bias enters both sides.
  7. Frame injection — a question that hands the subject the hypothesis.
  8. Demand characteristics — measured subjects who know they're being measured.
  1. 回忆而非推理——复述已记忆的答案，而非实际推导得出；系统早已知道它本应计算的结果。
  2. 错误的零假设——移除了表面标签但未移除系统实际利用的底层信号的消融实验，因此“对照组”仍存在泄露。
  3. 共同幻觉——两个组件互相验证；将循环逻辑转化为数值结果。
  4. 同义反复——评分器为自己划分的分组打分。
  5. 验证者=设计者——披着保留数据集外衣的私有、不可复现的流程。
  6. 共享池偏差——训练集和保留数据集来自同一标注者池，因此同一偏差会进入双方。
  7. 框架注入——向受试者直接提供假设的问题。
  8. 需求特征——被测量的受试者知道自己正在被测量。

Rules

规则

  • Subtlety that bites twice: you can blind the output value and still leak the collection recipe.
  • Read-only — name the leak and the independence fix; don't rewrite the experiment.
  • 双重陷阱：即使对输出值进行盲处理，仍可能泄露收集流程。
  • 仅可读——指出泄露问题和独立修复方案；不要重写实验。

Verification

验证

Turn mandela on this audit:
  1. Run patterns #3–#5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? is the verifier the designer (#5)? is your verdict a shared hallucination with the design's own claims (#3)?
  2. Could a reader who didn't run the audit reach your verdict from the cited evidence alone — independent ground-truth?
  3. The report names the root, not a laundry list.
用mandela审计自身：
  1. 对自身应用模式#3–#5：你是否在为自己划分的分组打分（同义反复，#4）？验证者是否就是设计者（#5）？你的结论是否与设计本身的主张构成共同幻觉（#3）？
  2. 未参与审计的读者能否仅从引用的证据得出你的结论——即是否基于独立真实数据？
  3. 报告应指出根源问题，而非罗列问题清单。