handling-failures

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Handling failures

故障处理

REQUIRED BACKGROUND: the
principal-engineering
skill.
必备背景知识:
principal-engineering
技能。

Overview

概述

A swallowed error is a bug with its evidence destroyed. Core contract: every failure path does exactly one of three things, and all three are loud. The system that looks healthy while serving wrong data is worse than the one that crashes, because the crash gets fixed today and the silent wrongness gets discovered in an audit.
被静默吞掉的错误相当于销毁了证据的bug。核心约定:每条故障路径必须执行以下三种操作之一,且所有操作都要做到fail-loud。 看似正常运行但返回错误数据的系统比直接崩溃的系统更糟糕,因为崩溃的问题当天就能得到修复,而静默的错误可能要等到审计时才会被发现。

The contract

约定

Every catch and failure path either:
  1. Logs at WARN or ERROR and rethrows, or
  2. Logs and returns a TYPED failure the caller must handle (a result type, a sealed error, a status the compiler or contract forces downstream code to acknowledge), or
  3. Logs and enters an explicitly documented degraded mode (the degradation is named in the code and its documentation, and something observable says the system is degraded).
Forbidden, no exceptions:
  • Bare catch-and-continue.
  • Catch-and-return-default (empty list, null, zero, cached copy) that masks the failure.
  • ?? fallback
    and its cousins where the fallback hides that the primary failed. A fallback is acceptable only when the absence is ALSO surfaced loudly elsewhere.
Why this is absolute: every one of these converts a detectable failure into silent wrong output, and silent wrong output on a path that matters is the most expensive class of defect a system produces.
每条捕获逻辑和故障路径必须:
  1. 记录WARN或ERROR级别的日志并重新抛出异常,或者
  2. 记录日志并返回调用方必须处理的类型化故障(如结果类型、密封错误、编译器或契约强制下游代码必须确认的状态),或者
  3. 记录日志并进入明确文档化的降级模式(降级状态需在代码和文档中命名,且有可观测指标表明系统处于降级状态)。
绝对禁止:
  • 无处理的捕获后继续执行。
  • 捕获后返回默认值(空列表、null、零值、缓存副本)以掩盖故障。
  • 使用
    ?? fallback
    及其类似语法,且回退逻辑隐藏了主逻辑失败的事实。仅当故障也在其他地方被显性暴露时,回退逻辑才是可接受的。
为何要严格执行:上述每种行为都会将可检测的故障转化为静默的错误输出,而关键路径上的静默错误输出是系统产生的代价最高的缺陷类型。

Corollaries

推论

  • A missing required entry fails loud. Something absent from a registry, config, or catalog is a build or startup failure, never a silent default; otherwise the single source quietly becomes optional.
  • Error states are visible. A workflow must not appear healthy while failing; surface the error state in the UI, the metrics, or the logs someone actually watches.
  • Operator-facing remediation is specific. The error message is read mid-incident; "connection failed, check REDIS_URL and whether redis responds to PING" beats "an error occurred" by the length of the outage.
  • Retries are bounded and observable, and replays of side-effecting operations are idempotent or they multiply the damage (a retry queue replaying charges is how an outage becomes a refund program).
  • Degraded modes have a bound. Skip-and-continue needs the explicit threshold where degradation becomes abort, as a named, operator-tunable constant. The skills demand the guard exists; its value is a judgment call to make with the owner, and an unbounded degraded mode is a slow-motion swallow.
  • On failure paths, observability is part of the minimum, not gold-plating: the log line, the counter, and the alert ship with the fix, because a failure path without them is the silent swallow with better intentions.
  • 缺失必填项时需显性失败。注册表、配置或目录中缺失的内容应导致构建或启动失败,绝不能静默使用默认值;否则单一数据源会悄悄变成可选项。
  • 错误状态必须可见。工作流不能在故障时仍显示正常;需在UI、指标或有人监控的日志中暴露错误状态。
  • 面向运维人员的修复提示需具体。错误消息会在事件响应过程中被查看;"连接失败,请检查REDIS_URL及redis是否响应PING命令"比"发生错误"能大幅缩短故障时长。
  • 重试需有边界且可观测,有副作用的操作重放必须是幂等的,否则会扩大损害(比如重试队列重复执行扣费操作,会让故障演变成退款事件)。
  • 降级模式需有边界。跳过并继续执行的逻辑需要明确的阈值,当降级达到该阈值时需终止执行,该阈值应为可命名、可由运维人员调整的常量。本规范要求必须存在该防护逻辑;具体阈值需与负责人共同决定,无边界的降级模式相当于缓慢的静默吞错。
  • 故障路径的可观测性是最低要求,而非额外优化:日志行、计数器和告警需与修复代码一同交付,因为没有这些的故障路径只是意图良好的静默吞错。

Touching existing swallows

处理已存在的静默吞错

Code you are editing that already swallows: fix it as part of the work, or explicitly flag it as owed with what it hides. Leaving it silently is endorsing it. In review, a NEW silent swallow is an automatic BLOCKER; a pre-existing one you touched and left unflagged is a WARNING against the change.
你正在编辑的代码中若已存在静默吞错:需在本次工作中修复,或明确标记为待处理事项并说明其隐藏的问题。保留静默吞错等同于认可该行为。在代码评审中,新增的静默吞错会直接被阻止;若你修改了存在静默吞错的代码但未标记,则会对该变更发出警告。

Common mistakes

常见错误

  • "It should never happen" as a reason to swallow. Paths that should never happen are exactly the ones that need a loud alarm when they do.
  • Logging at DEBUG and calling it handled. If nobody sees it in production, it is a swallow with extra steps.
  • Catching broad (
    Exception
    ,
    catch {}
    ) to handle narrow. The unexpected failure rides in with the expected one and dies silently beside it.
  • A degraded mode nobody documented. Degradation that only the author knows about is an outage the operator cannot diagnose.
  • Making the test pass by defaulting the failure. The test goes green; the defect graduates to production.
  • 以"这种情况永远不会发生"为理由吞掉错误。那些理论上不会发生的路径恰恰是发生时需要发出强警报的路径。
  • 记录DEBUG级别的日志就认为问题已处理。如果生产环境中没人能看到该日志,那这只是多了步骤的静默吞错。
  • 捕获宽泛的异常(
    Exception
    catch {}
    )来处理特定异常。意外故障会和预期故障一同被捕获并静默消失。
  • 未文档化的降级模式。只有作者知晓的降级状态会导致运维人员无法诊断故障。
  • 通过默认故障值让测试通过。测试变绿,但缺陷会被带到生产环境。