observability-triage

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Observability triage

可观测性错误分类

Query Cloudflare Workers Observability for prod errors, count them correctly, and skip the noise that has already been investigated to a dead end. Apply the known-noise list below without re-investigating those entries.

查询Cloudflare Workers可观测性中的生产错误，准确统计错误数量，并跳过已排查至无解的噪音条目。自动应用下方的已知噪音列表，无需重新调查这些条目。

Access

访问方式

Resolve the account at runtime — never hardcode it: the
```
cloudflare-api
```
MCP server pre-binds
```
accountId
```
in
```
mcp__cloudflare-api__execute
```
, and
```
npx wrangler whoami
```
prints it. The workers to triage are the ones this repo deploys (see
```
alchemy.run.ts
```
): the main app worker plus the aux workers (audit engine, landing, self-host).
Query via the
```
cloudflare-api
```
MCP server (
```
mcp__cloudflare-api__execute
```
). If its tools are absent, run its authenticate flow and give the user the URL — wrangler's OAuth token gets a 403 on the observability API (missing scope), so don't burn time on curl-with-wrangler-token.
PostHog is the second error source but cannot see
```
exceededMemory
```
/
```
canceled
```
/
```
responseStreamDisconnected
```
outcomes — worker-outcome questions are answerable only here.

运行时解析账户信息——切勿硬编码：
```
cloudflare-api
```
MCP服务器会在
```
mcp__cloudflare-api__execute
```
中预先绑定
```
accountId
```
，
```
npx wrangler whoami
```
命令可打印该ID。需要分类的Worker是本仓库部署的Worker（详见
```
alchemy.run.ts
```
）：主应用Worker以及辅助Worker（审计引擎、着陆页、自托管）。
通过
```
cloudflare-api
```
MCP服务器（
```
mcp__cloudflare-api__execute
```
）进行查询。若缺少相关工具，运行其认证流程并向用户提供URL——wrangler的OAuth令牌在可观测性API上会返回403（权限范围不足），因此不要浪费时间使用带wrangler令牌的curl命令。
PostHog是第二个错误来源，但无法查看
```
exceededMemory
```
/
```
canceled
```
/
```
responseStreamDisconnected
```
结果——关于Worker结果的问题只能在此处解答。

Query recipes (verified shapes)

经过验证的查询方案

POST /accounts/{account_id}/workers/observability/telemetry/query

. All of these are load-bearing; the API's 400s are opaque:

```
queryId: "adhoc"
```
is required. Timeframe is epoch milliseconds:
```
timeframe: { from, to }
```
.

Count invocations by outcome:

json

{
  "queryId": "adhoc",
  "timeframe": { "from": 0, "to": 0 },
  "parameters": {
    "datasets": ["cloudflare-workers"],
    "filters": [
      { "key": "$metadata.type", "operation": "eq", "value": "cf-worker-event", "type": "string" }
    ],
    "calculations": [{ "operator": "count", "alias": "count" }],
    "groupBys": [
      { "value": "$workers.scriptName", "type": "string" },
      { "value": "$workers.outcome", "type": "string" }
    ],
    "limit": 100
  },
  "view": "calculations",
  "limit": 100
}

The
```
cf-worker-event
```
filter is what makes counts mean invocations; without it you count every log line.

Raw samples:

view: "events"

, empty

calculations

groupBys

, small

limit

. Events carry

$metadata

(

service

trigger

level

message

fingerprint

requestId

) and

$workers

(

outcome

scriptName

scriptVersion

wallTimeMs

cpuTimeMs

eventType

Error-level app logs grouped by message: filter
```
$metadata.level eq error
```
, groupBy
```
$metadata.message
```
.
Verified filter operations:
```
eq
```
,
```
neq
```
. Percentiles: the API wants
```
"median"
```
, not
```
"p50"
```
. Filter for substrings client-side on fetched events.

Useful drill-downs: groupBy

$metadata.trigger

(route),

$workers.scriptVersion

(did a deploy change the rate mid-window),

$workers.event.request.headers.user-agent

POST /accounts/{account_id}/workers/observability/telemetry/query

。以下所有配置都是必需的；API返回的400错误信息不透明：

必须包含
```
queryId: "adhoc"
```
。时间范围为毫秒级时间戳：
```
timeframe: { from, to }
```
。

按结果统计调用次数：

json

{
  "queryId": "adhoc",
  "timeframe": { "from": 0, "to": 0 },
  "parameters": {
    "datasets": ["cloudflare-workers"],
    "filters": [
      { "key": "$metadata.type", "operation": "eq", "value": "cf-worker-event", "type": "string" }
    ],
    "calculations": [{ "operator": "count", "alias": "count" }],
    "groupBys": [
      { "value": "$workers.scriptName", "type": "string" },
      { "value": "$workers.outcome", "type": "string" }
    ],
    "limit": 100
  },
  "view": "calculations",
  "limit": 100
}

```
cf-worker-event
```
过滤器是确保统计结果代表调用次数的关键；没有它的话，统计的是每条日志行的数量。

原始样本：

view: "events"

，

calculations

groupBys

为空，设置较小的

limit

。事件包含

$metadata

（

service

、

trigger

、

level

、

message

、

fingerprint

、

requestId

）和

$workers

（

outcome

、

scriptName

、

scriptVersion

、

wallTimeMs

、

cpuTimeMs

、

eventType

）字段。

按消息分组的错误级应用日志：过滤
```
$metadata.level eq error
```
，按
```
$metadata.message
```
分组。
经过验证的过滤操作：
```
eq
```
、
```
neq
```
。百分位数：API要求使用
```
"median"
```
而非
```
"p50"
```
。对获取到的事件在客户端进行子字符串过滤。
有用的深入分析维度：按
```
$metadata.trigger
```
（路由）、
```
$workers.scriptVersion
```
（部署是否在时段内改变了错误率）、
```
$workers.event.request.headers.user-agent
```
分组。

Counting gotchas

计数注意事项

Grouped results are unsorted and effectively capped (~10 rows returned regardless of
limit
). A missing group ≠ zero. To see error outcomes, add
```
$workers.outcome neq ok
```
instead of hoping the error rows make the cut; sort client-side.
One isolate death fans out. An OOM kills every request pinned to the isolate at the same instant — cluster raw OOM events by timestamp before reading the count as user impact.
Message prefixes fragment groups. Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.
Prod deploys are manual — main being fixed doesn't mean prod runs the fix. Check
```
$workers.scriptVersion
```
and the scripts'
```
modified_on
```
before concluding a fix didn't work.

分组结果未排序且实际上限固定（无论设置
limit
多少，仅返回约10行）。缺失的分组≠数量为0。要查看错误结果，添加
```
$workers.outcome neq ok
```
过滤条件，而非寄希望于错误行能被包含在内；在客户端进行排序。
单个隔离进程死亡会扩散影响。OOM会同时终止绑定到该隔离进程的所有请求——在将计数视为用户影响前，先按时间戳对原始OOM事件进行聚类。
消息前缀会拆分分组。带有前置时间戳的日志（better-auth的格式）会将一个错误拆分为N个计数为1的分组；按消息分组的统计会严重低估错误数量。需在客户端对事件样本进行合并。
生产环境部署是手动的——主分支修复不代表生产环境运行了修复版本。在得出修复无效的结论前，先检查
```
$workers.scriptVersion
```
和脚本的
```
modified_on
```
字段。

Known noise — filter these out, do not re-investigate

已知噪音——过滤这些条目，无需重新调查

Entries land here only after an investigation proved there is no first-party emit site to fix or demote. Each keeps the one condition that would make it real signal again.

只有在调查确认没有可修复或降级的第一方触发点后，才可将条目添加至此列表。每个条目需保留一个可使其再次成为真实信号的条件。

1. SAM chat Durable Object lifecycle (close code 1006)

1. SAM聊天Durable Object生命周期（关闭代码1006）

Messages (one phenomenon, counted three ways):

Connection closed: this Durable Object instance is no longer active. Reconnect or retry the request.

(hibernation/eviction),

Durable Object reset because its code was updated.

(deploy), plus the paired invocation summary whose

$metadata.error

close

Identify by:

eventType: "hibernatableWebSocket"

, entrypoint

SamChatAgent

OnboardingChatAgent

webSocketType: "close", code: 1006, wasClean: false

outcome: "exception"

, single-digit

wallTimeMs

cpuTimeMs: 0

, no stack. Fingerprints

3aa4cac26653d09a0a41100a33d413ae

(exception),

0ae15457af49b4d9a117eeecf66b040a

(summary).

Why unfixable: the DO is destroyed under the JS — its IoContext is already aborted when workerd delivers
```
webSocketClose
```
, so the first
```
await
```
never settles.
```
partyserver
```
already try/catches the whole close path; an
```
onClose
```
override would catch nothing.
Nothing breaks: transcripts persist per message in DO SQLite, PartySocket reconnects unconditionally, and credit metering (
```
onChatResponse
```
) never fires on an aborted turn.
Real signal: a sustained rise that does not correlate with a deploy (would mean mid-conversation evictions beyond hibernation).

消息（同一现象的三种统计方式）：

Connection closed: this Durable Object instance is no longer active. Reconnect or retry the request.

（休眠/驱逐）、

Durable Object reset because its code was updated.

（部署），以及对应的调用摘要，其

$metadata.error

为

close

。

识别方式：

eventType: "hibernatableWebSocket"

，入口点

SamChatAgent

OnboardingChatAgent

，

webSocketType: "close", code: 1006, wasClean: false

，

outcome: "exception"

，个位数的

wallTimeMs

，

cpuTimeMs: 0

，无堆栈信息。指纹

3aa4cac26653d09a0a41100a33d413ae

（异常）、

0ae15457af49b4d9a117eeecf66b040a

（摘要）。

无法修复原因：DO在JS层面被销毁——当workerd传递
```
webSocketClose
```
时，其IoContext已终止，因此第一个
```
await
```
永远不会完成。
```
partyserver
```
已对整个关闭路径进行了try/catch处理；覆盖
```
onClose
```
不会捕获任何内容。
无影响：对话记录按消息持久化在DO SQLite中，PartySocket会无条件重新连接，且信用计量（
```
onChatResponse
```
）不会在终止的会话轮次中触发。
真实信号：持续增长且与部署无关（意味着对话中途出现超出休眠的驱逐情况）。

Network connection lost.

Network connection lost.

Identify by: fingerprint
```
be89d4ff7a64cb4d4dceae0f51cfe708
```
, or the message verbatim. Runtime-generated shape:
```
source.level
```
absent,
```
source.exception
```
present,
```
$metadata.origin: "fetch"
```
— the opposite of every app log (
```
source.level
```
present, no
```
exception
```
).
Why unfixable: workerd's own record of a client disconnecting mid-stream; the exception never enters app code — the sibling request event for the same
```
requestId
```
has
```
outcome: "ok"
```
. No emit site exists;
```
wrangler.jsonc
```
observability config has no per-message filter. Do not add try/catch around the stream handlers (dead ceremony).
Real signal: if the
```
/mcp
```
share of this group grows, treat it as a tool-call-latency symptom (clients timing out and cancelling) and route it to the /mcp performance track — not to this log group.

识别方式：指纹
```
be89d4ff7a64cb4d4dceae0f51cfe708
```
，或完全匹配的消息。运行时生成的特征：缺失
```
source.level
```
，存在
```
source.exception
```
，
```
$metadata.origin: "fetch"
```
——与所有应用日志相反（存在
```
source.level
```
，无
```
exception
```
）。
无法修复原因：这是workerd自身记录的客户端中途断开连接的情况；该异常从未进入应用代码——同一
```
requestId
```
的关联请求事件的
```
outcome: "ok"
```
。不存在触发点；
```
wrangler.jsonc
```
可观测性配置没有针对单条消息的过滤器。无需在流处理器周围添加try/catch（无意义的形式）。
真实信号：如果该组中
```
/mcp
```
的占比增长，则将其视为工具调用延迟的症状（客户端超时并取消请求），并将其路由至/mcp性能跟踪——而非此日志组。

Runtime-vs-app litmus test

运行时日志vs应用日志测试

Before investigating any unfamiliar error event:

source.level

absent +

source.exception

present ⇒ the Workers runtime wrote it, not the app. There is no call site to grep for; judge it by the sibling request's

outcome

在调查任何不熟悉的错误事件前：若

source.level

缺失且

source.exception

存在 ⇒ 该日志由Workers运行时生成，而非应用。不存在可搜索的调用点；需通过关联请求的

outcome

判断。

Adding an entry

添加条目

Add to this list only after establishing there is no first-party emit site (grep the message; check the runtime-vs-app litmus above) and nothing is left in a wrong state. Every entry must include identify-by markers (fingerprint if stable) and its real-signal condition.

只有在确认没有第一方触发点（搜索消息内容；通过上述运行时vs应用日志测试验证）且没有遗留异常状态后，才可添加至此列表。每个条目必须包含识别标记（若指纹稳定则包含指纹）及其真实信号条件。

observability-triage

Original

Translation

Observability triage

可观测性错误分类

Access

访问方式

Query recipes (verified shapes)

经过验证的查询方案

Counting gotchas

计数注意事项

Known noise — filter these out, do not re-investigate

已知噪音——过滤这些条目，无需重新调查

1. SAM chat Durable Object lifecycle (close code 1006)

1. SAM聊天Durable Object生命周期（关闭代码1006）

2.
`Network connection lost.`

2.
`Network connection lost.`

Runtime-vs-app litmus test

运行时日志vs应用日志测试

Adding an entry

添加条目

observability-triage

Original

Translation

Observability triage

可观测性错误分类

Access

访问方式

Query recipes (verified shapes)

经过验证的查询方案

Counting gotchas

计数注意事项

Known noise — filter these out, do not re-investigate

已知噪音——过滤这些条目，无需重新调查

1. SAM chat Durable Object lifecycle (close code 1006)

1. SAM聊天Durable Object生命周期（关闭代码1006）

2. Network connection lost.

2. Network connection lost.

Runtime-vs-app litmus test

运行时日志vs应用日志测试

Adding an entry

添加条目

2.
`Network connection lost.`

2.
`Network connection lost.`