Observability triage

Query Cloudflare Workers Observability for prod errors, count them correctly, and skip the noise that has already been investigated to a dead end. Apply the known-noise list below without re-investigating those entries.

Access

Resolve the account at runtime — never hardcode it: the
```
cloudflare-api
```
MCP server pre-binds
```
accountId
```
in
```
mcp__cloudflare-api__execute
```
, and
```
npx wrangler whoami
```
prints it. The workers to triage are the ones this repo deploys (see
```
alchemy.run.ts
```
): the main app worker plus the aux workers (audit engine, landing, self-host).
Query via the
```
cloudflare-api
```
MCP server (
```
mcp__cloudflare-api__execute
```
). If its tools are absent, run its authenticate flow and give the user the URL — wrangler's OAuth token gets a 403 on the observability API (missing scope), so don't burn time on curl-with-wrangler-token.
PostHog is the second error source but cannot see
```
exceededMemory
```
/
```
canceled
```
/
```
responseStreamDisconnected
```
outcomes — worker-outcome questions are answerable only here.

Query recipes (verified shapes)

POST /accounts/{account_id}/workers/observability/telemetry/query

. All of these are load-bearing; the API's 400s are opaque:

```
queryId: "adhoc"
```
is required. Timeframe is epoch milliseconds:
```
timeframe: { from, to }
```
.

Count invocations by outcome:

json

{
  "queryId": "adhoc",
  "timeframe": { "from": 0, "to": 0 },
  "parameters": {
    "datasets": ["cloudflare-workers"],
    "filters": [
      { "key": "$metadata.type", "operation": "eq", "value": "cf-worker-event", "type": "string" }
    ],
    "calculations": [{ "operator": "count", "alias": "count" }],
    "groupBys": [
      { "value": "$workers.scriptName", "type": "string" },
      { "value": "$workers.outcome", "type": "string" }
    ],
    "limit": 100
  },
  "view": "calculations",
  "limit": 100
}

The
```
cf-worker-event
```
filter is what makes counts mean invocations; without it you count every log line.

Raw samples:

view: "events"

, empty

calculations

groupBys

, small

limit

. Events carry

$metadata

(

service

trigger

level

message

fingerprint

requestId

) and

$workers

(

outcome

scriptName

scriptVersion

wallTimeMs

cpuTimeMs

eventType

Error-level app logs grouped by message: filter
```
$metadata.level eq error
```
, groupBy
```
$metadata.message
```
.
Verified filter operations:
```
eq
```
,
```
neq
```
. Percentiles: the API wants
```
"median"
```
, not
```
"p50"
```
. Filter for substrings client-side on fetched events.

Useful drill-downs: groupBy

$metadata.trigger

(route),

$workers.scriptVersion

(did a deploy change the rate mid-window),

$workers.event.request.headers.user-agent

Counting gotchas

Grouped results are unsorted and effectively capped (~10 rows returned regardless of
limit
). A missing group ≠ zero. To see error outcomes, add
```
$workers.outcome neq ok
```
instead of hoping the error rows make the cut; sort client-side.
One isolate death fans out. An OOM kills every request pinned to the isolate at the same instant — cluster raw OOM events by timestamp before reading the count as user impact.
Message prefixes fragment groups. Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.
Prod deploys are manual — main being fixed doesn't mean prod runs the fix. Check
```
$workers.scriptVersion
```
and the scripts'
```
modified_on
```
before concluding a fix didn't work.

Known noise — filter these out, do not re-investigate

Entries land here only after an investigation proved there is no first-party emit site to fix or demote. Each keeps the one condition that would make it real signal again.

1. SAM chat Durable Object lifecycle (close code 1006)

Messages (one phenomenon, counted three ways):

Connection closed: this Durable Object instance is no longer active. Reconnect or retry the request.

(hibernation/eviction),

Durable Object reset because its code was updated.

(deploy), plus the paired invocation summary whose

$metadata.error

close

Identify by:

eventType: "hibernatableWebSocket"

, entrypoint

SamChatAgent

OnboardingChatAgent

webSocketType: "close", code: 1006, wasClean: false

outcome: "exception"

, single-digit

wallTimeMs

cpuTimeMs: 0

, no stack. Fingerprints

3aa4cac26653d09a0a41100a33d413ae

(exception),

0ae15457af49b4d9a117eeecf66b040a

(summary).

Why unfixable: the DO is destroyed under the JS — its IoContext is already aborted when workerd delivers
```
webSocketClose
```
, so the first
```
await
```
never settles.
```
partyserver
```
already try/catches the whole close path; an
```
onClose
```
override would catch nothing.
Nothing breaks: transcripts persist per message in DO SQLite, PartySocket reconnects unconditionally, and credit metering (
```
onChatResponse
```
) never fires on an aborted turn.
Real signal: a sustained rise that does not correlate with a deploy (would mean mid-conversation evictions beyond hibernation).

Network connection lost.

Identify by: fingerprint
```
be89d4ff7a64cb4d4dceae0f51cfe708
```
, or the message verbatim. Runtime-generated shape:
```
source.level
```
absent,
```
source.exception
```
present,
```
$metadata.origin: "fetch"
```
— the opposite of every app log (
```
source.level
```
present, no
```
exception
```
).
Why unfixable: workerd's own record of a client disconnecting mid-stream; the exception never enters app code — the sibling request event for the same
```
requestId
```
has
```
outcome: "ok"
```
. No emit site exists;
```
wrangler.jsonc
```
observability config has no per-message filter. Do not add try/catch around the stream handlers (dead ceremony).
Real signal: if the
```
/mcp
```
share of this group grows, treat it as a tool-call-latency symptom (clients timing out and cancelling) and route it to the /mcp performance track — not to this log group.

Runtime-vs-app litmus test

Before investigating any unfamiliar error event:

source.level

absent +

source.exception

present ⇒ the Workers runtime wrote it, not the app. There is no call site to grep for; judge it by the sibling request's

outcome

Adding an entry

Add to this list only after establishing there is no first-party emit site (grep the message; check the runtime-vs-app litmus above) and nothing is left in a wrong state. Every entry must include identify-by markers (fingerprint if stable) and its real-signal condition.

observability-triage

NPX Install

Tags

SKILL.md Content

Observability triage

Access

Query recipes (verified shapes)

Counting gotchas

Known noise — filter these out, do not re-investigate

1. SAM chat Durable Object lifecycle (close code 1006)

2.
`Network connection lost.`

Runtime-vs-app litmus test

Adding an entry

observability-triage

NPX Install

Tags

SKILL.md Content

Observability triage

Access

Query recipes (verified shapes)

Counting gotchas

Known noise — filter these out, do not re-investigate

1. SAM chat Durable Object lifecycle (close code 1006)

2. Network connection lost.

Runtime-vs-app litmus test

Adding an entry

2.
`Network connection lost.`