diagnosing-failed-warehouse-syncs
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseDiagnosing failed data warehouse syncs
诊断失败的数据仓库同步
Work top-down when a data warehouse source or table is failing, stuck, or producing bad data: source → schema →
recovery action. Do not jump straight to "resync from scratch" — that discards synced data and restarts from
zero, which is rarely the right first step.
当数据仓库中的 source 或 table 同步失败、卡住或产生错误数据时,应按 source → schema → 恢复操作的顺序自顶向下排查。不要直接执行“从头 resync”——这会丢弃已同步的数据并从零重启,很少是正确的第一步。
When to use this skill
何时使用此 skill
- The user reports a specific sync is failing (e.g. "my Stripe source is red")
- A table has been in state far longer than expected
Running - Data in a warehouse table is stale, missing rows, or looks corrupt
- Latest rows aren't appearing despite the schema being marked
Completed - The user is choosing between cancel / reload / resync / delete-data and isn't sure which
- Another skill — typically — has surfaced a failing source or schema and the user wants to dig into it
auditing-warehouse-source-health
Both entry points (user-reported and audit-handoff) use the same workflow; the audit just means you already know
which item to diagnose and can skip Step 1's discovery search.
- 用户报告某个具体同步失败(例如“我的 Stripe source 标红了”)
- 某张 table 处于 状态的时间远超预期
Running - 数据仓库 table 中的数据过期、缺少行或看起来已损坏
- schema 已标记为 ,但最新行仍未出现
Completed - 用户正在 cancel / reload / resync / delete-data 之间选择,但不确定应该采用哪一种
- 另一个 skill(通常是 )发现某个 source 或 schema 存在故障,用户希望进一步排查
auditing-warehouse-source-health
两种入口(用户直接报告和审计移交)使用相同的工作流;审计移交只是意味着你已经知道需要诊断的项目,因此可以跳过步骤 1 中的发现搜索。
Available tools
可用工具
| Tool | Purpose |
|---|---|
| List all sources with connection status and latest error |
| Full details for one source including all its schemas |
| All table schemas across all sources, with per-table status + latest_error |
| Full details for one schema including sync_type_config |
| Cancel a sync currently in |
| Trigger a sync using the configured sync method (respects incremental) |
| Full resync — wipes synced data and restarts. Destructive |
| Delete the synced table but keep the schema entry |
| Change sync_type / incremental_field / cdc_table_mode |
| Update a source's credentials ( |
| Retrigger syncs for every enabled schema on a source |
| Re-fetch the source's table list to pick up new tables |
| Verify Postgres CDC setup for a source |
| Refresh candidate incremental fields when the source schema has changed |
| Check webhook registration state and external service status |
| Re-register a webhook that was lost or never registered |
| Update the signing secret after rotation on the source side |
| Remove a broken webhook before re-registering |
| 工具 | 用途 |
|---|---|
| 列出所有 source,并显示连接状态和最新错误 |
| 获取一个 source 的完整详情,包括其所有 schema |
| 列出所有 source 下的 table schema,并显示每个 table 的状态和 |
| 获取一个 schema 的完整详情,包括 |
| 取消当前处于 |
| 使用已配置的同步方式触发同步(遵循增量策略) |
| 完整重新同步——清除已同步的数据并重新开始。破坏性操作 |
| 删除已同步的 table,但保留 schema 条目 |
| 修改 |
| 轮换凭证后,更新 source 的凭证( |
| 为某个 source 上所有已启用的 schema 重新触发同步 |
| 重新获取 source 的 table 列表,以识别新增的 table |
| 验证某个 source 的 Postgres CDC 设置 |
| 当 source schema 发生变化时,刷新候选增量字段 |
| 检查 webhook 注册状态和外部服务状态 |
| 重新注册已丢失或从未注册的 webhook |
| 在 source 端轮换 signing secret 后进行更新 |
| 在重新注册前删除损坏的 webhook |
Workflow
工作流
Step 1 — Locate the failing item
步骤 1——定位故障项
If the user named a source, go straight to . Otherwise start with
and to find what's red.
external-data-sources-retrieveexternal-data-sources-listexternal-data-schemas-listTwo kinds of failure:
- Source-level (): the connection itself is broken — credentials expired, host unreachable, account disabled. Affects every table.
ExternalDataSource.status = "Error" - Schema-level — the source connects fine but one or more tables are failing. In the serialized API response
from , look for
external-data-schemas-listvaluesstatus,"Failed", or"Billing limits". (The underlying model enum values are"Billing limits too low"andBillingLimitReached, but the serializer rewrites them — match on both the human-readable and enum forms to be safe.)BillingLimitTooLow
A source can look at the top level while one of its schemas is — always check both.
CompletedFailed如果用户指定了 source,直接调用 。否则先调用 和 ,找出标红的项目。
external-data-sources-retrieveexternal-data-sources-listexternal-data-schemas-list故障分为两种:
- Source 级():连接本身已断开——凭证过期、主机不可达或账户被禁用。这会影响该 source 下的所有 table。
ExternalDataSource.status = "Error" - Schema 级——source 连接正常,但一个或多个 table 同步失败。在 返回的序列化 API 响应中,查找
external-data-schemas-list、Failed或Billing limits等Billing limits too low值。(底层模型枚举值为status和BillingLimitReached,但序列化器会重写这些值——为稳妥起见,应同时匹配人类可读形式和枚举形式。)BillingLimitTooLow
一个 source 在顶层可能显示为 ,但其某个 schema 仍为 ——必须同时检查两者。
CompletedFailedStep 2 — Classify the schema status
步骤 2——对 schema 状态进行分类
From , each schema has a :
external-data-schemas-liststatus| Status | Meaning | Usually means |
|---|---|---|
| Sync currently executing | Normal, unless stuck for hours |
| Last sync finished successfully | Healthy |
| Last sync errored — see | Needs diagnosis |
| User disabled sync ( | Intentional |
| Team hit its warehouse row quota | Billing issue, not a technical failure |
| Team has insufficient credit | Billing issue |
Always check alongside status. A schema in with from 12 hours ago is
almost certainly stuck, even though the status isn't .
last_synced_atRunninglast_synced_atFailed在 中,每个 schema 都有一个 :
external-data-schemas-liststatus| 状态 | 含义 | 通常表示 |
|---|---|---|
| 同步当前正在执行 | 正常,除非已卡住数小时 |
| 上次同步已成功完成 | 健康 |
| 上次同步发生错误——请查看 | 需要诊断 |
| 用户已禁用同步( | 有意为之 |
| 团队已达到数据仓库行数配额 | 账单问题,而非技术故障 |
| 团队信用额度不足 | 账单问题 |
检查状态时,始终应同时查看 。如果某个 schema 仍为 ,但 已是 12 小时前,那么它几乎可以肯定是卡住了,即使状态不是 。
last_synced_atRunninglast_synced_atFailedStep 3 — Interpret latest_error
latest_error步骤 3——解读 latest_error
latest_errorMap the string to a root cause. Common patterns:
latest_error| Error substring | Root cause | Fix |
|---|---|---|
| Credentials expired or rotated | User rotates creds, then |
| SSH tunnel misconfigured or remote host down | User checks SSH host/key/bastion |
| Table has no PK and sync_type is | Either add PK in source, or switch schema to |
| Declared PK columns aren't actually unique | Pick different PK columns via |
| Source's saved integration was deleted | Recreate the source |
| Schema drift — incremental field or tracked column removed | Use |
| Source table was dropped/renamed | Remove schema or rename source-side |
| Network / firewall / host reachability | User side — check host/port/allowlist |
| CDC prerequisites broken | Run |
| Billing limit | Upgrade plan or disable the schema |
If is null but the schema is , retrieve the schema directly — the error may only be populated
on the detail view.
latest_errorFailed将 字符串映射到根因。常见模式如下:
latest_error| 错误子串 | 根因 | 修复方法 |
|---|---|---|
| 凭证过期或已轮换 | 用户轮换凭证,然后调用 |
| SSH tunnel 配置错误或远程主机已宕机 | 用户检查 SSH host、key 和 bastion |
| Table 没有 PK,且 | 在 source 中添加 PK,或将 schema 切换为 |
| 声明的 PK 列实际上并不唯一 | 通过 |
| Source 中保存的 integration 已被删除 | 重新创建 source |
| Schema 漂移——增量字段或受跟踪的列已被删除 | 使用 |
| Source table 已被删除或重命名 | 删除 schema,或在 source 端重命名 |
| 网络、防火墙或主机可达性问题 | 用户侧处理——检查 host、port 和 allowlist |
| CDC 前置条件已损坏 | 运行 |
| 账单限制 | 升级套餐或禁用该 schema |
如果 为 null,但 schema 状态为 ,请直接获取 schema 详情——错误可能只填充在详情视图中。
latest_errorFailedStep 4 — Pick the recovery action
步骤 4——选择恢复操作
The recovery action depends on root cause, not just status. Match the user's situation to one of these:
A. Transient failure (network blip, temporary API outage)
- Data synced so far is still valid.
- Action: to retry using the configured sync method.
external-data-schemas-reload - Incremental/append syncs pick up where they left off.
B. Credentials expired or rotated
- Every schema under the source is failing with an auth error.
- Action: user rotates creds → with the new
external-data-sources-partial-update→ the reload happens automatically when the source status flips back to running, or trigger manually withjob_inputs.external-data-sources-reload
C. Schema drift — column renamed, dropped, or type changed
- Error mentions a specific column that no longer matches the source.
- Action: to get the current fields, then
external-data-schemas-incremental-fields-createwith the correctedexternal-data-schemas-partial-update/incremental_field/incremental_field_type. Usually no need to wipe data.primary_key_columns
C2. Added / renamed tables in the source database
- User mentions "I added a new table to Postgres but it isn't appearing", or a source table was renamed.
- Action: to pick up the new table list, then configure sync on any new schemas.
external-data-sources-refresh-schemas
D. Incremental state is wrong (duplicates, missing rows, data looks corrupt)
- Schema status may be — this isn't a "failure" per se, it's bad data.
Completed - Action: to wipe synced data and re-import from source. Destructive but often the right call for data-quality issues.
external-data-schemas-resync
E. CDC pipeline broken on Postgres
- Error mentions replication slot, publication, WAL.
- Action: to enumerate what's broken, fix on the Postgres side, then
external-data-sources-check-cdc-prerequisites-create. If the WAL position was lost, aexternal-data-schemas-reloadis sometimes unavoidable.resync
F. Sync is stuck in for hours
Running- Check . If it's hours old and still
last_synced_at, the job is orphaned.Running - Action: to stop it, then
external-data-schemas-cancel.external-data-schemas-reload
G. Table data is corrupt but you want to keep the schema config
- Action: to drop the synced table but preserve the schema entry. Next reload re-imports from scratch without losing the configured sync_type/incremental_field.
external-data-schemas-delete-data
H. Billing limit
- Action isn't technical. Explain the limit, recommend upgrading the plan or disabling lower-priority schemas so the important ones fit under quota.
I. Webhook-backed schema isn't receiving events
- Symptoms: schema has , initial bulk sync finished, but no new rows arrive despite activity on the source side. Status may still read
sync_type: "webhook"because the bulk sync (the safety-net cadence) is succeeding — the problem is the push path.Completed - Action:
- .
external-data-sources-webhook-info-retrieve({source_id}) - If → the webhook was never registered, or was deleted. Call
exists: falseto register it.create-webhook-create - If but
exists: trueis set → typically "API key doesn't have permission to read webhooks" or similar. The webhook may have been deleted on the source's dashboard. Re-create it.external_status.error - If isn't
external_status.status→ the source disabled the webhook (usually after repeated delivery failures). Re-enable or re-register."enabled" - If payloads are arriving but failing signature verification → the signing secret was rotated. Get the new
one from the source's dashboard and call .
update-webhook-inputs-create({source_id}, {inputs: {signing_secret: "..."}})
- After any fix, check the source's webhook logs (on their side) to confirm PostHog is now responding 2xx.
恢复操作取决于根因,而不只是状态。请将用户的情况与以下选项之一匹配:
A. 暂时性故障(网络波动、临时 API 中断)
- 目前已同步的数据仍然有效。
- 操作:使用 ,按已配置的同步方式重试。
external-data-schemas-reload - 增量/追加同步会从上次中断的位置继续。
B. 凭证过期或已轮换
- Source 下的每个 schema 都在出现认证错误。
- 操作:用户轮换凭证 → 调用 并传入新的
external-data-sources-partial-update→ 当 source 状态恢复为 running 时会自动执行 reload,也可以手动调用job_inputs。external-data-sources-reload
C. Schema 漂移——列被重命名、删除或更改了类型
- 错误提到某个具体列,该列已与 source 不匹配。
- 操作:使用 获取当前字段,然后调用
external-data-schemas-incremental-fields-create,更新正确的external-data-schemas-partial-update/incremental_field/incremental_field_type。通常不需要清除数据。primary_key_columns
C2. Source 数据库中新增或重命名了 table
- 用户提到“我在 Postgres 中新增了一张表,但它没有出现”,或者 source 中的 table 被重命名。
- 操作:使用 获取新的 table 列表,然后为任何新增 schema 配置同步。
external-data-sources-refresh-schemas
D. 增量状态错误(重复数据、缺少行、数据看似损坏)
- Schema 状态可能为 ——严格来说,这不算“故障”,而是数据质量不佳。
Completed - 操作:使用 清除已同步的数据,并从 source 重新导入。该操作具有破坏性,但通常是数据质量问题的正确选择。
external-data-schemas-resync
E. Postgres 上的 CDC pipeline 已损坏
- 错误提到 replication slot、publication 或 WAL。
- 操作:使用 找出损坏的项目,在 Postgres 端进行修复,然后调用
external-data-sources-check-cdc-prerequisites-create。如果 WAL position 已丢失,有时不可避免地需要执行external-data-schemas-reload。resync
F. 同步卡在 状态数小时
Running- 检查 。如果它已是数小时前且状态仍为
last_synced_at,则该 job 已成为孤儿任务。Running - 操作:使用 停止任务,然后调用
external-data-schemas-cancel。external-data-schemas-reload
G. Table 数据已损坏,但希望保留 schema 配置
- 操作:使用 删除已同步的 table,但保留 schema 条目。下次 reload 会从头重新导入数据,同时不会丢失已配置的
external-data-schemas-delete-data/sync_type。incremental_field
H. 账单限制
- 这不属于技术问题。解释具体限制,并建议升级套餐,或禁用优先级较低的 schema,使重要 schema 的数据量控制在配额以内。
I. 由 Webhook 驱动的 schema 未收到事件
- 症状:schema 的 为
sync_type,初始批量同步已经完成,但即使 source 端有活动,仍没有新行到达。状态可能仍显示为webhook,因为批量同步(兜底定时同步)成功了——真正的问题是推送链路。Completed - 操作:
- 。
external-data-sources-webhook-info-retrieve({source_id}) - 如果 → webhook 从未注册或已被删除。调用
exists: false注册 webhook。create-webhook-create - 如果 ,但
exists: true已被设置 → 通常是external_status.error或类似错误。Webhook 可能已从 source 的 dashboard 中删除,需要重新创建。API key doesn't have permission to read webhooks - 如果 不是
external_status.status→ source 已禁用 webhook(通常发生在重复投递失败后)。重新启用或重新注册 webhook。enabled - 如果 payload 正在到达但签名验证失败 → signing secret 已被轮换。从 source 的 dashboard 获取新值,然后调用 。
update-webhook-inputs-create({source_id}, {inputs: {signing_secret: "..."}})
- 完成任何修复后,检查 source 端的 webhook 日志,确认 PostHog 现在能够返回 2xx。
Step 5 — Confirm before destructive actions
步骤 5——执行破坏性操作前确认
Three recovery actions discard data and cannot be undone:
- — wipes synced rows, re-imports from scratch
external-data-schemas-resync - — drops the synced table
external-data-schemas-delete-data - — deletes the source and all its schemas
external-data-sources-destroy
Always present the fix you're proposing and wait for explicit approval before calling any of these. "Just try
resync" is rarely the right default.
以下三种恢复操作会丢弃数据且无法撤销:
- ——清除已同步的行并从头重新导入
external-data-schemas-resync - ——删除已同步的 table
external-data-schemas-delete-data - ——删除 source 及其所有 schema
external-data-sources-destroy
始终应先说明准备执行的修复,并等待用户明确批准,然后再调用上述操作。“直接试试 resync”很少是正确的默认方案。
Example interaction
示例交互
text
User: "Our Stripe sync is broken, can you check?"
Agent:
- external-data-sources-list → find Stripe source, status = Error
- external-data-sources-retrieve({id}) → latest_error: "authentication failed: 401 Unauthorized"
- Report: "Your Stripe source's API key is no longer authenticating.
All 8 tables under it are failing with 401s. This usually means the key was rotated on the Stripe side.
To fix:
1. Grab a fresh restricted API key from the Stripe dashboard.
2. I'll update the source with the new key.
3. Syncs will resume automatically — no data loss.
Paste the new key here when ready."
User: "sk_live_..."
Agent:
- external-data-sources-partial-update({id}, {job_inputs: {stripe_secret_key: "sk_live_..."}})
- external-data-sources-reload({id}) to trigger retry
- Report: "Updated and re-triggered. Check back in a few minutes — latest_error should clear."text
User: "Our Stripe sync is broken, can you check?"
Agent:
- external-data-sources-list → find Stripe source, status = Error
- external-data-sources-retrieve({id}) → latest_error: "authentication failed: 401 Unauthorized"
- Report: "Your Stripe source's API key is no longer authenticating.
All 8 tables under it are failing with 401s. This usually means the key was rotated on the Stripe side.
To fix:
1. Grab a fresh restricted API key from the Stripe dashboard.
2. I'll update the source with the new key.
3. Syncs will resume automatically — no data loss.
Paste the new key here when ready."
User: "sk_live_..."
Agent:
- external-data-sources-partial-update({id}, {job_inputs: {stripe_secret_key: "sk_live_..."}})
- external-data-sources-reload({id}) to trigger retry
- Report: "Updated and re-triggered. Check back in a few minutes — latest_error should clear."Important notes
重要说明
- Source status overrides schema status for diagnosis. If the source is , nothing under it will work; fixing the source usually fixes all its schemas at once.
Error - isn't always healthy. Cross-check
Running. A sync stuck inlast_synced_atneedsRunningthencancel, notreload.resync - Resync is destructive. It discards synced data. Only recommend it when the data itself is bad (duplicates, missing rows, corrupt) or when recovery genuinely requires a clean slate (lost WAL position on CDC). Never use it as a first-try for transient errors.
- Delete-data preserves config. When a user says "I just want to start this table over from scratch", prefer
+
delete-dataoverreload+ new schema entry — it keeps the configured sync_type / incremental_field / PK setup.resync - A failure can belong to a destination, not the source. Some projects sync a source to their own database as
well as to PostHog. When they do, is prefixed with the destination's name, e.g.
latest_error. That is the customer's database refusing the write, not the source failing to extract — check their database and its credentials rather than the source's. Every destination shares one lifecycle, so one unreachable destination holds up the whole sync, PostHog included.customer postgres: connection refused - Billing limits aren't technical failures. Don't try to retry or reconfigure your way out. Route to billing.
- Webhook failures can hide behind a green status. A webhook-type schema whose bulk fallback sync succeeded looks
even when the push channel is broken. When users say "my data is hours behind" on a webhook schema, call
Completedbefore looking at schema status. Webhook issues don't surface onwebhook-info-retrieve.external-data-schemas-list
- 诊断时,source 状态优先于 schema 状态。 如果 source 为 ,其下的任何内容都无法正常工作;修复 source 通常可以一次解决所有 schema 的问题。
Error - 不一定代表健康。 应交叉检查
Running。卡在last_synced_at状态的同步需要先Running再cancel,而不是reload。resync - Resync 具有破坏性。 仅当数据本身存在问题(重复、缺行、损坏),或恢复过程确实需要从干净状态开始(CDC 的 WAL position 丢失)时,才建议使用。绝不能将 transient error 的首选方案设为 resync。
- Delete-data 会保留配置。 当用户表示“只想让这张 table 从头开始”时,应优先使用 +
delete-data,而不是reload+ 新建 schema 条目——这样可以保留已配置的resync/sync_type/ PK 设置。incremental_field - 故障可能属于 destination,而不是 source。 某些项目除了同步到 PostHog,还会将 source 同步到自己的数据库。在这种情况下,会带 destination 名称前缀,例如
latest_error。这表示客户的数据库拒绝写入,而不是 source 无法提取数据;应检查其数据库和凭证,而不是 source。每个 destination 都共享同一个 lifecycle,因此只要有一个 destination 不可达,整个同步(包括 PostHog)就会停滞。customer postgres: connection refused - 账单限制不属于技术故障。 不要尝试通过重试或重新配置来绕过限制,应转交账单团队处理。
- Webhook 故障可能隐藏在绿色状态之下。 当 webhook 类型的 schema 批量兜底同步成功时,它会显示为 ,即使推送链路已经损坏。当用户表示 webhook schema 的“数据已经落后数小时”时,应先调用
Completed,再查看 schema 状态。Webhook 问题不会出现在webhook-info-retrieve中。external-data-schemas-list