Back to Details

data-sync-workflows

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Data Sync Workflows

数据同步工作流

Use this skill when data moves between systems and the application must preserve correctness across retries, partial failure, and repeated runs.
当数据在系统间传输，且应用需在重试、部分失败及重复运行场景下保持数据正确性时，可使用本skill。

Design Questions

设计问题

  • What is the source of truth for each field?
  • Is the sync full, incremental, event-driven, scheduled, or manual?
  • What is the stable identity key across systems?
  • What checkpoint or cursor proves where the last successful run stopped?
  • Can the operation be retried safely?
  • What does a partial success mean?
  • How will operators see what changed, failed, skipped, or needs repair?
  • 每个字段的source of truth是什么？
  • 同步方式是全量、增量、事件驱动、定时还是手动？
  • 跨系统的稳定标识键（stable identity key）是什么？
  • 有哪些checkpoint或游标可以证明上一次成功运行的终止位置？
  • 操作能否安全重试？
  • 部分成功意味着什么？
  • 运维人员如何查看哪些数据已变更、失败、跳过或需要修复？

Core Design Rules

核心设计规则

  • Define identity and idempotency before implementation.
  • Store checkpoints only after durable success for the covered range.
  • Separate fetch, normalize, validate, apply, and evidence steps.
  • Make every write idempotent or protected by a deterministic conflict rule.
  • Keep raw external data out of domain code after normalization.
  • Design backfill and replay paths before production use.
  • Treat deletion, missing records, and tombstones as explicit product decisions.
  • 在实施前定义identity和idempotency规则。
  • 仅在覆盖范围内的操作持久化成功后，才存储checkpoint。
  • 将数据获取（fetch）、标准化（normalize）、验证（validate）、应用（apply）及证据留存（evidence）步骤分离。
  • 确保每一次写入操作都是idempotent的，或受确定性冲突规则保护。
  • 标准化完成后，避免在领域代码中保留原始外部数据。
  • 在投入生产使用前，设计回填（backfill）和重放（replay）路径。
  • 将删除操作、缺失记录及墓碑记录（tombstones）视为明确的产品决策。

Workflow Shape

工作流结构

text
load checkpoint
fetch external page or range
normalize records
validate records
deduplicate by stable identity
apply idempotent writes
record per-record evidence
advance checkpoint
emit run summary
text
load checkpoint
fetch external page or range
normalize records
validate records
deduplicate by stable identity
apply idempotent writes
record per-record evidence
advance checkpoint
emit run summary

Checkpoint Rules

Checkpoint规则

  • Use monotonic cursors, updated-at windows, sequence IDs, or source-provided tokens when available.
  • Include enough metadata to resume without rereading ambiguous ranges.
  • For time-window syncs, use overlap windows and deduplicate by stable identity.
  • Never advance the checkpoint past unapplied records.
  • 优先使用单调游标（monotonic cursors）、更新时间窗口（updated-at windows）、序列ID（sequence IDs）或源系统提供的令牌（source-provided tokens）。
  • 包含足够的元数据，以便在恢复运行时无需重新读取模糊范围的数据。
  • 对于时间窗口同步，使用重叠窗口并通过stable identity去重。
  • 绝不能将checkpoint推进到未处理完成的记录之后。

Failure Handling

故障处理

  • Classify failures as retryable, permanent, data-quality, auth/config, or human decision required.
  • Keep per-record failures visible instead of collapsing the whole run into one generic error.
  • Use bounded retries for transient infrastructure errors.
  • Stop on repeated deterministic failures and preserve evidence for repair.
  • 将故障分类为可重试、永久故障、数据质量问题、认证/配置问题或需人工决策的类型。
  • 保留每条记录的故障详情，而非将整个运行过程的错误合并为一个通用错误。
  • 对临时基础设施错误使用有限次数的重试。
  • 当出现重复的确定性故障时停止运行，并保留证据以便修复。

Evidence

证据留存

Each run should produce a summary with:
  • run ID
  • source system
  • checkpoint before and after
  • records fetched, created, updated, skipped, failed
  • retry count
  • failure categories
  • links or IDs for detailed diagnostics
每次运行应生成包含以下内容的汇总报告：
  • 运行ID（run ID）
  • 源系统
  • 运行前后的checkpoint
  • 获取、创建、更新、跳过、失败的记录数
  • 重试次数
  • 故障分类
  • 详细诊断信息的链接或ID

Review Checklist

评审检查清单

  • Identity key is stable and documented.
  • Sync writes are idempotent.
  • Checkpoint advancement is safe.
  • Backfill/replay behavior is defined.
  • Partial failures are visible and repairable.
  • Tests cover duplicate input, out-of-order input, missing fields, retry, and resume.
  • 标识键（identity key）稳定且已文档化。
  • 同步写入操作具有idempotency。
  • Checkpoint的推进是安全的。
  • 回填/重放行为已明确定义。
  • 部分故障可见且可修复。
  • 测试覆盖重复输入、乱序输入、缺失字段、重试及恢复场景。