Back to Details

alibabacloud-cksync-plan

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

ClickHouse Sync Plan (cksync-plan)

ClickHouse 同步计划（cksync-plan）

A skill for planning ClickHouse cluster data migration solutions, including migration plans, risks, and considerations.
一款用于规划ClickHouse集群数据迁移方案的技能，涵盖迁移计划、风险和注意事项。

When to Use

适用场景

  • Data migration between different ClickHouse clusters
  • Horizontal scaling (adding/removing nodes) for ClickHouse clusters
  • Disk downgrade operations
  • Cross-availability zone migrations
  • Upgrading to multi-replica, multi-AZ deployments
  • 不同ClickHouse集群之间的数据迁移
  • ClickHouse集群水平扩缩容（新增/移除节点）
  • 磁盘降级操作
  • 跨可用区迁移
  • 升级至多副本多可用区部署

Workflow

工作流程

Step 1: Gather Source Cluster Information

步骤1：收集源集群信息

Ask user for source cluster type:
  • Self-built ClickHouse or non-Alibaba Cloud ClickHouse
  • Alibaba Cloud ClickHouse Community Edition
  • Alibaba Cloud ClickHouse Enterprise Edition
Ask user for source cluster version (e.g., 20.8, 22.8, 23.8, 24.3):
  • Version affects migration method compatibility
  • BACKUP/RESTORE requires ≥22.8
  • Incremental cksync migration requires target ≥20.8
询问用户源集群类型：
  • 自建ClickHouse或非阿里云ClickHouse
  • 阿里云ClickHouse社区版
  • 阿里云ClickHouse企业版
询问用户源集群版本（例如20.8、22.8、23.8、24.3）：
  • 版本会影响迁移方法的兼容性
  • BACKUP/RESTORE功能要求版本≥22.8
  • 增量cksync迁移要求目标端版本≥20.8

Step 2: Gather Target Cluster Information

步骤2：收集目标集群信息

Ask user for target cluster type:
  • Alibaba Cloud ClickHouse Community Edition
  • Alibaba Cloud ClickHouse Enterprise Edition
  • To be determined
询问用户目标集群类型：
  • 阿里云ClickHouse社区版
  • 阿里云ClickHouse企业版
  • 待定

Step 3: Collect Cluster Details (REQUIRED)

步骤3：收集集群详情（必填）

This step is mandatory. You MUST collect database and table information before proceeding to migration plan selection.
本步骤为强制性要求，在选择迁移方案前，你必须收集数据库和表的相关信息。

Required Information

所需信息

  1. Database list with engines
  2. Table list with engines, partition counts, data sizes, and write speeds
  1. 数据库列表及对应引擎
  2. 表列表及对应引擎、分区数量、数据大小、写入速度

Option A: User Executes SQL

选项A：用户执行SQL

Provide SQL queries from references/sql.md section 1 for user to execute:
  1. Database Information - Query 
    system.databases
    for database names and engines
  2. Table Information - Comprehensive query including table names, engines, engine_full (for TTL), partition counts, data sizes, and write speeds
Key fields to collect:
  • engine_full
    : Contains TTL clause (e.g., 
    TTL event_time + INTERVAL 7 DAY
    )
  • part_count
    : Partition count per table
  • data_bytes
    : Data size per shard
  • write_speed_bytes_per_sec
    : Write speed calculated from part_log
For complete SQL queries, see references/sql.md section 1.
提供references/sql.md第1节中的SQL查询语句供用户执行：
  1. 数据库信息 - 查询
    system.databases
    获取数据库名称和引擎
  2. 表信息 - 综合查询，包含表名、引擎、完整引擎配置（用于TTL）、分区数量、数据大小和写入速度
需要收集的关键字段：
  • engine_full
    ：包含TTL语句（例如
    TTL event_time + INTERVAL 7 DAY
    ）
  • part_count
    ：单表分区数
  • data_bytes
    ：每个分片的数据大小
  • write_speed_bytes_per_sec
    ：通过part_log计算得到的写入速度
完整SQL查询语句见references/sql.md第1节。

Option B: Direct Query via HTTP

选项B：通过HTTP直接查询

Request connection details from user:
  • HOST_NAME
    : Cluster endpoint (e.g., 
    cc-xxx.clickhouse.rds.aliyuncs.com
    )
  • HTTP_PORT
    : HTTP port (default: 
    8123
    )
  • USER_NAME
    : Database username
  • PASSWORD
    : Database password
Use secure credential handling and HTTP query examples from references/sql.md section 5.
向用户索要连接信息：
  • HOST_NAME
    ：集群端点（例如
    cc-xxx.clickhouse.rds.aliyuncs.com
    ）
  • HTTP_PORT
    ：HTTP端口（默认：
    8123
    ）
  • USER_NAME
    ：数据库用户名
  • PASSWORD
    ：数据库密码
请使用references/sql.md第5节中的安全凭证处理方式和HTTP查询示例。

Analysis Checklist

分析检查清单

After collecting data, verify:
  • Required metadata is complete (database engine, table engine, 
    engine_full
    , partitions, data size, write speed)
  • Migration compatibility checks are completed using references/plans.md (method-specific conditions)
  • Version and read-only window constraints are mapped to candidate methods
  • Risks and mitigations are identified and recorded in the plan
收集数据后，验证以下内容：
  • 所需元数据完整（数据库引擎、表引擎、
    engine_full
    、分区、数据大小、写入速度）
  • 已参照references/plans.md完成迁移兼容性检查（各方法的特定条件）
  • 版本和只读窗口约束已匹配到候选迁移方法
  • 风险和缓解措施已识别并记录在方案中

Step 4: Business Requirements

步骤4：收集业务需求

Ask for allowed read-only time:
  • 0 minutes
  • Within 30 minutes
  • Within 1 day
  • Not sure yet
询问允许的只读时长：
  • 0分钟
  • 30分钟以内
  • 1天以内
  • 暂不确定

Step 5: Select and Present Migration Plan

步骤5：选择并输出迁移方案

Based on gathered information, analyze and recommend from these migration methods:
MethodBest ForMin Read-Only Time
Console (cksync)Most migrations to Alibaba Cloud~10 min
BACKUP/RESTORELarge data, same edition type, version ≥22.8Varies by data size
INSERT FROM REMOTEFlexible control, small-medium data~10 min per batch
Business Double-WriteZero downtime required0
Kafka Double-WriteExisting Kafka pipelines or business writes switched to Kafka0
Big Cluster FederationLarge scale, complex scenarios0
Hard requirement: MUST output a plan, never output empty content.
Even when information is incomplete, you MUST output a provisional migration plan. The provisional plan must include:
  • assumptions used,
  • missing-information checklist,
  • confidence level and key uncertainties,
  • next steps to finalize recommendation after user provides missing inputs.
基于收集到的信息，从以下迁移方法中分析并推荐合适的方案：
方法适用场景最小只读时长
控制台（cksync）大多数迁移到阿里云的场景~10分钟
BACKUP/RESTORE大数据量、同版本类型、版本≥22.8随数据大小变化
INSERT FROM REMOTE需要灵活控制、中小数据量每批次~10分钟
业务双写要求零停机的场景0
Kafka双写现有Kafka管道或业务写入可切换到Kafka的场景0
大集群联邦大规模复杂场景0
硬性要求：必须输出方案，绝对不能输出空内容。
即便信息不完整，你也必须输出临时迁移方案。临时方案必须包含：
  • 所用假设
  • 缺失信息清单
  • 置信度和核心不确定点
  • 用户提供缺失信息后确定最终推荐方案的后续步骤

Migration Methods Overview

迁移方法概览

1. Console (cksync) Migration

1. 控制台（cksync）迁移

Default choice for most Alibaba Cloud migration scenarios, especially in-place operations. For support boundaries, engine constraints, TTL/write-speed checks, merge risk, and resource prerequisites, see references/plans.md section 1.
大多数阿里云迁移场景的默认选择，尤其是原地操作场景。支持边界、引擎约束、TTL/写入速度检查、合并风险和资源前提条件见references/plans.md第1节。

2. BACKUP/RESTORE Migration

2. BACKUP/RESTORE迁移

Suitable for same-edition migrations where full backup/restore workflow is acceptable. For version/edition constraints, supported engines, command patterns, and progress monitoring, see references/plans.md section 2.
适用于可接受全量备份/恢复流程的同版本迁移。版本/版本类型约束、支持的引擎、命令模式和进度监控见references/plans.md第2节。

3. INSERT FROM REMOTE Migration

3. INSERT FROM REMOTE迁移

Best when fine-grained table/partition/time-range control is needed. For applicability boundaries and operational constraints, see references/plans.md section 3. For SQL templates and detailed steps, see references/sql.md section 2.
最适合需要细粒度控制表/分区/时间范围的场景。适用边界和操作约束见references/plans.md第3节。SQL模板和详细步骤见references/sql.md第2节。

4. Business Double-Write

4. 业务双写

Use when zero downtime is required and application-side dual-write is feasible. For detailed conditions, see references/plans.md section 4.
当要求零停机且应用侧可实现双写时使用。详细条件见references/plans.md第4节。

5. Kafka Double-Write

5. Kafka双写

Use when dual-consumer switchover via Kafka is feasible, including both existing Kafka pipelines and cases where business writes can be switched to Kafka first. For detailed conditions, see references/plans.md section 5.
当可通过Kafka实现双消费者切换时使用，包括现有Kafka管道和业务写入可先切换到Kafka的场景。详细条件见references/plans.md第5节。

6. Big Cluster Federation

6. 大集群联邦

Advanced option for large/complex migrations with strong business and technical collaboration.
  • Community + Enterprise: See references/big-cluster-community-enterprise.md
  • Self-built + Cloud: See references/big-cluster-self-built-community.md
适用于需要业务和技术深度协作的大规模/复杂迁移的高级选项。
  • 社区版+企业版：见references/big-cluster-community-enterprise.md
  • 自建+云：见references/big-cluster-self-built-community.md

Output Format

输出格式

Default deliverable: Produce one migration plan only. Structure it using assets/migration-plan-template.md and include the key sections below (cluster facts and commands may appear inline in the plan; that counts as the single deliverable).
Additional files only on request: Do not create separate files for cluster-information documentation, scripts, or SQL unless the customer explicitly asks for them. When they do, use assets/cluster-info-template.md for cluster documentation and place scripts/SQL in clearly named files as requested.
Key sections in the migration plan:
  1. Executive Summary - Method, data size, duration, downtime
  2. Source Cluster Analysis - Databases, tables, compatibility check
  3. Migration Method Selection - Rationale and alternatives
  4. Migration Steps - Pre/execution/post with commands
  5. Risks & Mitigations - With probability and impact
  6. Rollback Plan - Trigger conditions and steps
  7. Timeline - Phase schedule with owners
  8. Reference Links - Documentation URLs
默认交付物：仅生成一份迁移方案。使用assets/migration-plan-template.md构建结构，包含以下核心部分（集群事实和命令可以内嵌在方案中，也算作单份交付物）。
仅在请求时提供额外文件： 除非客户明确要求，否则不要为集群信息文档、脚本或SQL创建单独文件。如果客户要求，使用assets/cluster-info-template.md作为集群文档模板，按要求将脚本/SQL放在命名清晰的文件中。
迁移方案的核心部分：
  1. 执行摘要 - 迁移方法、数据大小、耗时、停机时间
  2. 源集群分析 - 数据库、表、兼容性检查
  3. 迁移方法选择 - 选型理由和备选方案
  4. 迁移步骤 - 前置/执行/后置步骤及对应命令
  5. 风险与缓解措施 - 包含发生概率和影响程度
  6. 回滚方案 - 触发条件和步骤
  7. 时间线 - 各阶段安排及负责人
  8. 参考链接 - 文档URL

Method Selection Reference

方法选择参考

For quick scenario-to-method mapping and method-specific constraints (including in-place migration priority and Enterprise → Enterprise options), see references/plans.md section "Method Selection Priority" and related method sections.
如需快速的场景到方法映射以及各方法的特定约束（包括原地迁移优先级和企业版→企业版选项），见references/plans.md的“方法选择优先级”章节和相关方法章节。

Additional Resources

额外资源

  • references/plans.md - Detailed migration plan conditions
  • references/sql.md - SQL templates and commands
  • references/stop-merge-storm.md - How to stop post-sync merge storm
  • references/big-cluster-community-enterprise.md - Community + Enterprise federation
  • references/big-cluster-self-built-community.md - Self-built + Cloud federation
  • references/plans.md - 详细迁移方案条件
  • references/sql.md - SQL模板和命令
  • references/stop-merge-storm.md - 如何停止同步后的合并风暴
  • references/big-cluster-community-enterprise.md - 社区版+企业版联邦
  • references/big-cluster-self-built-community.md - 自建+云联邦