Back to Details

schema-e2e-validation

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Schema E2E Validation

Schema E2E 验证

When to Use This Skill

何时使用该Skill

Use this skill when:
  • Validating schema changes before commit
  • Verifying YAML schema matches live ClickHouse Cloud
  • Regenerating Python types, DDL, or docs
  • Running full schema workflow validation
在以下场景中使用该Skill：
  • 提交前验证schema变更
  • 验证YAML schema与ClickHouse Cloud实际环境是否匹配
  • 重新生成Python类型、DDL或文档
  • 运行完整的schema工作流验证

Prerequisites

前置条件

Docker Runtime (Required)

Docker 运行时（必填）

Earthly requires Docker. Start Colima before running:
bash
colima start
Check if running:
bash
docker ps  # Should not error
Earthly 依赖 Docker 运行环境。运行前请启动 Colima：
bash
colima start
检查是否正常运行：
bash
docker ps  # 执行后不应报错

Doppler Access (For validation targets)

Doppler 访问权限（用于验证目标）

Required for 
+test-schema-validate
and 
+test-schema-e2e
:
bash
doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd
+test-schema-validate
和 
+test-schema-e2e
目标需要以下配置：
bash
doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

Earthly 安装

bash
brew install earthly
bash
brew install earthly

Quick Commands

快速命令

Generation only (no secrets)

仅执行生成操作（无需密钥）

bash
cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate
bash
cd /Users/terryli/eon/gapless-network-data
colima start  # 若未启动则执行此命令
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

带验证的完整E2E流程（需Doppler）

bash
cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e
bash
cd /Users/terryli/eon/gapless-network-data
colima start  # 若未启动则执行此命令
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

所有非密钥依赖目标

bash
cd /Users/terryli/eon/gapless-network-data
earthly +all
bash
cd /Users/terryli/eon/gapless-network-data
earthly +all

Artifacts

生成产物

After running 
+test-schema-generate
or 
+test-schema-e2e
, check 
./earthly-artifacts/
:
PathContents
types/blocks.py
Pydantic + TypedDict models
types/__init__.py
Package init
ddl/ethereum_mainnet.sql
ClickHouse DDL
docs/ethereum_mainnet.md
Markdown documentation
For E2E, artifacts are under 
e2e/types/
, 
e2e/ddl/
, 
e2e/docs/
.
执行 
+test-schema-generate
或 
+test-schema-e2e
后，可在 
./earthly-artifacts/
目录下查看产物：
路径内容
types/blocks.py
Pydantic + TypedDict 模型
types/__init__.py
包初始化文件
ddl/ethereum_mainnet.sql
ClickHouse DDL
docs/ethereum_mainnet.md
Markdown 文档
对于E2E流程，产物存储在 
e2e/types/
、
e2e/ddl/
、
e2e/docs/
目录下。

Earthfile Targets Reference

Earthfile 目标参考

TargetSecretsPurpose
+deps
NoInstall uv + dependencies
+build
NoCopy source files
+test-unit
NoRun pytest
+test-schema-generate
NoGenerate types/DDL/docs
+test-schema-validate
YesValidate vs ClickHouse
+test-schema-e2e
YesFull workflow + artifacts
+all
NoRun all non-secret targets
目标是否需要密钥用途
+deps
否安装 uv 及相关依赖
+build
否复制源文件
+test-unit
否运行 pytest 测试
+test-schema-generate
否生成类型文件/DDL/文档
+test-schema-validate
是与ClickHouse进行schema一致性验证
+test-schema-e2e
是执行完整工作流并生成产物
+all
否运行所有无需密钥的目标

Troubleshooting

故障排查

"could not determine buildkit address - is Docker or Podman running?"

错误：无法确定buildkit地址 - Docker或Podman是否在运行？

Cause: Docker/Colima not running
Fix:
bash
colima start
原因：Docker/Colima 未启动
解决方法：
bash
colima start

Wait for "done" message, then retry

等待显示「done」信息后重试

earthly +test-schema-generate
undefined
earthly +test-schema-generate
undefined

"unable to parse --secret-file argument"

错误：无法解析 --secret-file 参数

Cause: Wrong flag name or malformed secrets file
Fix: The correct flag is 
--secret-file-path
(NOT 
--secret-file
). The wrapper script handles this, but if running manually:
bash
undefined
原因：参数名称错误或密钥文件格式不正确
解决方法：正确的参数是 
--secret-file-path
（而非 
--secret-file
）。包装脚本已处理此问题，若手动运行请使用：
bash
undefined

WRONG

错误写法

earthly --secret-file=/path/to/secrets +target
earthly --secret-file=/path/to/secrets +target

CORRECT

正确写法

earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

```bash
earthly --secret-file-path=/path/to/secrets +target

同时确保密钥文件中的值不带引号：

```bash

WRONG format

错误格式

CLICKHOUSE_HOST="host.cloud"
CLICKHOUSE_HOST="host.cloud"

CORRECT format

正确格式

CLICKHOUSE_HOST=host.cloud
undefined
CLICKHOUSE_HOST=host.cloud
undefined

"OSError: Readme file does not exist: README.md"

错误：OSError: 不存在README.md文件

Cause: hatchling build backend requires README.md in container
Fix: Ensure Earthfile copies README.md in deps target:
earthfile
deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!
原因：hatchling 构建后端要求容器中存在README.md文件
解决方法：确保Earthfile的deps目标中复制了README.md：
earthfile
deps:
    COPY pyproject.toml uv.lock README.md ./  # 必须包含README.md!

"missing secret" during validation

验证时出现「缺少密钥」错误

Cause: Doppler not configured or secrets not passed
Fix:
bash
undefined
原因：Doppler 未配置或密钥未正确传递
解决方法：
bash
undefined

Verify Doppler has the secrets

验证Doppler是否包含所需密钥

doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

Use the wrapper script (handles secret injection)

使用包装脚本（已处理密钥注入）

./scripts/earthly-with-doppler.sh +test-schema-validate
undefined
./scripts/earthly-with-doppler.sh +test-schema-validate
undefined

Cache Issues

缓存问题

Force rebuild without cache:
bash
earthly --no-cache +test-schema-e2e
强制不使用缓存重新构建：
bash
earthly --no-cache +test-schema-e2e

Implementation Details

实现细节

Doppler Secret Injection

Doppler 密钥注入

The wrapper script 
scripts/earthly-with-doppler.sh
:
  1. Downloads secrets from Doppler
  2. Filters for 
    CLICKHOUSE_*
    variables
  3. Strips quotes (Doppler outputs 
    KEY="value"
    , Earthly needs 
    KEY=value
    )
  4. Passes via 
    --secret-file-path
    flag
  5. Cleans up temp file on exit
包装脚本 
scripts/earthly-with-doppler.sh
的执行流程：
  1. 从Doppler下载密钥
  2. 筛选出
    CLICKHOUSE_*
    开头的变量
  3. 去除引号（Doppler输出格式为
    KEY="value"
    ，Earthly需要
    KEY=value
    格式）
  4. 通过
    --secret-file-path
    参数传递密钥
  5. 执行完成后清理临时文件

Secrets Required

所需密钥

SecretPurpose
CLICKHOUSE_HOST_READONLY
ClickHouse Cloud host
CLICKHOUSE_USER_READONLY
Read-only user
CLICKHOUSE_PASSWORD_READONLY
Read-only password
密钥名称用途
CLICKHOUSE_HOST_READONLY
ClickHouse Cloud 主机地址
CLICKHOUSE_USER_READONLY
只读用户账号
CLICKHOUSE_PASSWORD_READONLY
只读用户密码

Related Files

相关文件

FilePurpose
/Users/terryli/eon/gapless-network-data/Earthfile
Main build file
/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh
Secret injection wrapper
/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml
SSoT schema
/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md
ADR
文件路径用途
/Users/terryli/eon/gapless-network-data/Earthfile
主构建文件
/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh
密钥注入包装脚本
/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml
单一可信源（SSoT）schema文件
/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md
架构决策记录（ADR）

Validation History

验证历史

  • 2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
  • Lessons Learned: 
    • --secret-file-path
      not 
      --secret-file
      (Earthly v0.8.16)
    • Doppler 
      --format env
      outputs quotes, must strip with 
      sed 's/"//g'
    • README.md must be copied for hatchling build backend
    • Colima must be started before Earthly runs
  • 2025-12-03：创建完成并通过针对ClickHouse Cloud的完整E2E验证
  • 经验总结：
    • Earthly v0.8.16版本需使用
      --secret-file-path
      而非
      --secret-file
    • Doppler的
      --format env
      输出带引号，需用
      sed 's/"//g'
      去除
    • hatchling构建后端要求必须复制README.md文件
    • 运行Earthly前必须启动Colima

Design Authority

设计依据

<!-- ADR: 2025-12-10-clickhouse-skill-delegation -->
This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke 
quality-tools:clickhouse-architect
 first.
<!-- ADR: 2025-12-10-clickhouse-skill-delegation -->
该Skill仅负责验证schema，不负责schema设计。如需schema设计指导（如ORDER BY、压缩、分区策略），请先调用**
quality-tools:clickhouse-architect
**。

Related Skills

相关Skill

SkillPurpose
quality-tools:clickhouse-architect
Schema design before validation
devops-tools:clickhouse-cloud-management
Cloud credentials for E2E tests
devops-tools:clickhouse-pydantic-config
Client configuration
Skill名称用途
quality-tools:clickhouse-architect
验证前的Schema设计指导
devops-tools:clickhouse-cloud-management
E2E测试所需的云凭证管理
devops-tools:clickhouse-pydantic-config
客户端配置管理