schema-e2e-validation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Schema E2E Validation

Schema E2E 验证

When to Use This Skill

何时使用该Skill

Use this skill when:

Validating schema changes before commit
Verifying YAML schema matches live ClickHouse Cloud
Regenerating Python types, DDL, or docs
Running full schema workflow validation

在以下场景中使用该Skill：

提交前验证schema变更
验证YAML schema与ClickHouse Cloud实际环境是否匹配
重新生成Python类型、DDL或文档
运行完整的schema工作流验证

Prerequisites

前置条件

Docker Runtime (Required)

Docker 运行时（必填）

Earthly requires Docker. Start Colima before running:

bash

colima start

Check if running:

bash

docker ps  # Should not error

Earthly 依赖 Docker 运行环境。运行前请启动 Colima：

bash

colima start

检查是否正常运行：

bash

docker ps  # 执行后不应报错

Doppler Access (For validation targets)

Doppler 访问权限（用于验证目标）

Required for

+test-schema-validate

and

+test-schema-e2e

bash

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

+test-schema-validate

和

+test-schema-e2e

目标需要以下配置：

bash

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

Earthly 安装

bash

brew install earthly

bash

brew install earthly

Quick Commands

快速命令

Generation only (no secrets)

仅执行生成操作（无需密钥）

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # 若未启动则执行此命令
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

带验证的完整E2E流程（需Doppler）

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # 若未启动则执行此命令
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

所有非密钥依赖目标

bash

cd /Users/terryli/eon/gapless-network-data
earthly +all

bash

cd /Users/terryli/eon/gapless-network-data
earthly +all

Artifacts

生成产物

After running

+test-schema-generate

+test-schema-e2e

, check

./earthly-artifacts/

Path	Contents
`types/blocks.py`	Pydantic + TypedDict models
`types/__init__.py`	Package init
`ddl/ethereum_mainnet.sql`	ClickHouse DDL
`docs/ethereum_mainnet.md`	Markdown documentation

For E2E, artifacts are under

e2e/types/

e2e/ddl/

e2e/docs/

执行

+test-schema-generate

或

+test-schema-e2e

后，可在

./earthly-artifacts/

目录下查看产物：

路径	内容
`types/blocks.py`	Pydantic + TypedDict 模型
`types/__init__.py`	包初始化文件
`ddl/ethereum_mainnet.sql`	ClickHouse DDL
`docs/ethereum_mainnet.md`	Markdown 文档

对于E2E流程，产物存储在

e2e/types/

、

e2e/ddl/

、

e2e/docs/

目录下。

Earthfile Targets Reference

Earthfile 目标参考

Target	Secrets	Purpose
`+deps`	No	Install uv + dependencies
`+build`	No	Copy source files
`+test-unit`	No	Run pytest
`+test-schema-generate`	No	Generate types/DDL/docs
`+test-schema-validate`	Yes	Validate vs ClickHouse
`+test-schema-e2e`	Yes	Full workflow + artifacts
`+all`	No	Run all non-secret targets

目标	是否需要密钥	用途
`+deps`	否	安装 uv 及相关依赖
`+build`	否	复制源文件
`+test-unit`	否	运行 pytest 测试
`+test-schema-generate`	否	生成类型文件/DDL/文档
`+test-schema-validate`	是	与ClickHouse进行schema一致性验证
`+test-schema-e2e`	是	执行完整工作流并生成产物
`+all`	否	运行所有无需密钥的目标

Troubleshooting

故障排查

"could not determine buildkit address - is Docker or Podman running?"

错误：无法确定buildkit地址 - Docker或Podman是否在运行？

Cause: Docker/Colima not running

Fix:

bash

colima start

原因：Docker/Colima 未启动

解决方法：

bash

colima start

Wait for "done" message, then retry

等待显示「done」信息后重试

earthly +test-schema-generate

undefined

earthly +test-schema-generate

undefined

"unable to parse --secret-file argument"

错误：无法解析 --secret-file 参数

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is

--secret-file-path

(NOT

--secret-file

). The wrapper script handles this, but if running manually:

bash

undefined

原因：参数名称错误或密钥文件格式不正确

解决方法：正确的参数是

--secret-file-path

（而非

--secret-file

）。包装脚本已处理此问题，若手动运行请使用：

bash

undefined

WRONG

错误写法

earthly --secret-file=/path/to/secrets +target

CORRECT

正确写法

earthly --secret-file-path=/path/to/secrets +target


Also ensure secrets file has no quotes around values:

```bash

earthly --secret-file-path=/path/to/secrets +target


同时确保密钥文件中的值不带引号：

```bash

WRONG format

错误格式

CLICKHOUSE_HOST="host.cloud"

CORRECT format

正确格式

CLICKHOUSE_HOST=host.cloud

undefined

CLICKHOUSE_HOST=host.cloud

undefined

"OSError: Readme file does not exist: README.md"

错误：OSError: 不存在README.md文件

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

earthfile

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

原因：hatchling 构建后端要求容器中存在README.md文件

解决方法：确保Earthfile的deps目标中复制了README.md：

earthfile

deps:
    COPY pyproject.toml uv.lock README.md ./  # 必须包含README.md!

"missing secret" during validation

验证时出现「缺少密钥」错误

Cause: Doppler not configured or secrets not passed

Fix:

bash

undefined

原因：Doppler 未配置或密钥未正确传递

解决方法：

bash

undefined

Verify Doppler has the secrets

验证Doppler是否包含所需密钥

doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

Use the wrapper script (handles secret injection)

使用包装脚本（已处理密钥注入）

./scripts/earthly-with-doppler.sh +test-schema-validate

undefined

./scripts/earthly-with-doppler.sh +test-schema-validate

undefined

Cache Issues

缓存问题

Force rebuild without cache:

bash

earthly --no-cache +test-schema-e2e

强制不使用缓存重新构建：

bash

earthly --no-cache +test-schema-e2e

Implementation Details

实现细节

Doppler Secret Injection

Doppler 密钥注入

The wrapper script

scripts/earthly-with-doppler.sh

Downloads secrets from Doppler
Filters for
```
CLICKHOUSE_*
```
variables
Strips quotes (Doppler outputs
```
KEY="value"
```
, Earthly needs
```
KEY=value
```
)
Passes via
```
--secret-file-path
```
flag
Cleans up temp file on exit

包装脚本

scripts/earthly-with-doppler.sh

的执行流程：

从Doppler下载密钥
筛选出
```
CLICKHOUSE_*
```
开头的变量
去除引号（Doppler输出格式为
```
KEY="value"
```
，Earthly需要
```
KEY=value
```
格式）
通过
```
--secret-file-path
```
参数传递密钥
执行完成后清理临时文件

Secrets Required

所需密钥

Secret	Purpose
`CLICKHOUSE_HOST_READONLY`	ClickHouse Cloud host
`CLICKHOUSE_USER_READONLY`	Read-only user
`CLICKHOUSE_PASSWORD_READONLY`	Read-only password

密钥名称	用途
`CLICKHOUSE_HOST_READONLY`	ClickHouse Cloud 主机地址
`CLICKHOUSE_USER_READONLY`	只读用户账号
`CLICKHOUSE_PASSWORD_READONLY`	只读用户密码

Related Files

File	Purpose
`/Users/terryli/eon/gapless-network-data/Earthfile`	Main build file
`/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh`	Secret injection wrapper
`/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml`	SSoT schema
`/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md`	ADR

文件路径	用途
`/Users/terryli/eon/gapless-network-data/Earthfile`	主构建文件
`/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh`	密钥注入包装脚本
`/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml`	单一可信源（SSoT）schema文件
`/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md`	架构决策记录（ADR）

Validation History

验证历史

2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
Lessons Learned:
- ```
--secret-file-path
```
  not
```
--secret-file
```
  (Earthly v0.8.16)
- Doppler
```
--format env
```
  outputs quotes, must strip with
```
sed 's/"//g'
```
- README.md must be copied for hatchling build backend
- Colima must be started before Earthly runs

2025-12-03：创建完成并通过针对ClickHouse Cloud的完整E2E验证
经验总结：
- Earthly v0.8.16版本需使用
```
--secret-file-path
```
  而非
```
--secret-file
```
- Doppler的
```
--format env
```
  输出带引号，需用
```
sed 's/"//g'
```
  去除
- hatchling构建后端要求必须复制README.md文件
- 运行Earthly前必须启动Colima

Design Authority

设计依据

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect
first.

该Skill仅负责验证schema，不负责schema设计。如需schema设计指导（如ORDER BY、压缩、分区策略），请先调用**

quality-tools:clickhouse-architect

**。

Skill	Purpose
`quality-tools:clickhouse-architect`	Schema design before validation
`devops-tools:clickhouse-cloud-management`	Cloud credentials for E2E tests
`devops-tools:clickhouse-pydantic-config`	Client configuration

Skill名称	用途
`quality-tools:clickhouse-architect`	验证前的Schema设计指导
`devops-tools:clickhouse-cloud-management`	E2E测试所需的云凭证管理
`devops-tools:clickhouse-pydantic-config`	客户端配置管理

schema-e2e-validation

Original

Translation

Schema E2E Validation

Schema E2E 验证

When to Use This Skill

何时使用该Skill

Prerequisites

前置条件

Docker Runtime (Required)

Docker 运行时（必填）

Doppler Access (For validation targets)

Doppler 访问权限（用于验证目标）

Earthly Installation

Earthly 安装

Quick Commands

快速命令

Generation only (no secrets)

仅执行生成操作（无需密钥）

Full E2E with validation (requires Doppler)

带验证的完整E2E流程（需Doppler）

All non-secret targets

所有非密钥依赖目标

Artifacts

生成产物

Earthfile Targets Reference

Earthfile 目标参考

Troubleshooting

故障排查

"could not determine buildkit address - is Docker or Podman running?"

错误：无法确定buildkit地址 - Docker或Podman是否在运行？

Wait for "done" message, then retry

等待显示「done」信息后重试

"unable to parse --secret-file argument"

错误：无法解析 --secret-file 参数

WRONG

错误写法

CORRECT

正确写法

WRONG format

错误格式

CORRECT format

正确格式

"OSError: Readme file does not exist: README.md"

错误：OSError: 不存在README.md文件

"missing secret" during validation

验证时出现「缺少密钥」错误

Verify Doppler has the secrets

验证Doppler是否包含所需密钥

Use the wrapper script (handles secret injection)

使用包装脚本（已处理密钥注入）

Cache Issues

缓存问题

Implementation Details

实现细节

Doppler Secret Injection

Doppler 密钥注入

Secrets Required

所需密钥

Related Files

相关文件

Validation History

验证历史

Design Authority

设计依据

Related Skills

相关Skill