schema-e2e-validation

Original：🇺🇸 English

Translated

Earthly E2E validation for YAML schema contracts. TRIGGERS - schema validation, YAML schema, schema contracts, regenerate types.

1installs

Sourceterrylica/cc-skills

Added on2026-02-07

NPX Install

npx skill4agent add terrylica/cc-skills schema-e2e-validation

SKILL.md Content

View Translation Comparison →

Schema E2E Validation

When to Use This Skill

Use this skill when:

Validating schema changes before commit
Verifying YAML schema matches live ClickHouse Cloud
Regenerating Python types, DDL, or docs
Running full schema workflow validation

Prerequisites

Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

bash

colima start

Check if running:

bash

docker ps  # Should not error

Doppler Access (For validation targets)

Required for

+test-schema-validate

and

+test-schema-e2e

bash

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

bash

brew install earthly

Quick Commands

Generation only (no secrets)

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

bash

cd /Users/terryli/eon/gapless-network-data
earthly +all

Artifacts

After running

+test-schema-generate

+test-schema-e2e

, check

./earthly-artifacts/

Path	Contents
`types/blocks.py`	Pydantic + TypedDict models
`types/__init__.py`	Package init
`ddl/ethereum_mainnet.sql`	ClickHouse DDL
`docs/ethereum_mainnet.md`	Markdown documentation

For E2E, artifacts are under

e2e/types/

e2e/ddl/

e2e/docs/

Earthfile Targets Reference

Target	Secrets	Purpose
`+deps`	No	Install uv + dependencies
`+build`	No	Copy source files
`+test-unit`	No	Run pytest
`+test-schema-generate`	No	Generate types/DDL/docs
`+test-schema-validate`	Yes	Validate vs ClickHouse
`+test-schema-e2e`	Yes	Full workflow + artifacts
`+all`	No	Run all non-secret targets

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

Cause: Docker/Colima not running

Fix:

bash

colima start
# Wait for "done" message, then retry
earthly +test-schema-generate

"unable to parse --secret-file argument"

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is

--secret-file-path

(NOT

--secret-file

). The wrapper script handles this, but if running manually:

bash

# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

bash

# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud

"OSError: Readme file does not exist: README.md"

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

earthfile

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

"missing secret" during validation

Cause: Doppler not configured or secrets not passed

Fix:

bash

# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate

Cache Issues

Force rebuild without cache:

bash

earthly --no-cache +test-schema-e2e

Implementation Details

Doppler Secret Injection

The wrapper script

scripts/earthly-with-doppler.sh

Downloads secrets from Doppler
Filters for
```
CLICKHOUSE_*
```
variables
Strips quotes (Doppler outputs
```
KEY="value"
```
, Earthly needs
```
KEY=value
```
)
Passes via
```
--secret-file-path
```
flag
Cleans up temp file on exit

Secrets Required

Secret	Purpose
`CLICKHOUSE_HOST_READONLY`	ClickHouse Cloud host
`CLICKHOUSE_USER_READONLY`	Read-only user
`CLICKHOUSE_PASSWORD_READONLY`	Read-only password

Related Files

File	Purpose
`/Users/terryli/eon/gapless-network-data/Earthfile`	Main build file
`/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh`	Secret injection wrapper
`/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml`	SSoT schema
`/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md`	ADR

Validation History

2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
Lessons Learned:
- ```
--secret-file-path
```
  not
```
--secret-file
```
  (Earthly v0.8.16)
- Doppler
```
--format env
```
  outputs quotes, must strip with
```
sed 's/"//g'
```
- README.md must be copied for hatchling build backend
- Colima must be started before Earthly runs

Design Authority

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect
first.

Related Skills

Skill	Purpose
`quality-tools:clickhouse-architect`	Schema design before validation
`devops-tools:clickhouse-cloud-management`	Cloud credentials for E2E tests
`devops-tools:clickhouse-pydantic-config`	Client configuration

Schema E2E Validation

When to Use This Skill

Use this skill when:

Validating schema changes before commit
Verifying YAML schema matches live ClickHouse Cloud
Regenerating Python types, DDL, or docs
Running full schema workflow validation

Prerequisites

Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

bash

colima start

Check if running:

bash

docker ps  # Should not error

Doppler Access (For validation targets)

Required for

+test-schema-validate

and

+test-schema-e2e

bash

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

bash

brew install earthly

Quick Commands

Generation only (no secrets)

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

bash

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

bash

cd /Users/terryli/eon/gapless-network-data
earthly +all

Artifacts

After running

+test-schema-generate

+test-schema-e2e

, check

./earthly-artifacts/

Path	Contents
`types/blocks.py`	Pydantic + TypedDict models
`types/__init__.py`	Package init
`ddl/ethereum_mainnet.sql`	ClickHouse DDL
`docs/ethereum_mainnet.md`	Markdown documentation

For E2E, artifacts are under

e2e/types/

e2e/ddl/

e2e/docs/

Earthfile Targets Reference

Target	Secrets	Purpose
`+deps`	No	Install uv + dependencies
`+build`	No	Copy source files
`+test-unit`	No	Run pytest
`+test-schema-generate`	No	Generate types/DDL/docs
`+test-schema-validate`	Yes	Validate vs ClickHouse
`+test-schema-e2e`	Yes	Full workflow + artifacts
`+all`	No	Run all non-secret targets

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

Cause: Docker/Colima not running

Fix:

bash

colima start
# Wait for "done" message, then retry
earthly +test-schema-generate

"unable to parse --secret-file argument"

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is

--secret-file-path

(NOT

--secret-file

). The wrapper script handles this, but if running manually:

bash

# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

bash

# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud

"OSError: Readme file does not exist: README.md"

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

earthfile

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

"missing secret" during validation

Cause: Doppler not configured or secrets not passed

Fix:

bash

# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate

Cache Issues

Force rebuild without cache:

bash

earthly --no-cache +test-schema-e2e

Implementation Details

Doppler Secret Injection

The wrapper script

scripts/earthly-with-doppler.sh

Downloads secrets from Doppler
Filters for
```
CLICKHOUSE_*
```
variables
Strips quotes (Doppler outputs
```
KEY="value"
```
, Earthly needs
```
KEY=value
```
)
Passes via
```
--secret-file-path
```
flag
Cleans up temp file on exit

Secrets Required

Secret	Purpose
`CLICKHOUSE_HOST_READONLY`	ClickHouse Cloud host
`CLICKHOUSE_USER_READONLY`	Read-only user
`CLICKHOUSE_PASSWORD_READONLY`	Read-only password

Related Files

File	Purpose
`/Users/terryli/eon/gapless-network-data/Earthfile`	Main build file
`/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh`	Secret injection wrapper
`/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml`	SSoT schema
`/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md`	ADR

Validation History

2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
Lessons Learned:
- ```
--secret-file-path
```
  not
```
--secret-file
```
  (Earthly v0.8.16)
- Doppler
```
--format env
```
  outputs quotes, must strip with
```
sed 's/"//g'
```
- README.md must be copied for hatchling build backend
- Colima must be started before Earthly runs

Design Authority

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect
first.

Related Skills

Skill	Purpose
`quality-tools:clickhouse-architect`	Schema design before validation
`devops-tools:clickhouse-cloud-management`	Cloud credentials for E2E tests
`devops-tools:clickhouse-pydantic-config`	Client configuration

schema-e2e-validation

NPX Install

Tags

SKILL.md Content

Schema E2E Validation

When to Use This Skill

Prerequisites

Docker Runtime (Required)

Doppler Access (For validation targets)

Earthly Installation

Quick Commands

Generation only (no secrets)

Full E2E with validation (requires Doppler)

All non-secret targets

Artifacts

Earthfile Targets Reference

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

"unable to parse --secret-file argument"

"OSError: Readme file does not exist: README.md"

"missing secret" during validation

Cache Issues

Implementation Details

Doppler Secret Injection

Secrets Required

Related Files

Validation History

Design Authority

Related Skills

schema-e2e-validation

NPX Install

Tags

SKILL.md Content

Schema E2E Validation

When to Use This Skill

Prerequisites

Docker Runtime (Required)

Doppler Access (For validation targets)

Earthly Installation

Quick Commands

Generation only (no secrets)

Full E2E with validation (requires Doppler)

All non-secret targets

Artifacts

Earthfile Targets Reference

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

"unable to parse --secret-file argument"

"OSError: Readme file does not exist: README.md"

"missing secret" during validation

Cache Issues

Implementation Details

Doppler Secret Injection

Secrets Required

Related Files

Validation History

Design Authority

Related Skills