mirror

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Goldsky Mirror Pipelines

Goldsky Mirror 管道

Mirror is Goldsky's original streaming pipeline product. It reads onchain data from a source (a subgraph entity or a direct-indexing dataset), optionally applies transforms, and writes the result to a sink (your database or message queue).

Mirror是Goldsky的原生流处理管道产品。它从数据源（子图实体或直接索引数据集）读取链上数据，可选择性地应用转换操作，然后将结果写入数据接收器（你的数据库或消息队列）。

Mirror vs Turbo — which should you use?

Mirror vs Turbo — 如何选择？

	Mirror	Turbo
Subgraph sources	Yes	No
Speed & reliability	Good	Faster, more reliable
Sink variety	11 sink types	Growing — new sinks added regularly
Config complexity	Moderate	Simpler YAML
Dataset coverage	130+ chains	130+ chains, richer catalog

Use Turbo unless you need a subgraph source. Turbo is faster, more reliable, and actively gaining feature parity with Mirror — especially sink support. If you don't have a subgraph requirement, say "help me build a Turbo pipeline" and the

/turbo-builder

skill will guide you through a faster setup.

	Mirror	Turbo
子图数据源	支持	不支持
速度与可靠性	良好	更快、更可靠
接收器类型数量	11种	持续增加——定期新增接收器类型
配置复杂度	中等	更简洁的YAML
数据集覆盖范围	130+条链	130+条链，更丰富的数据集目录

除非你需要子图数据源，否则请使用Turbo。Turbo速度更快、更可靠，并且正在逐步实现与Mirror的功能对等——尤其是在接收器支持方面。如果你没有子图需求，请输入"help me build a Turbo pipeline"，

/turbo-builder

技能将引导你完成更快速的设置。

How Mirror Pipelines Work

Mirror管道的工作原理

Source (subgraph entity or direct-indexing dataset)
  ↓
Transforms (optional SQL or external handlers)
  ↓
Sink (PostgreSQL, ClickHouse, Kafka, S3, etc.)

A pipeline is defined in a YAML file (

apiVersion: 3

) and deployed with

goldsky pipeline apply

数据源（子图实体或直接索引数据集）
  ↓
转换操作（可选的SQL或外部处理器）
  ↓
数据接收器（PostgreSQL、ClickHouse、Kafka、S3等）

管道通过YAML文件定义（

apiVersion: 3

），并使用

goldsky pipeline apply

命令部署。

Pipeline YAML Structure

管道YAML结构

Top-level fields:

Field	Type	Required	Description
`name`	string	yes	Lowercase letters, numbers, hyphens only. Under 50 characters.
`apiVersion`	number	yes	Always `3`
`resource_size`	string	no	`s` (default), `m` , `l` , `xl` , `xxl`
`description`	string	no	Pipeline description
`sources`	object	yes	At least one source
`transforms`	object	no	Use `{}` if none needed
`sinks`	object	yes	At least one sink

顶级字段：

字段	类型	是否必填	描述
`name`	string	是	仅允许小写字母、数字和连字符，长度不超过50个字符。
`apiVersion`	number	是	固定为 `3`
`resource_size`	string	否	`s` （默认）、 `m` 、 `l` 、 `xl` 、 `xxl`
`description`	string	否	管道描述
`sources`	object	是	至少一个数据源
`transforms`	object	否	若无转换操作，使用 `{}`
`sinks`	object	是	至少一个数据接收器

Sources

数据源

Source type	YAML `type` value	Description
Subgraph entity	`subgraph_entity`	Mirror data from Goldsky-hosted subgraphs
Dataset (direct indexing)	`dataset`	Raw onchain datasets (blocks, logs, transactions, traces, transfers)

数据源类型	YAML `type` 值	描述
子图实体	`subgraph_entity`	从Goldsky托管的子图中同步数据
数据集（直接索引）	`dataset`	原始链上数据集（区块、日志、交易、追踪记录、转账记录）

Subgraph entity source

子图实体数据源

yaml

sources:
  subgraph_account:
    type: subgraph_entity
    name: account                # Entity name in your subgraph
    start_at: latest             # "earliest" or "latest" (default: latest)
    filter: ""                   # Optional SQL WHERE clause for fast scan
    subgraphs:
      - name: my-subgraph        # Deployed subgraph name
        version: 1.0.0
      - name: my-subgraph-arb    # Cross-chain: add more subgraphs
        version: 1.0.0

Fields:

type

(required:

subgraph_entity

name

(required: entity name),

subgraphs

(required: list of

{name, version}

start_at

(optional),

filter

(optional),

description

(optional).

yaml

sources:
  subgraph_account:
    type: subgraph_entity
    name: account                # 你的子图中的实体名称
    start_at: latest             # "earliest"或"latest"（默认值：latest）
    filter: ""                   # 可选的SQL WHERE子句，用于快速扫描
    subgraphs:
      - name: my-subgraph        # 已部署的子图名称
        version: 1.0.0
      - name: my-subgraph-arb    # 跨链：添加更多子图
        version: 1.0.0

字段：

type

（必填：

subgraph_entity

）、

name

（必填：实体名称）、

subgraphs

（必填：

{name, version}

列表）、

start_at

（可选）、

filter

（可选）、

description

（可选）。

Dataset source

数据集数据源

yaml

sources:
  base_logs:
    type: dataset
    dataset_name: base.logs      # Use `goldsky dataset list` to discover names
    version: 1.0.0               # Use `goldsky dataset get <name>` for versions
    start_at: latest             # "earliest" or "latest" (default: latest)
    filter: "address = '0x...'"  # Optional — enables Fast Scan for backfills

Fields:

type

(required:

dataset

dataset_name

(required),

version

(required),

start_at

(optional),

filter

(optional),

description

(optional).

Fast Scan: When

filter

is defined on a dataset source with

start_at: earliest

, the filter is pre-applied at the source level, making historical backfill much faster. Use attributes that exist in the dataset schema (

goldsky dataset get <dataset_name>

to check).

See docs.goldsky.com/mirror/sources/supported-sources.

yaml

sources:
  base_logs:
    type: dataset
    dataset_name: base.logs      # 使用`goldsky dataset list`命令查看可用数据集名称
    version: 1.0.0               # 使用`goldsky dataset get <name>`命令查看版本
    start_at: latest             # "earliest"或"latest"（默认值：latest）
    filter: "address = '0x...'"  # 可选——启用快速扫描以加速历史数据回填

字段：

type

（必填：

dataset

）、

dataset_name

（必填）、

version

（必填）、

start_at

（可选）、

filter

（可选）、

description

（可选）。

快速扫描：当在

start_at: earliest

的数据集数据源上定义

filter

时，过滤器会在数据源层面预应用，大幅加快历史数据回填速度。请使用数据集架构中存在的属性（使用

goldsky dataset get <dataset_name>

命令查看架构）。

查看docs.goldsky.com/mirror/sources/supported-sources。

Sinks

数据接收器

Sink	YAML `type` value	Notes
PostgreSQL	`postgres`	Most common — OLTP, auto-creates tables, upsert via INSERT ON CONFLICT. Hosted option via NeonDB.
ClickHouse	`clickhouse`	OLAP — uses ReplacingMergeTree by default, `append_only_mode` for best performance
MySQL	`mysql`	OLTP workloads
Elasticsearch	`elasticsearch`	Real-time search and analytics
Kafka	`kafka`	High-throughput streaming to a topic, configurable `topic_partitions`
Object Storage	`file`	S3, GCS, or R2 — Parquet format, append-only, supports `partition_columns`
AWS SQS	`sqs`	Message queuing
Webhook	`webhook`	HTTP POST to an external endpoint

All sinks writing to user-managed destinations require a Goldsky Secret (

secret_name

). Create one with

goldsky secret create

Sinks support

schema_override

for casting column types at the sink level (e.g.,

string

jsonb

接收器	YAML `type` 值	说明
PostgreSQL	`postgres`	最常用——OLTP，自动创建表，通过INSERT ON CONFLICT实现更新插入。支持NeonDB托管选项。
ClickHouse	`clickhouse`	OLAP——默认使用ReplacingMergeTree， `append_only_mode` 可获得最佳性能
MySQL	`mysql`	OLTP工作负载
Elasticsearch	`elasticsearch`	实时搜索与分析
Kafka	`kafka`	高吞吐量流传输到主题，可配置 `topic_partitions`
对象存储	`file`	S3、GCS或R2——Parquet格式，仅追加写入，支持 `partition_columns`
AWS SQS	`sqs`	消息队列
Webhook	`webhook`	HTTP POST到外部端点

所有写入用户托管目标的接收器都需要Goldsky Secret（

secret_name

）。使用

goldsky secret create

命令创建。

接收器支持

schema_override

，用于在接收器层面转换列类型（例如：

string

转

jsonb

）。

Sink examples

接收器示例

yaml

undefined

yaml

undefined

PostgreSQL

sinks: my_pg: type: postgres table: transfers schema: public secret_name: MY_PG_SECRET from: my_transform

ClickHouse

sinks: my_ch: type: clickhouse table: transfers database: my_db secret_name: MY_CH_SECRET from: my_source

Kafka

sinks: my_kafka: type: kafka topic: accounts topic_partitions: 2 secret_name: MY_KAFKA_SECRET from: my_source

Object Storage (S3/GCS/R2)

对象存储（S3/GCS/R2）

sinks: my_s3: type: file path: s3://bucket/path/ format: parquet secret_name: MY_S3_SECRET from: my_source

SQS

sinks: my_sqs: type: sqs url: https://sqs.us-east-1.amazonaws.com/123456/my-queue secret_name: MY_SQS_SECRET from: my_source


See [docs.goldsky.com/mirror/sinks/supported-sinks](https://docs.goldsky.com/mirror/sinks/supported-sinks).

---

sinks: my_sqs: type: sqs url: https://sqs.us-east-1.amazonaws.com/123456/my-queue secret_name: MY_SQS_SECRET from: my_source


查看[docs.goldsky.com/mirror/sinks/supported-sinks](https://docs.goldsky.com/mirror/sinks/supported-sinks)。

---

Transforms

转换操作

Type	YAML `type` value	Description
SQL	(none — default)	Filter, join, or reshape records with SQL
External handler	`handler`	POST records to an HTTP endpoint for custom logic

类型	YAML `type` 值	描述
SQL	(无——默认)	使用SQL过滤、连接或重塑记录
外部处理器	`handler`	将记录POST到HTTP端点以执行自定义逻辑

SQL transform

SQL转换

yaml

transforms:
  filtered_logs:
    sql: SELECT id, block_number, address FROM base_logs WHERE block_number > 1000
    primary_key: id

SQL transforms reference source or transform names as table names. Supports chaining (one transform reads from another).

Built-in decode functions:

```
_gs_log_decode(abi, topics, data)
```
— decode raw log events
```
_gs_tx_decode(abi, input, output)
```
— decode raw trace/transaction data
```
_gs_fetch_abi(url, type)
```
— fetch ABI from URL (etherscan-compatible or raw JSON); fetched once at pipeline start

yaml

transforms:
  filtered_logs:
    sql: SELECT id, block_number, address FROM base_logs WHERE block_number > 1000
    primary_key: id

SQL转换将数据源或转换操作的名称作为表名引用。支持链式调用（一个转换操作读取另一个转换操作的结果）。

内置解码函数：

```
_gs_log_decode(abi, topics, data)
```
—— 解码原始日志事件
```
_gs_tx_decode(abi, input, output)
```
—— 解码原始追踪/交易数据
```
_gs_fetch_abi(url, type)
```
—— 从URL获取ABI（兼容etherscan或原始JSON）；在管道启动时仅获取一次

External handler transform

外部处理器转换

yaml

transforms:
  my_handler:
    type: handler
    primary_key: id
    url: http://example.com/transform
    from: my_source
    batch_size: 100              # Records per batch (default: 100)
    batch_flush_interval: 1s     # Flush interval (default: 1s)
    payload_columns: [col1,col2] # Optional: send subset of columns
    headers:                     # Optional custom headers
      X-Api-Key: my-key

At-least-once delivery with exponential backoff on failure
Max response time: 5 minutes; max connection time: 1 minute
Supports
```
schema_override
```
for return type casting

See docs.goldsky.com/mirror/transforms/sql-transforms.

yaml

transforms:
  my_handler:
    type: handler
    primary_key: id
    url: http://example.com/transform
    from: my_source
    batch_size: 100              # 每批记录数（默认值：100）
    batch_flush_interval: 1s     # 刷新间隔（默认值：1s）
    payload_columns: [col1,col2] # 可选：仅发送部分列
    headers:                     # 可选自定义请求头
      X-Api-Key: my-key

至少一次交付，失败时使用指数退避重试
最大响应时间：5分钟；最大连接时间：1分钟
支持
```
schema_override
```
用于返回类型转换

查看docs.goldsky.com/mirror/transforms/sql-transforms。

Full YAML Examples

完整YAML示例

Subgraph entity to PostgreSQL

子图实体到PostgreSQL

yaml

name: my-subgraph-sync
apiVersion: 3
resource_size: s
sources:
  subgraph_transfer:
    type: subgraph_entity
    name: Transfer
    subgraphs:
      - name: uniswap-v3
        version: 1.0.0
transforms: {}
sinks:
  my_postgres:
    type: postgres
    table: transfers
    schema: public
    secret_name: MY_PG_SECRET
    from: subgraph_transfer

yaml

name: my-subgraph-sync
apiVersion: 3
resource_size: s
sources:
  subgraph_transfer:
    type: subgraph_entity
    name: Transfer
    subgraphs:
      - name: uniswap-v3
        version: 1.0.0
transforms: {}
sinks:
  my_postgres:
    type: postgres
    table: transfers
    schema: public
    secret_name: MY_PG_SECRET
    from: subgraph_transfer

Dataset (direct indexing) to PostgreSQL with SQL transform

数据集（直接索引）到PostgreSQL并使用SQL转换

yaml

name: base-logs-filtered
apiVersion: 3
resource_size: s
sources:
  base_logs:
    type: dataset
    dataset_name: base.logs
    version: 1.0.0
    start_at: earliest
    filter: "address = '0x833589fcd6edb6e08f4c7c32d4f71b54bda02913'"
transforms:
  select_fields:
    sql: SELECT id, block_number, transaction_hash, data FROM base_logs
    primary_key: id
sinks:
  pg_logs:
    type: postgres
    table: base_logs
    schema: public
    secret_name: MY_PG_SECRET
    from: select_fields

No subgraph source? You should almost certainly use Turbo instead — it's faster, more reliable, and has a richer dataset catalog with simpler syntax. Use
/turbo-builder
to get started.

yaml

name: base-logs-filtered
apiVersion: 3
resource_size: s
sources:
  base_logs:
    type: dataset
    dataset_name: base.logs
    version: 1.0.0
    start_at: earliest
    filter: "address = '0x833589fcd6edb6e08f4c7c32d4f71b54bda02913'"
transforms:
  select_fields:
    sql: SELECT id, block_number, transaction_hash, data FROM base_logs
    primary_key: id
sinks:
  pg_logs:
    type: postgres
    table: base_logs
    schema: public
    secret_name: MY_PG_SECRET
    from: select_fields

无需子图数据源？ 你几乎肯定应该使用Turbo替代——它更快、更可靠，拥有更丰富的数据集目录和更简洁的语法。使用
/turbo-builder
开始创建。

CLI Reference — All Pipeline Commands

CLI参考 — 所有管道命令

Global options available on every command:

--token <string>

(CLI auth token),

--color

(colorize output, default true),

-h, --help

所有命令都支持全局选项：

--token <string>

（CLI认证令牌）、

--color

（彩色输出，默认启用）、

-h, --help

。

goldsky pipeline apply <config-path>

goldsky pipeline apply <config-path>

Create or update a pipeline from a YAML config file. Idempotent.

Flag	Type	Description
`--status`	`ACTIVE \| INACTIVE \| PAUSED`	Desired pipeline status
`--from-snapshot`	string	Snapshot to start from: `last` , `new` , `none` , or a snapshot ID. `last` = latest available. `new` = create a fresh snapshot first. `none` = start from scratch. Default: `new`
`--force`	boolean	Skip confirmation prompts (useful for CI)
`--skip-transform-validation`	boolean	Skip transform validation on update
`--save-progress`	boolean	(deprecated, use `--from-snapshot` ) Attempt snapshot before applying
`--use-latest-snapshot`	boolean	(deprecated, use `--from-snapshot` ) Start from latest snapshot
`--skip-validation`	boolean	(deprecated) Same as `--skip-transform-validation`

bash

goldsky pipeline apply my-pipeline.yaml --status ACTIVE
goldsky pipeline apply my-pipeline.yaml --status ACTIVE --from-snapshot last
goldsky pipeline apply my-pipeline.yaml --force   # CI/CD usage

从YAML配置文件创建或更新管道。操作具有幂等性。

标志	类型	描述
`--status`	`ACTIVE \| INACTIVE \| PAUSED`	期望的管道状态
`--from-snapshot`	string	启动时使用的快照： `last` 、 `new` 、 `none` 或快照ID。 `last` = 最新可用快照。 `new` = 先创建新快照。 `none` = 从头开始。默认值： `new`
`--force`	boolean	跳过确认提示（适用于CI场景）
`--skip-transform-validation`	boolean	更新时跳过转换操作验证
`--save-progress`	boolean	(已废弃，使用 `--from-snapshot` ) 应用前尝试创建快照
`--use-latest-snapshot`	boolean	(已废弃，使用 `--from-snapshot` ) 从最新快照启动
`--skip-validation`	boolean	(已废弃) 与 `--skip-transform-validation` 功能相同

bash

goldsky pipeline apply my-pipeline.yaml --status ACTIVE
goldsky pipeline apply my-pipeline.yaml --status ACTIVE --from-snapshot last
goldsky pipeline apply my-pipeline.yaml --force   # CI/CD场景使用

goldsky pipeline start <nameOrConfigPath>

goldsky pipeline start <nameOrConfigPath>

Start a pipeline (equivalent to apply with

--status ACTIVE

Flag	Type	Description
`--from-snapshot`	string	`last` , `new` , `none` , or snapshot ID
`--use-latest-snapshot`	boolean	(deprecated, use `--from-snapshot` )

启动管道（等同于带

--status ACTIVE

参数的apply命令）。

标志	类型	描述
`--from-snapshot`	string	`last` 、 `new` 、 `none` 或快照ID
`--use-latest-snapshot`	boolean	(已废弃，使用 `--from-snapshot` )

goldsky pipeline stop <nameOrConfigPath>

goldsky pipeline stop <nameOrConfigPath>

Stop a pipeline without taking a snapshot. Sets status to INACTIVE, runtime to TERMINATED.

No additional flags beyond global options.

停止管道不创建快照。将状态设置为INACTIVE，运行状态设置为TERMINATED。

除全局选项外无其他标志。

goldsky pipeline pause <nameOrConfigPath>

goldsky pipeline pause <nameOrConfigPath>

Pause a pipeline with a snapshot so it can resume from where it left off. Sets status to PAUSED, runtime to TERMINATED.

No additional flags beyond global options.

暂停管道并创建快照，以便后续从暂停处恢复。将状态设置为PAUSED，运行状态设置为TERMINATED。

除全局选项外无其他标志。

goldsky pipeline restart <nameOrConfigPath>

goldsky pipeline restart <nameOrConfigPath>

Restart a pipeline without configuration changes. Useful when the sink database was restarted, connection is stuck, etc.

Flag	Type	Description
`--from-snapshot`	string	Required. `last` , `new` , `none` , or snapshot ID
`--disable-monitoring`	boolean	Skip monitoring after restart (default: false)

bash

goldsky pipeline restart my-pipeline --from-snapshot last
goldsky pipeline restart my-pipeline --from-snapshot none  # restart from scratch

无需更改配置即可重启管道。适用于接收器数据库重启、连接卡住等场景。

标志	类型	描述
`--from-snapshot`	string	必填。 `last` 、 `new` 、 `none` 或快照ID
`--disable-monitoring`	boolean	重启后跳过监控（默认值：false）

bash

goldsky pipeline restart my-pipeline --from-snapshot last
goldsky pipeline restart my-pipeline --from-snapshot none  # 从头开始重启

goldsky pipeline get <nameOrConfigPath>

goldsky pipeline get <nameOrConfigPath>

Get pipeline configuration and status.

Flag	Type	Description
`--outputFormat, --output`	`json \| table \| yaml`	Output format (default: yaml)
`--definition`	boolean	Print only the pipeline definition (sources, transforms, sinks)
`-v, --version`	string	Pipeline version (default: latest)

获取管道配置和状态。

标志	类型	描述
`--outputFormat, --output`	`json \| table \| yaml`	输出格式（默认值：yaml）
`--definition`	boolean	仅打印管道定义（数据源、转换操作、接收器）
`-v, --version`	string	管道版本（默认值：最新版本）

goldsky pipeline list

goldsky pipeline list

List all pipelines in the project.

Flag	Type	Description
`--output, --outputFormat`	`json \| table \| yaml`	Output format (default: table)
`--outputVerbosity`	`summary \| usablewithapplycmd \| all`	Detail level (default: summary)
`--include-runtime-details`	boolean	Include runtime status and errors (default: false)

bash

goldsky pipeline list --output json
goldsky pipeline list --include-runtime-details

列出项目中的所有管道。

标志	类型	描述
`--output, --outputFormat`	`json \| table \| yaml`	输出格式（默认值：table）
`--outputVerbosity`	`summary \| usablewithapplycmd \| all`	详细程度（默认值：summary）
`--include-runtime-details`	boolean	包含运行状态和错误信息（默认值：false）

bash

goldsky pipeline list --output json
goldsky pipeline list --include-runtime-details

goldsky pipeline info <nameOrConfigPath>

goldsky pipeline info <nameOrConfigPath>

Display pipeline information (status, config, runtime details).

Flag	Type	Description
`-v, --version`	string	Pipeline version (default: latest)

显示管道信息（状态、配置、运行详情）。

标志	类型	描述
`-v, --version`	string	管道版本（默认值：最新版本）

goldsky pipeline monitor <nameOrConfigPath>

goldsky pipeline monitor <nameOrConfigPath>

Monitor pipeline runtime — status, metrics (records received/written), errors. Refreshes every 10 seconds.

Flag	Type	Description
`--update-request`	boolean	Monitor an in-flight update request
`--max-refreshes, --maxRefreshes`	number	Max number of data refreshes
`-v, --version`	string	Pipeline version (default: latest)

监控管道运行状态——状态、指标（接收/写入记录数）、错误信息。每10秒刷新一次。

标志	类型	描述
`--update-request`	boolean	监控正在进行的更新请求
`--max-refreshes, --maxRefreshes`	number	最大数据刷新次数
`-v, --version`	string	管道版本（默认值：最新版本）

goldsky pipeline delete <nameOrConfigPath>

goldsky pipeline delete <nameOrConfigPath>

Delete a pipeline permanently.

Flag	Type	Description
`-f, --force`	boolean	Force deletion without confirmation prompt (default: false)

永久删除管道。

标志	类型	描述
`-f, --force`	boolean	强制删除，跳过确认提示（默认值：false）

goldsky pipeline resize <nameOrConfigPath> <resourceSize>

goldsky pipeline resize <nameOrConfigPath> <resourceSize>

Change the compute resources for a pipeline.

Positional	Description
`resourceSize`	One of: `s` , `m` , `l` , `xl` , `xxl` (default: `s` )

bash

goldsky pipeline resize my-pipeline l

更改管道的计算资源。

位置参数	描述
`resourceSize`	可选值： `s` 、 `m` 、 `l` 、 `xl` 、 `xxl` （默认值： `s` ）

bash

goldsky pipeline resize my-pipeline l

goldsky pipeline validate [config-path]

goldsky pipeline validate [config-path]

Validate a pipeline YAML config without deploying.

Flag	Type	Description
`--definition`	string	(deprecated) Inline JSON definition
`--definition-path`	string	(deprecated) Path to JSON/YAML definition

bash

goldsky pipeline validate my-pipeline.yaml

验证管道YAML配置，无需部署。

标志	类型	描述
`--definition`	string	(已废弃) 内联JSON定义
`--definition-path`	string	(已废弃) JSON/YAML定义文件路径

bash

goldsky pipeline validate my-pipeline.yaml

goldsky pipeline export [name]

goldsky pipeline export [name]

Export pipeline configuration.

Flag	Type	Description
`--all`	boolean	Export configs for all pipelines

导出管道配置。

标志	类型	描述
`--all`	boolean	导出所有管道的配置

goldsky pipeline cancel-update <nameOrConfigPath>

goldsky pipeline cancel-update <nameOrConfigPath>

Cancel an in-flight update or snapshot request. Useful when a long-running snapshot blocks a needed update.

No additional flags beyond global options.

取消正在进行的更新或快照请求。适用于长时间运行的快照阻塞了必要更新的场景。

除全局选项外无其他标志。

goldsky pipeline create <name>

(interactive/guided)

goldsky pipeline create <name>

(交互式引导)

Guided CLI experience for creating a pipeline interactively.

Flag	Type	Description
`--resource-size, --resourceSize`	`s \| m \| l \| xl \| xxl`	Resource size (default: s)
`--use-dedicated-ip`	boolean	Use dedicated egress IPs (default: false)
`--skip-transform-validation`	boolean	Skip transform validation
`--status`	`ACTIVE \| INACTIVE`	(deprecated, use `pipeline start/stop/pause` )
`--description`	string	(deprecated, use `pipeline apply` )
`--definition`	string	(deprecated, use `pipeline apply` )
`--definition-path`	string	(deprecated, use `pipeline apply` )
`--output, --outputFormat`	`json \| table \| yaml`	Output format (default: yaml)

交互式CLI体验，引导创建管道。

标志	类型	描述
`--resource-size, --resourceSize`	`s \| m \| l \| xl \| xxl`	资源大小（默认值：s）
`--use-dedicated-ip`	boolean	使用专用出口IP（默认值：false）
`--skip-transform-validation`	boolean	跳过转换操作验证
`--status`	`ACTIVE \| INACTIVE`	(已废弃，使用 `pipeline start/stop/pause` )
`--description`	string	(已废弃，使用 `pipeline apply` )
`--definition`	string	(已废弃，使用 `pipeline apply` )
`--definition-path`	string	(已废弃，使用 `pipeline apply` )
`--output, --outputFormat`	`json \| table \| yaml`	输出格式（默认值：yaml）

goldsky pipeline get-definition <name>

(deprecated)

goldsky pipeline get-definition <name>

(已废弃)

Get a shareable pipeline definition. Use
goldsky pipeline get <name> --definition
instead.

获取可共享的管道定义。请改用
goldsky pipeline get <name> --definition
。

Snapshot Commands

快照命令

bash

undefined

bash

undefined

List snapshots for a pipeline

列出管道的所有快照

goldsky pipeline snapshots list <nameOrConfigPath> [-v <version>]

Create a snapshot manually

手动创建快照

goldsky pipeline snapshots create <nameOrConfigPath>


`snapshots list` supports `-v, --version` to filter by pipeline version (default: all versions).

---

goldsky pipeline snapshots create <nameOrConfigPath>


`snapshots list`支持`-v, --version`参数按管道版本过滤（默认值：所有版本）。

---

Lifecycle Quick Reference

生命周期速查

Action	Command
Deploy / start	`goldsky pipeline apply <file.yaml> --status ACTIVE`
Start (existing)	`goldsky pipeline start <name>`
Pause (with snapshot)	`goldsky pipeline pause <name>`
Stop (no snapshot)	`goldsky pipeline stop <name>`
Restart (no config change)	`goldsky pipeline restart <name> --from-snapshot last`
Update config	`goldsky pipeline apply <file.yaml>` (edit YAML first)
Resize	`goldsky pipeline resize <name> <size>`
Validate YAML	`goldsky pipeline validate <file.yaml>`
Monitor	`goldsky pipeline monitor <name>`
Get config	`goldsky pipeline get <name> --definition`
Export config	`goldsky pipeline export <name>`
Delete	`goldsky pipeline delete <name> -f`
Cancel in-flight op	`goldsky pipeline cancel-update <name>`
List snapshots	`goldsky pipeline snapshots list <name>`
Create snapshot	`goldsky pipeline snapshots create <name>`
List all pipelines	`goldsky pipeline list`

Pause vs. Stop:

```
pause
```
— takes a snapshot and suspends the pipeline (status: PAUSED + TERMINATED). Can resume from where it left off.
```
stop
```
— stops without taking a snapshot (status: INACTIVE + TERMINATED). Resuming may reprocess data.

Desired statuses: ACTIVE, INACTIVE, PAUSED Runtime statuses: STARTING, RUNNING, FAILING, TERMINATED

操作	命令
部署/启动	`goldsky pipeline apply <file.yaml> --status ACTIVE`
启动（现有管道）	`goldsky pipeline start <name>`
暂停（带快照）	`goldsky pipeline pause <name>`
停止（无快照）	`goldsky pipeline stop <name>`
重启（无配置更改）	`goldsky pipeline restart <name> --from-snapshot last`
更新配置	`goldsky pipeline apply <file.yaml>` （先编辑YAML）
调整资源大小	`goldsky pipeline resize <name> <size>`
验证YAML	`goldsky pipeline validate <file.yaml>`
监控	`goldsky pipeline monitor <name>`
获取配置	`goldsky pipeline get <name> --definition`
导出配置	`goldsky pipeline export <name>`
删除	`goldsky pipeline delete <name> -f`
取消正在进行的操作	`goldsky pipeline cancel-update <name>`
列出快照	`goldsky pipeline snapshots list <name>`
创建快照	`goldsky pipeline snapshots create <name>`
列出所有管道	`goldsky pipeline list`

暂停 vs 停止：

```
pause
```
—— 创建快照并暂停管道（状态：PAUSED + TERMINATED）。可从暂停处恢复。
```
stop
```
—— 停止管道但不创建快照（状态：INACTIVE + TERMINATED）。恢复时可能需要重新处理数据。

期望状态： ACTIVE、INACTIVE、PAUSED 运行状态： STARTING、RUNNING、FAILING、TERMINATED

Snapshots

快照

Snapshots capture a point-in-time state of a RUNNING pipeline for resumption. They contain progress on reading sources and SQL transform state — not sink state.

Automatic snapshots are taken every 4 hours for healthy RUNNING pipelines.
Before updates: a snapshot is created automatically before applying config changes to a RUNNING pipeline.
On pause: a snapshot is created when pausing.

Manual:

goldsky pipeline snapshots create <name>

Resume: only the latest snapshot can be used. For older snapshots, contact support.

The

--from-snapshot

flag (on

apply

start

restart

) controls snapshot behavior:

```
new
```
— create a fresh snapshot, then start from it (default)
```
last
```
— use the latest existing snapshot (no new snapshot)
```
none
```
— start from scratch, no snapshot
```
<snapshot-id>
```
— use a specific snapshot

快照捕获RUNNING状态管道的时间点状态，用于恢复。快照包含数据源读取进度和SQL转换操作状态——不包含接收器状态。

自动快照：健康的RUNNING管道每4小时自动创建一次快照。
更新前：在对RUNNING管道应用配置更改前，会自动创建快照。
暂停时：暂停管道时会创建快照。

手动创建：

goldsky pipeline snapshots create <name>

。

恢复：仅能使用最新快照。如需使用旧快照，请联系支持团队。

--from-snapshot

标志（用于

apply

、

start

、

restart

命令）控制快照行为：

```
new
```
—— 创建新快照，然后从该快照启动（默认值）
```
last
```
—— 使用最新现有快照（不创建新快照）
```
none
```
—— 从头开始，不使用快照
```
<snapshot-id>
```
—— 使用指定快照

Resource Sizing

资源大小设置

Set via

resource_size

in YAML or

goldsky pipeline resize <name> <size>

Size	Description
`s`	Default. Handles most use cases, backfill of small chains, up to 300K records/sec, up to ~8 subgraph sources
`m` , `l` , `xl` , `xxl`	Larger compute — for backfilling large chains or large JOINs

Start small and scale up if needed. Resource size affects pricing.

通过YAML中的

resource_size

字段或

goldsky pipeline resize <name> <size>

命令设置。

大小	描述
`s`	默认值。适用于大多数场景、小链的数据回填，支持最高30万条记录/秒，最多支持约8个子图数据源
`m` 、 `l` 、 `xl` 、 `xxl`	更大的计算资源——适用于大链的数据回填或大型JOIN操作

从小资源大小开始，按需扩容。资源大小会影响定价。

Networking

网络

Mirror pipelines write data from AWS us-west-2. Ensure your sink allows inbound connections from this region.
IP addresses are dynamic by default.
Dedicated egress IPs available on request — use
```
--use-dedicated-ip
```
on
```
pipeline create
```
, or contact support@goldsky.com.
VPC peering available on request.
For external handler transforms, deploy close to us-west-2 for best performance (aim for p95 < 100ms).

Mirror管道从AWS us-west-2区域写入数据。请确保你的接收器允许来自该区域的入站连接。
默认情况下IP地址是动态的。
专用出口IP可按需申请——在
```
pipeline create
```
命令中使用
```
--use-dedicated-ip
```
参数，或联系support@goldsky.com。
可按需申请VPC对等连接。
对于外部处理器转换操作，请将部署位置靠近us-west-2区域以获得最佳性能（目标p95延迟 < 100ms）。

Dataset Discovery

数据集发现

bash

undefined

bash

undefined

List available datasets

列出可用数据集

goldsky dataset list

Get schema for a specific dataset

获取特定数据集的架构

goldsky dataset get <dataset_name>

---

goldsky dataset get <dataset_name>

---

Common Questions

常见问题

Can Mirror pipelines use subgraphs as a source? Yes — this is Mirror's primary advantage over Turbo. Set

type: subgraph_entity

in your source and reference your deployed subgraph.

Can Mirror handle multiple sources or cross-chain data? Yes — define multiple sources in the YAML and use SQL transforms to join or merge them. For subgraphs, you can list multiple subgraphs (different chains) in a single source's

subgraphs

array.

My pipeline needs more resources / is too slow? Run

goldsky pipeline resize <name> l

(or

xl

xxl

). Start small and scale up.

My pipeline is ACTIVE but TERMINATED — what happened? The desired status is ACTIVE but the runtime failed (e.g., bad secret, sink unavailable, resource issues). Check errors with

goldsky pipeline monitor <name> --include-runtime-details

or view the dashboard. Fix the issue and restart.

How do I update a pipeline without losing progress? Edit your YAML and run

goldsky pipeline apply <file.yaml>

. By default, a snapshot is taken before the update is applied. Use

--from-snapshot last

to skip creating a new snapshot and use the latest existing one.

A long snapshot is blocking my update — what do I do? Run

goldsky pipeline cancel-update <name>

to cancel the in-flight operation, then reapply with

--from-snapshot last

--from-snapshot none

Mirror管道能否使用子图作为数据源？ 可以——这是Mirror相对于Turbo的主要优势。在数据源中设置

type: subgraph_entity

并引用已部署的子图即可。

Mirror能否处理多个数据源或跨链数据？ 可以——在YAML中定义多个数据源，并使用SQL转换操作进行连接或合并。对于子图，你可以在单个数据源的

subgraphs

数组中列出多个子图（不同链）。

我的管道需要更多资源/速度太慢？ 运行

goldsky pipeline resize <name> l

（或

xl

、

xxl

）。从小资源大小开始，按需扩容。

我的管道状态是ACTIVE但运行状态是TERMINATED——发生了什么？ 期望状态为ACTIVE但运行时失败（例如：无效密钥、接收器不可用、资源问题）。使用

goldsky pipeline monitor <name> --include-runtime-details

命令查看错误信息，或查看控制台。修复问题后重启管道。

如何在不丢失进度的情况下更新管道？ 编辑YAML文件后运行

goldsky pipeline apply <file.yaml>

。默认情况下，更新前会自动创建快照。使用

--from-snapshot last

参数可跳过创建新快照，直接使用最新现有快照。

长时间运行的快照阻塞了我的更新——该怎么办？ 运行

goldsky pipeline cancel-update <name>

命令取消正在进行的操作，然后使用

--from-snapshot last

或

--from-snapshot none

参数重新应用配置。

mirror

Original

Translation

Goldsky Mirror Pipelines

Goldsky Mirror 管道

Mirror vs Turbo — which should you use?

Mirror vs Turbo — 如何选择？

How Mirror Pipelines Work

Mirror管道的工作原理

Pipeline YAML Structure

管道YAML结构

Sources

数据源

Subgraph entity source

子图实体数据源

Dataset source

数据集数据源

Sinks

数据接收器

Sink examples

接收器示例

PostgreSQL

PostgreSQL

ClickHouse

ClickHouse

Kafka

Kafka

Object Storage (S3/GCS/R2)

对象存储（S3/GCS/R2）

SQS

SQS

Transforms

转换操作

SQL transform

SQL转换

External handler transform

外部处理器转换

Full YAML Examples

完整YAML示例

Subgraph entity to PostgreSQL

子图实体到PostgreSQL

Dataset (direct indexing) to PostgreSQL with SQL transform

数据集（直接索引）到PostgreSQL并使用SQL转换

CLI Reference — All Pipeline Commands

CLI参考 — 所有管道命令

goldsky pipeline apply <config-path>

goldsky pipeline apply <config-path>

goldsky pipeline start <nameOrConfigPath>

goldsky pipeline start <nameOrConfigPath>

goldsky pipeline stop <nameOrConfigPath>

goldsky pipeline stop <nameOrConfigPath>

goldsky pipeline pause <nameOrConfigPath>

goldsky pipeline pause <nameOrConfigPath>

goldsky pipeline restart <nameOrConfigPath>

goldsky pipeline restart <nameOrConfigPath>

goldsky pipeline get <nameOrConfigPath>

goldsky pipeline get <nameOrConfigPath>

goldsky pipeline list

goldsky pipeline list

goldsky pipeline info <nameOrConfigPath>

goldsky pipeline info <nameOrConfigPath>

goldsky pipeline monitor <nameOrConfigPath>

goldsky pipeline monitor <nameOrConfigPath>

goldsky pipeline delete <nameOrConfigPath>

goldsky pipeline delete <nameOrConfigPath>

goldsky pipeline resize <nameOrConfigPath> <resourceSize>

goldsky pipeline resize <nameOrConfigPath> <resourceSize>

goldsky pipeline validate [config-path]

goldsky pipeline validate [config-path]

goldsky pipeline export [name]

goldsky pipeline export [name]

goldsky pipeline cancel-update <nameOrConfigPath>

goldsky pipeline cancel-update <nameOrConfigPath>

goldsky pipeline create <name> (interactive/guided)

goldsky pipeline create <name> (交互式引导)

goldsky pipeline get-definition <name> (deprecated)

goldsky pipeline get-definition <name> (已废弃)

Snapshot Commands

快照命令

List snapshots for a pipeline

`goldsky pipeline apply <config-path>`

`goldsky pipeline apply <config-path>`

`goldsky pipeline start <nameOrConfigPath>`

`goldsky pipeline start <nameOrConfigPath>`

`goldsky pipeline stop <nameOrConfigPath>`

`goldsky pipeline stop <nameOrConfigPath>`

`goldsky pipeline pause <nameOrConfigPath>`

`goldsky pipeline pause <nameOrConfigPath>`

`goldsky pipeline restart <nameOrConfigPath>`

`goldsky pipeline restart <nameOrConfigPath>`

`goldsky pipeline get <nameOrConfigPath>`

`goldsky pipeline get <nameOrConfigPath>`

`goldsky pipeline list`

`goldsky pipeline list`

`goldsky pipeline info <nameOrConfigPath>`

`goldsky pipeline info <nameOrConfigPath>`

`goldsky pipeline monitor <nameOrConfigPath>`

`goldsky pipeline monitor <nameOrConfigPath>`

`goldsky pipeline delete <nameOrConfigPath>`

`goldsky pipeline delete <nameOrConfigPath>`

`goldsky pipeline resize <nameOrConfigPath> <resourceSize>`

`goldsky pipeline resize <nameOrConfigPath> <resourceSize>`

`goldsky pipeline validate [config-path]`

`goldsky pipeline validate [config-path]`

`goldsky pipeline export [name]`

`goldsky pipeline export [name]`

`goldsky pipeline cancel-update <nameOrConfigPath>`

`goldsky pipeline cancel-update <nameOrConfigPath>`

`goldsky pipeline create <name>`
(interactive/guided)

`goldsky pipeline create <name>`
(交互式引导)

`goldsky pipeline get-definition <name>`
(deprecated)

`goldsky pipeline get-definition <name>`
(已废弃)