bigquery-migration-data-loader

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

BigQuery Migration Data Loader

BigQuery 迁移数据加载器

This skill coordinates the automated provisioning of BigQuery datasets, direct ingestion of staged Parquet files into native BigQuery tables by using the

bq load

command, and automated view deployment directly from BigQuery Migration Service (BQMS) translation output directories (local or Cloud Storage).

本 skill 编排 BigQuery 数据集的自动预配、通过

bq load

直接导入暂存的 Parquet 文件，以及从 BigQuery Migration Service (BQMS) 翻译输出目录（本地或 Cloud Storage）自动部署视图。

Usage attribution

使用归属说明

MUST READ: Before executing any

gcloud

, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

必须阅读：在执行任何

gcloud

、REST API 或 MCP 命令前，你必须阅读并遵循使用归属说明中的说明。

Before you begin

开始之前

Google Cloud CLI, project, and location setup: Install the Google Cloud CLI (including the
```
gcloud
```
and
```
bq
```
command-line tools), confirm that your Google Cloud project has an active Cloud Billing account attached, and configure the target project and default Compute Engine region:
bash
```
gcloud config set project PROJECT_ID
gcloud config set compute/region REGION
```
```
gcloud config set compute/region
```
sets the default Compute Engine region; the
```
bq
```
CLI does not read
```
compute/region
```
and defaults to
```
US
```
unless
```
--location
```
is passed on
```
bq
```
commands or set in
```
~/.bigqueryrc
```
(or configured through a project or organization
```
default_location
```
). BigQuery is retiring the multi-region designation—including
```
US
```
and
```
EU
```
in the single-regions list colocated with
```
us-central1
```
and
```
europe-west4
```
, with no change to existing data residency (see BigQuery locations)—so always pass an explicit
```
--location
```
on
```
bq
```
commands that matches your staged Cloud Storage bucket, as described in step 5 (check regional service availability at Google Cloud locations).

Enable the APIs: Ask the user before enabling APIs, then enable the BigQuery and Cloud Storage APIs (

bigquery.googleapis.com

and

storage.googleapis.com

bash

gcloud services enable bigquery.googleapis.com storage.googleapis.com

Authentication and required IAM roles: Authenticate with
```
gcloud auth login
```
and
```
gcloud auth application-default login
```
(or set
```
GOOGLE_APPLICATION_CREDENTIALS
```
for service account authentication). Verify that your identity or service account has the required IAM roles on each resource:
- Grant BigQuery Job User (
```
roles/bigquery.jobUser
```
  ) on the target Google Cloud project.
- Grant BigQuery Data Editor (
```
roles/bigquery.dataEditor
```
  ) on the target Google Cloud project (to create new datasets with
```
bq mk
```
  ) or on an existing target dataset (to create or overwrite tables and views).
- Grant Storage Object Viewer (
```
roles/storage.objectViewer
```
  ) on the Cloud Storage bucket holding staged Parquet files and translation metadata.
Python environment: Ask the user for confirmation before installing Python packages, then install PyYAML in a virtual environment:
bash
```
pip install pyyaml==6.0.2
```
Location, cost, and cleanup: Use the location of your staged Cloud Storage bucket for the dataset and every
```
bq
```
command: the bucket's region for a regional bucket, or
```
US
```
or
```
EU
```
for a
```
US
```
or
```
EU
```
multi-region bucket. Default to
```
us-central1
```
only when you also choose the bucket location. Loading from a bucket in a different location incurs data transfer charges (see BigQuery locations). Review Google Cloud pricing before loading large datasets or running audit queries. Ask the user for confirmation before deleting temporary staged Parquet files (
```
gcloud storage rm -r gs://BUCKET/EXPORT_PREFIX
```
) or test datasets (
```
bq rm -r -f PROJECT_ID:DATASET_ID
```
) when no longer needed.

Google Cloud CLI、项目和位置设置：安装 Google Cloud CLI（包括
```
gcloud
```
和
```
bq
```
命令行工具），确认你的 Google Cloud 项目已绑定有效的 Cloud Billing 账户，并配置目标项目和默认 Compute Engine 区域：
bash
```
gcloud config set project PROJECT_ID
gcloud config set compute/region REGION
```
```
gcloud config set compute/region
```
会设置默认 Compute Engine 区域；
```
bq
```
CLI 不读取
```
compute/region
```
，除非在
```
bq
```
命令中传入
```
--location
```
，或在
```
~/.bigqueryrc
```
中设置（也可通过项目或组织的
```
default_location
```
配置），否则默认使用
```
US
```
。BigQuery 正在弃用 multi-region 标识——其中包括把
```
US
```
和
```
EU
```
与
```
us-central1
```
和
```
europe-west4
```
并列在单一区域列表中的做法——但不会改变现有的数据驻留（请参阅 BigQuery 位置）——因此，始终在
```
bq
```
命令中传入与暂存 Cloud Storage 存储桶匹配的显式
```
--location
```
，如步骤 5 所述（请在 Google Cloud 位置检查区域服务可用性）。

启用 API：启用前先征得用户同意，然后启用 BigQuery 和 Cloud Storage API（

bigquery.googleapis.com

和

storage.googleapis.com

）：

bash

gcloud services enable bigquery.googleapis.com storage.googleapis.com

身份验证和必需的 IAM 角色：使用
```
gcloud auth login
```
和
```
gcloud auth application-default login
```
进行身份验证（或设置
```
GOOGLE_APPLICATION_CREDENTIALS
```
以使用服务账号认证）。确认你的身份或服务账号在各资源上拥有必需的 IAM 角色：
- 在目标 Google Cloud 项目上授予 BigQuery Job User（
```
roles/bigquery.jobUser
```
  ）。
- 在目标 Google Cloud 项目上授予 BigQuery Data Editor（
```
roles/bigquery.dataEditor
```
  ），以通过
```
bq mk
```
  创建新数据集；或在现有目标数据集上授予该角色，以创建或覆盖表和视图。
- 在保存暂存 Parquet 文件和翻译元数据的 Cloud Storage 存储桶上授予 Storage Object Viewer（
```
roles/storage.objectViewer
```
  ）。
Python 环境：安装 Python 包前先请求用户确认，然后在虚拟环境中安装 PyYAML：
bash
```
pip install pyyaml==6.0.2
```
位置、成本和清理：为数据集及每条
```
bq
```
命令使用暂存 Cloud Storage 存储桶所在的位置：区域存储桶使用存储桶所在区域，
```
US
```
或
```
EU
```
multi-region 存储桶则分别使用
```
US
```
或
```
EU
```
。只有在同时选择存储桶位置为
```
us-central1
```
时，才将其作为默认值。从不同位置的存储桶加载数据会产生数据传输费用（请参阅 BigQuery 位置）。加载大型数据集或运行审计查询前，请查看 Google Cloud 定价。不再需要时，删除临时暂存的 Parquet 文件（
```
gcloud storage rm -r gs://BUCKET/EXPORT_PREFIX
```
）或测试数据集（
```
bq rm -r -f PROJECT_ID:DATASET_ID
```
）前，请先请求用户确认。

1. Loader workflow overview

1. 加载器工作流概览

The following diagram illustrates the four-stage ingestion and view deployment pipeline:

mermaid

flowchart TD
    subgraph 1. Inputs
        A1["Translation Output Directory / Cloud Storage URI<br>gs://bucket/translation_output/metadata/"]
        A2["Staged Parquet Files<br>gs://bucket/data_unload/TABLE/*.parquet"]
    end

    subgraph 2. Ingestion Pipeline
        B1["1. Ensure Dataset Exists<br>bq mk --dataset --location=LOCATION DATASET"]
        B2["2. Concurrent Parquet Ingestion<br>bq load --source_format=PARQUET --replace=true<br>Auto-creates native tables"]
        B3["3. Row Count Audit<br>Print BigQuery Row Counts for Comparison"]
        B4["4. Deploy Consumption Views<br>Deploys views directly from consolidated views.sql or relations/"]
    end

    A1 & A2 --> B1 --> B2 --> B3 --> B4

下图展示了四阶段导入与视图部署流水线：

mermaid

flowchart TD
    subgraph 1. 输入
        A1["翻译输出目录 / Cloud Storage URI<br>gs://bucket/translation_output/metadata/"]
        A2["暂存的 Parquet 文件<br>gs://bucket/data_unload/TABLE/*.parquet"]
    end

    subgraph 2. 导入流水线
        B1["1. 确保数据集存在<br>bq mk --dataset --location=LOCATION DATASET"]
        B2["2. 并发 Parquet 导入<br>bq load --source_format=PARQUET --replace=true<br>自动创建原生表"]
        B3["3. 行数审计<br>输出 BigQuery 行数供比较"]
        B4["4. 部署消费视图<br>直接从合并后的 views.sql 或 relations/ 部署视图"]
    end

    A1 & A2 --> B1 --> B2 --> B3 --> B4

2. Direct translation output ingestion

2. 直接导入翻译输出

[!TIP] Migration workflow context: Before running the loader, translate source schemas and views into GoogleSQL by using the
bigquery-migration-sql-translator
skill and stage exported source table Parquet files in Cloud Storage. After loading tables and deploying views, compare row counts and data aggregates against the source database.

The data loader directly consumes BQMS translation output directories (the

--translation-metadata-dir

flag):

Zero-DDL native ingestion: Table schemas are inferred and created automatically from staged Parquet files during
```
bq load
```
. Inferred types follow the Parquet types, not the translated DDL: naive source timestamps (no time zone) load as
```
TIMESTAMP
```
instead of
```
DATETIME
```
, and
```
STRING(n)
```
and
```
NUMERIC(p, s)
```
parameters are dropped (see Step 2 in Section 3).
Automated object discovery: Base tables and views are auto-discovered directly from the translation output metadata. Pass the BQMS
```
metadata/
```
directory (or its parent), which contains
```
config/DATABASE/SCHEMA/*.yaml
```
files whose
```
sourceName
```
names each object (table configs also set
```
externalTableName
```
or
```
externalTableDdlPath
```
; the others are views) and view DDL in
```
relations/db_DATABASE/schema_SCHEMA/VIEW.sql
```
(or a single
```
views.sql
```
).
Automated view deployment: Translated views are deployed directly into the target dataset with source references rewritten to the target BigQuery project and dataset. Plain
```
CREATE VIEW
```
statements are deployed as
```
CREATE OR REPLACE VIEW
```
so that reruns don't fail on existing views.
Flexible Parquet path resolution: Under the
```
--parquet-base-uri
```
(
```
BASE_URI
```
) flag, the loader automatically checks schema-scoped subfolders (
```
BASE_URI/SCHEMA/TABLE/*.parquet
```
) first if present and falls back to flat table paths (
```
BASE_URI/TABLE/*.parquet
```
). For example,
```
gcloud storage cp -r LOCAL_DIR/SCHEMA gs://BUCKET/data_unload/
```
stages a local schema folder of table subfolders under
```
data_unload/SCHEMA/
```
.
Single target dataset: All discovered schemas load into one
```
--dataset
```
by table or view name. If tables or views from different source databases or schemas share a name, the loader stops before creating the dataset and lists the conflicting objects; load each schema into its own dataset instead.

[!TIP] 迁移工作流上下文：在运行加载器之前，使用
bigquery-migration-sql-translator
skill 将源 schema 和视图转换为 GoogleSQL，并在 Cloud Storage 中暂存导出的源表 Parquet 文件。加载表并部署视图后，将行数和数据聚合结果与源数据库进行比较。

数据加载器直接使用 BQMS 翻译输出目录（

--translation-metadata-dir

标志）：

零 DDL 原生导入：在
```
bq load
```
期间，从暂存的 Parquet 文件推断并自动创建表 schema。推断出的类型遵循 Parquet 类型，而不是转换后的 DDL：无时区的源时间戳会加载为
```
TIMESTAMP
```
而不是
```
DATETIME
```
，并且会丢弃
```
STRING(n)
```
和
```
NUMERIC(p, s)
```
参数（请参阅第 3 节中的步骤 2）。
自动发现对象：直接从翻译输出元数据中自动发现基础表和视图。传入 BQMS 的
```
metadata/
```
目录（或其父目录），其中包含
```
config/DATABASE/SCHEMA/*.yaml
```
文件，
```
sourceName
```
用于命名每个对象（表配置还会设置
```
externalTableName
```
或
```
externalTableDdlPath
```
，其余对象为视图），视图 DDL 位于
```
relations/db_DATABASE/schema_SCHEMA/VIEW.sql
```
（或单个
```
views.sql
```
）中。
自动部署视图：将转换后的视图直接部署到目标数据集，并将源引用重写为目标 BigQuery 项目和数据集。普通的
```
CREATE VIEW
```
语句会部署为
```
CREATE OR REPLACE VIEW
```
，因此重新运行时不会因视图已存在而失败。
灵活的 Parquet 路径解析：在
```
--parquet-base-uri
```
（
```
BASE_URI
```
）标志下，加载器会优先检查按 schema 划分的子文件夹（
```
BASE_URI/SCHEMA/TABLE/*.parquet
```
），如果存在则使用该路径，否则回退到扁平表路径（
```
BASE_URI/TABLE/*.parquet
```
）。例如，
```
gcloud storage cp -r LOCAL_DIR/SCHEMA gs://BUCKET/data_unload/
```
会将包含表子文件夹的本地 schema 文件夹暂存到
```
data_unload/SCHEMA/
```
下。
单一目标数据集：所有发现的 schema 都按表名或视图名加载到同一个
```
--dataset
```
中。如果来自不同源数据库或 schema 的表或视图同名，加载器会在创建数据集前停止并列出冲突对象；此时应改为将每个 schema 加载到各自独立的数据集。

3. CLI execution interface (

bq load

)

3. CLI 执行接口（

bq load

）

Step 1: Provision the target dataset

步骤 1：预配目标数据集

Set the shell variables for your environment, and then create the target dataset if it doesn't already exist:

bash

PROJECT_ID="PROJECT_ID"
DATASET_ID="DATASET_ID"
LOCATION="LOCATION"
TABLE_NAME="TABLE_NAME"
BUCKET="BUCKET"
PREFIX="data_unload"

bq show --dataset --project_id=${PROJECT_ID} ${DATASET_ID} >/dev/null 2>&1 || \
bq mk --dataset --project_id=${PROJECT_ID} --location=${LOCATION} ${DATASET_ID}

Replace the following:

```
PROJECT_ID
```
: the target Google Cloud project ID.
```
DATASET_ID
```
: the target BigQuery dataset ID.
```
LOCATION
```
: the BigQuery location for the dataset and load jobs (for example,
```
us-central1
```
).
```
TABLE_NAME
```
: the name of the table to load.
```
BUCKET
```
: the name of the Cloud Storage bucket that holds the staged Parquet files.

为环境设置 shell 变量，然后在目标数据集尚不存在时创建该数据集：

bash

PROJECT_ID="PROJECT_ID"
DATASET_ID="DATASET_ID"
LOCATION="LOCATION"
TABLE_NAME="TABLE_NAME"
BUCKET="BUCKET"
PREFIX="data_unload"

bq show --dataset --project_id=${PROJECT_ID} ${DATASET_ID} >/dev/null 2>&1 || \
bq mk --dataset --project_id=${PROJECT_ID} --location=${LOCATION} ${DATASET_ID}

替换以下变量：

```
PROJECT_ID
```
：目标 Google Cloud 项目 ID。
```
DATASET_ID
```
：目标 BigQuery 数据集 ID。
```
LOCATION
```
：数据集和加载作业所在的 BigQuery 位置（例如
```
us-central1
```
）。
```
TABLE_NAME
```
：要加载的表名。
```
BUCKET
```
：保存暂存 Parquet 文件的 Cloud Storage 存储桶名称。

Step 2: Ingest Parquet tables (auto-creates native table schema)

步骤 2：导入 Parquet 表（自动创建原生表 schema）

Pass

--decimal_target_types=NUMERIC --decimal_target_types=BIGNUMERIC --decimal_target_types=STRING

so BigQuery selects the smallest compatible type (

NUMERIC

→

BIGNUMERIC

→

STRING

) based on each Parquet

DECIMAL(p, s)

column's precision and scale: columns with

p - s <= 29, s <= 9

map to

NUMERIC

, columns with

p - s <= 38, s <= 38

(such as

DECIMAL(38, 10)

) widen to

BIGNUMERIC

, and larger

DECIMAL256

precisions or scales fall back to

STRING

bash

bq load \
    --project_id=${PROJECT_ID} \
    --location=${LOCATION} \
    --source_format=PARQUET \
    --replace=true \
    --decimal_target_types=NUMERIC \
    --decimal_target_types=BIGNUMERIC \
    --decimal_target_types=STRING \
    ${DATASET_ID}.${TABLE_NAME} \
    "gs://${BUCKET}/${PREFIX}/${TABLE_NAME}/*.parquet"

The

--replace=true

flag overwrites the destination table, including its schema, with the schema inferred from the Parquet files. If you pre-created tables from translated BQMS DDL statements to preserve target types such as

JSON

INTERVAL

, or

RANGE

, either run

TRUNCATE TABLE

and then load with

--replace=false

(so that reruns stay idempotent and don't duplicate rows), or keep

--replace=true

and pass an explicit

--schema

. If you use the loader template with native

JSON

columns, also set

_WRAP_JSON_QUERY_WITH_PARSE_JSON = False

in the copied script: by default it wraps

JSON_QUERY()

calls in

SAFE.PARSE_JSON()

for

JSON

source columns that Parquet loads as

STRING

, which fails on native

JSON

columns.

Parquet

TIMESTAMP

columns always load as BigQuery

TIMESTAMP

, including naive timestamps (

isAdjustedToUTC=false

, such as source

TIMESTAMP WITHOUT TIME ZONE

columns) that the translated DDL maps to

DATETIME

; the wall-clock values are kept and read as UTC. To restore

DATETIME

after loading, rewrite those columns with

DATETIME(column)

, which converts at UTC by default:

sql

CREATE OR REPLACE TABLE `PROJECT_ID.DATASET_ID.TABLE_NAME` AS
SELECT * REPLACE (DATETIME(created_at) AS created_at)
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`;

传入

--decimal_target_types=NUMERIC --decimal_target_types=BIGNUMERIC --decimal_target_types=STRING

，使 BigQuery 根据每个 Parquet

DECIMAL(p, s)

列的精度和小数位选择兼容的最小类型（

NUMERIC

→

BIGNUMERIC

→

STRING

）：满足

p - s <= 29, s <= 9

的列映射为

NUMERIC

，满足

p - s <= 38, s <= 38

的列（例如

DECIMAL(38, 10)

）扩展为

BIGNUMERIC

，更大的

DECIMAL256

精度或小数位则回退为

STRING

：

bash

bq load \
    --project_id=${PROJECT_ID} \
    --location=${LOCATION} \
    --source_format=PARQUET \
    --replace=true \
    --decimal_target_types=NUMERIC \
    --decimal_target_types=BIGNUMERIC \
    --decimal_target_types=STRING \
    ${DATASET_ID}.${TABLE_NAME} \
    "gs://${BUCKET}/${PREFIX}/${TABLE_NAME}/*.parquet"

--replace=true

标志会使用从 Parquet 文件推断出的 schema 覆盖目标表，包括其 schema。如果已根据转换后的 BQMS DDL 语句预先创建表，以保留

JSON

、

INTERVAL

或

RANGE

等目标类型，请选择运行

TRUNCATE TABLE

后再使用

--replace=false

加载（这样重新运行仍可保持幂等，并且不会重复插入数据），或者保留

--replace=true

并传入显式

--schema

。如果加载器模板包含原生

JSON

列，还必须在复制后的脚本中将

_WRAP_JSON_QUERY_WITH_PARSE_JSON = False

设为该值：默认情况下，它会针对由 Parquet 加载为

STRING

的

JSON

源列，将

JSON_QUERY()

调用包装在

SAFE.PARSE_JSON()

中，这会在原生

JSON

列上失败。

Parquet

TIMESTAMP

列始终会加载为 BigQuery

TIMESTAMP

，包括无时区时间戳（

isAdjustedToUTC=false

，例如源

TIMESTAMP WITHOUT TIME ZONE

列），即使转换后的 DDL 将其映射为

DATETIME

。原始时钟值会被保留，并按 UTC 读取。要在加载后恢复为

DATETIME

，请使用

DATETIME(column)

重写这些列；该函数默认按 UTC 转换：

sql

CREATE OR REPLACE TABLE `PROJECT_ID.DATASET_ID.TABLE_NAME` AS
SELECT * REPLACE (DATETIME(created_at) AS created_at)
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`;

Step 3: Run a row-count audit

步骤 3：运行行数审计

Count the rows in the loaded table so that you can compare the result with the source table:

bash

bq query \
    --project_id=${PROJECT_ID} \
    --location=${LOCATION} \
    --use_legacy_sql=false \
    "SELECT '${TABLE_NAME}' AS table_name, count(*) AS row_count FROM \`${PROJECT_ID}\`.\`${DATASET_ID}\`.\`${TABLE_NAME}\`;"

统计已加载表中的行数，以便将结果与源表进行比较：

bash

bq query \
    --project_id=${PROJECT_ID} \
    --location=${LOCATION} \
    --use_legacy_sql=false \
    "SELECT '${TABLE_NAME}' AS table_name, count(*) AS row_count FROM \`${PROJECT_ID}\`.\`${DATASET_ID}\`.\`${TABLE_NAME}\`;"

4. Automated loader asset template (

assets/bq_loader_template.py

)

4. 自动化加载器资产模板（

assets/bq_loader_template.py

）

This skill provides an [automated BigQuery loader template][loader_template] (

assets/bq_loader_template.py

) for dataset creation, concurrent Parquet loading, direct view deployment, and table row-count auditing.

此 skill 提供了一个[自动化 BigQuery 加载器模板][loader_template]（

assets/bq_loader_template.py

），用于创建数据集、并发加载 Parquet、直接部署视图和审计表行数。

Agent workflow: Copy, conditionally customize, and execute

Agent 工作流：复制、按需定制并执行

Copy the asset template: Always instruct the user to copy the [automated BigQuery loader template][loader_template] (
```
assets/bq_loader_template.py
```
) from this skill's directory (
```
SKILL_DIR
```
) into their working directory (
```
./bq_loader.py
```
) by using the
```
cp
```
command—never write, synthesize, or inline a custom Python loader script from scratch, and never paste or re-type the template contents into the response. The template already handles quote-aware SQL splitting, destination name collision checks, chunked row-count audits, and multi-pass view retries, which a rewritten script would likely miss.
```
SKILL_DIR
```
is the directory containing this
```
SKILL.md
```
:
bash
```
cp SKILL_DIR/assets/bq_loader_template.py ./bq_loader.py
```
Conditionally customize extension points: Run the copy as-is. Modify it only if one of the following applies (prefer editing the marked
```
# CUSTOMIZE:
```
sections;
```
# DEFAULT SCAFFOLDING
```
sections can be kept unchanged unless deeper orchestration adjustments are needed):
- Schema overrides and partitioning (
  # CUSTOMIZE: Schema Overrides & Partitioning
  ): Add custom
```
bq load
```
  flags (such as
```
--time_partitioning_field
```
  ,
```
--clustering_fields
```
  , or explicit schema JSON files) inside
```
load_parquet_table()
```
  .
- Custom view rewriting (
  # CUSTOMIZE: Custom View Rewriting & Deployment
  ): Adjust view SQL regular expressions, cross-project dataset mapping rules, or retry bounds in
```
_rewrite_view_sql()
```
  and
```
deploy_views()
```
  . Set
```
_WRAP_JSON_QUERY_WITH_PARSE_JSON = False
```
  if you pre-created tables with native
```
JSON
```
  columns.
- Connection and CLI wrapper (
  # CUSTOMIZE: Connection & CLI Wrapper
  ): Pass custom
```
bq
```
  CLI flags (for example,
```
--default_kms_key
```
  in
```
ensure_dataset()
```
  , or
```
--destination_kms_key
```
  and
```
--reservation_id
```
  in
```
load_parquet_table()
```
  and
```
execute_sql()
```
  ).
- Post-load auditing (
  # CUSTOMIZE: Post-Load Auditing
  ): Extend
```
audit_row_counts()
```
  with custom data integrity or column-level checksum queries.

Execute the script in your working directory: Run the copied script against a Cloud Storage URI or a local translation output folder:

bash

# Direct from a Cloud Storage translation_output URI with concurrency:
python3 ./bq_loader.py \
    --project "PROJECT_ID" \
    --dataset "DATASET_ID" \
    --location "LOCATION" \
    --translation-metadata-dir "gs://BUCKET/translation_output/metadata/" \
    --parquet-base-uri "gs://BUCKET/data_unload/" \
    --concurrency 8

# Using local translation output folder:
python3 ./bq_loader.py \
    --project "PROJECT_ID" \
    --dataset "DATASET_ID" \
    --location "LOCATION" \
    --translation-metadata-dir "TRANSLATION_OUTPUT/metadata" \
    --parquet-base-uri "gs://BUCKET/data_unload/" \
    --concurrency 4

When a view depends on another view, the first batch and first pass print

bq

errors such as

Not found: Table ...

for the dependent view. These errors are expected if the run ends with

All N views successfully deployed across M passes.

复制资产模板：始终指导用户使用
```
cp
```
命令，将此 skill 目录（
```
SKILL_DIR
```
）中的[自动化 BigQuery 加载器模板][loader_template]（
```
assets/bq_loader_template.py
```
）复制到其工作目录（
```
./bq_loader.py
```
）——绝不从头编写、合成或内联自定义 Python 加载器脚本，也绝不将模板内容粘贴或重新键入回复中。模板已经处理了可识别引号的 SQL 拆分、目标名称冲突检查、分块行数审计和多轮视图重试，重写脚本很可能会遗漏这些处理。
```
SKILL_DIR
```
是包含此
```
SKILL.md
```
的目录：
bash
```
cp SKILL_DIR/assets/bq_loader_template.py ./bq_loader.py
```
按需定制扩展点：先按原样运行复制后的脚本。仅在以下情况之一适用时修改；优先编辑标记为
```
# CUSTOMIZE:
```
的部分，除非需要更深层的编排调整，否则
```
# DEFAULT SCAFFOLDING
```
部分可以保持不变：
- Schema 覆盖和分区（
  # CUSTOMIZE: Schema Overrides & Partitioning
  ）：在
```
load_parquet_table()
```
  中添加自定义
```
bq load
```
  标志，例如
```
--time_partitioning_field
```
  、
```
--clustering_fields
```
  或显式 schema JSON 文件。
- 自定义视图重写和部署（
  # CUSTOMIZE: Custom View Rewriting & Deployment
  ）：在
```
_rewrite_view_sql()
```
  和
```
deploy_views()
```
  中调整视图 SQL 正则表达式、跨项目数据集映射规则或重试次数上限。如果已预先创建包含原生
```
JSON
```
  列的表，请将
```
_WRAP_JSON_QUERY_WITH_PARSE_JSON
```
  设为
```
False
```
  。
- 连接与 CLI 包装器（
  # CUSTOMIZE: Connection & CLI Wrapper
  ）：传入自定义
```
bq
```
  CLI 标志，例如在
```
ensure_dataset()
```
  中使用
```
--default_kms_key
```
  ，或在
```
load_parquet_table()
```
  和
```
execute_sql()
```
  中使用
```
--destination_kms_key
```
  和
```
--reservation_id
```
  。
- 加载后审计（
  # CUSTOMIZE: Post-Load Auditing
  ）：使用自定义数据完整性或列级校验和查询扩展
```
audit_row_counts()
```
  。

在工作目录中执行脚本：针对 Cloud Storage URI 或本地翻译输出文件夹运行复制后的脚本：

bash

# Directly from a Cloud Storage translation_output URI with concurrency:
python3 ./bq_loader.py \
    --project "PROJECT_ID" \
    --dataset "DATASET_ID" \
    --location "LOCATION" \
    --translation-metadata-dir "gs://BUCKET/translation_output/metadata/" \
    --parquet-base-uri "gs://BUCKET/data_unload/" \
    --concurrency 8

# Using local translation output folder:
python3 ./bq_loader.py \
    --project "PROJECT_ID" \
    --dataset "DATASET_ID" \
    --location "LOCATION" \
    --translation-metadata-dir "TRANSLATION_OUTPUT/metadata" \
    --parquet-base-uri "gs://BUCKET/data_unload/" \
    --concurrency 4

当一个视图依赖另一个视图时，第一批和第一轮会输出

bq

错误，例如

Not found: Table ...

。如果运行最终显示

All N views successfully deployed across M passes.

，则这些错误属于预期情况。

Arguments

参数

```
--project
```
(Required): Google Cloud project ID.
```
--dataset
```
(Required): Target BigQuery dataset name.
```
--parquet-base-uri
```
(Required): Base Cloud Storage URI containing table Parquet subfolders (
```
gs://BUCKET/path/
```
).
```
--translation-metadata-dir
```
(Required): Local folder or Cloud Storage URI to translation output metadata directory. Auto-discovers tables and views.
```
--location
```
(Optional, Default:
us-central1
): Target BigQuery location.
```
--concurrency
```
(Optional, Default:
4
): Number of parallel workers for concurrent
```
bq load
```
operations.

```
--project
```
（必需）：Google Cloud 项目 ID。
```
--dataset
```
（必需）：目标 BigQuery 数据集名称。
```
--parquet-base-uri
```
（必需）：包含表 Parquet 子文件夹的基础 Cloud Storage URI（
```
gs://BUCKET/path/
```
）。
```
--translation-metadata-dir
```
（必需）：翻译输出元数据目录的本地文件夹或 Cloud Storage URI。自动发现表和视图。
```
--location
```
（可选，默认值：
us-central1
）：目标 BigQuery 位置。
```
--concurrency
```
（可选，默认值：
4
）：并发执行
```
bq load
```
操作的并行工作进程数。

bigquery-migration-data-loader

Original

Translation

BigQuery Migration Data Loader

BigQuery 迁移数据加载器

Usage attribution

使用归属说明

Before you begin

开始之前

1. Loader workflow overview

1. 加载器工作流概览

2. Direct translation output ingestion

2. 直接导入翻译输出

3. CLI execution interface (
`bq load`
)

3. CLI 执行接口（
`bq load`
）

Step 1: Provision the target dataset

步骤 1：预配目标数据集

Step 2: Ingest Parquet tables (auto-creates native table schema)

步骤 2：导入 Parquet 表（自动创建原生表 schema）

Step 3: Run a row-count audit

步骤 3：运行行数审计

4. Automated loader asset template (
`assets/bq_loader_template.py`
)

4. 自动化加载器资产模板（
`assets/bq_loader_template.py`
）

Agent workflow: Copy, conditionally customize, and execute

Agent 工作流：复制、按需定制并执行

Arguments

参数

bigquery-migration-data-loader

Original

Translation

BigQuery Migration Data Loader

BigQuery 迁移数据加载器

Usage attribution

使用归属说明

Before you begin

开始之前

1. Loader workflow overview

1. 加载器工作流概览

2. Direct translation output ingestion

2. 直接导入翻译输出

3. CLI execution interface (bq load)

3. CLI 执行接口（bq load）

Step 1: Provision the target dataset

步骤 1：预配目标数据集

Step 2: Ingest Parquet tables (auto-creates native table schema)

步骤 2：导入 Parquet 表（自动创建原生表 schema）

Step 3: Run a row-count audit

步骤 3：运行行数审计

4. Automated loader asset template (assets/bq_loader_template.py)

4. 自动化加载器资产模板（assets/bq_loader_template.py）

Agent workflow: Copy, conditionally customize, and execute

Agent 工作流：复制、按需定制并执行

Arguments

参数

3. CLI execution interface (
`bq load`
)

3. CLI 执行接口（
`bq load`
）

4. Automated loader asset template (
`assets/bq_loader_template.py`
)

4. 自动化加载器资产模板（
`assets/bq_loader_template.py`
）