bigquery-migration-sql-translator
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseBigQuery Migration Service (BQMS) SQL Translator
BigQuery Migration Service (BQMS) SQL Translator
The BigQuery Migration Service (BQMS) translates legacy database SQL scripts,
queries, and schemas into GoogleSQL. This skill coordinates translation
workflows, Cloud Storage staging, compiler error triage, and YAML configuration.
BigQuery Migration Service (BQMS) 将传统数据库的 SQL 脚本、查询和 schema 翻译为 GoogleSQL。该 skill 协调翻译工作流、Cloud Storage 暂存、编译错误排查和 YAML 配置。
Before you begin
开始之前
Before translating queries or schemas, verify the following prerequisites:
-
Google Cloud SDK & Project Setup: Ensure the Google Cloud SDK is installed, verify that your Google Cloud project has an active Cloud Billing account attached, and select the target project:bash
gcloud config set project {project_id} -
Enable the API: Enable the BigQuery Migration API () in your Google Cloud project:
bigquerymigration.googleapis.combashgcloud services enable bigquerymigration.googleapis.com -
Required IAM Roles: Verify that your identity or service account has:
- (MCP Tool User)
roles/mcp.toolUser - (Migration Workflow Editor)
roles/bigquerymigration.editor - For batch translation, and
roles/storage.objectViewer(orroles/storage.objectCreator) on the Cloud Storage buckets that hold the input and output files.roles/storage.objectAdmin
-
MCP Server Connection: If BigQuery Migration Service (BQMS) tools (e.g.,
translate_query) are not present in your active tool list, configure the remote BigQuery Migration Service (BQMS) MCP server. For setup instructions across clients (Claude, Cursor, Gemini CLI, VS Code), see mcp_config.md.translate_batch_queries -
Cost & Cloud Storage Cleanup: Review Google Cloud Pricing for Cloud Storage and BigQuery usage details. After completing a batch migration, clean up temporary staged input and output folders in Cloud Storage:bash
gcloud storage rm -r gs://{bucket}/{folder} -
Translation Tracking: For batch translations, and for queries that need more than one translation iteration, keep a translation tracker (e.g.) to document goals, planned steps, and progress. After each translation, record one row with this template:
translation_tracker.mdIteration Status Total Errors Correctness Errors Completeness Errors Compatibility Errors Key Actions / Transformations A single query that translates cleanly on the first attempt needs no tracker: report its translation ID and logs in your response.
在翻译查询或 schema 之前,请验证以下先决条件:
-
Google Cloud SDK 与项目设置:确保已安装 Google Cloud SDK,验证 Google Cloud project 已关联处于启用状态的 Cloud Billing account,并选择目标 project:bash
gcloud config set project {project_id} -
启用 API:在 Google Cloud project 中启用 BigQuery Migration API():
bigquerymigration.googleapis.combashgcloud services enable bigquerymigration.googleapis.com -
必需的 IAM Roles:验证你的身份或 service account 是否拥有:
- (MCP Tool User)
roles/mcp.toolUser - (Migration Workflow Editor)
roles/bigquerymigration.editor - 对于 batch translation,需要对存放输入和输出文件的 Cloud Storage buckets 拥有 和
roles/storage.objectViewer(或roles/storage.objectCreator)。roles/storage.objectAdmin
-
MCP Server 连接:如果当前 tool 列表中没有 BigQuery Migration Service (BQMS) tools(例如、
translate_query),请配置远程 BigQuery Migration Service (BQMS) MCP server。有关各客户端(Claude、Cursor、Gemini CLI、VS Code)的设置说明,请参阅 mcp_config.md。translate_batch_queries -
成本与 Cloud Storage 清理:请查看 Google Cloud Pricing,了解 Cloud Storage 和 BigQuery 的使用详情。完成 batch migration 后,清理 Cloud Storage 中临时暂存的输入和输出文件夹:bash
gcloud storage rm -r gs://{bucket}/{folder} -
翻译跟踪:对于 batch translations,以及需要多次翻译迭代的 queries,请维护一个 translation tracker(例如),记录目标、计划步骤和进度。每次翻译后,使用以下模板记录一行:
translation_tracker.mdIteration Status Total Errors Correctness Errors Completeness Errors Compatibility Errors Key Actions / Transformations 如果单条查询首次翻译即顺利完成,则无需 tracker:在回复中报告其 translation ID 和 logs。
Workflow Selection
工作流选择
Choose the workflow based on your migration scope:
- Interactive Translation (Single Query): Best for ad-hoc queries, exploratory testing, dialect syntax comparison, or quick error debugging.
- Batch Translation (Multiple Files & Cloud Storage): Use for more than
one file or a directory, including SQL embedded in application code (extract
it into files first). For multi-file Python codebases containing embedded SQL, use python_embedded_sql_migrator_template.py to extract queries into staged files and re-inject translated SQL. Batch translation is faster and cheaper than translating files one by one, and it resolves cross-file dependencies, so it usually gives better results. Run
.sqlon the staged files first; usetranslate_batch_queriesonly afterwards if needed to debug a specific statement fromtranslate_query.translation_report.csv
根据迁移范围选择工作流:
- 交互式翻译(单条查询):最适合临时查询、探索性测试、dialect 语法比较或快速错误调试。
- Batch Translation(多个文件与 Cloud Storage):适用于多个文件或目录,包括嵌在应用程序代码中的 SQL(请先将其提取到 文件中)。对于包含嵌入式 SQL 的多文件 Python codebase,请使用 python_embedded_sql_migrator_template.py 将查询提取到暂存文件中,并重新注入翻译后的 SQL。Batch translation 比逐个翻译文件更快、更便宜,而且能够解决跨文件依赖,因此通常能获得更好的结果。首先对暂存文件运行
.sql;仅在之后需要调试translate_batch_queries中的特定语句时,才使用translation_report.csv。translate_query
1. Interactive Translation Workflow
1. 交互式翻译工作流
Use the interactive workflow to translate a single query string, inspect
compiler logs, and iteratively resolve issues.
使用交互式工作流翻译单条 query string、检查编译器日志,并迭代解决问题。
Steps
步骤
-
Submit Query (): Call
translate_querywithtranslate_query,projectNumber,location,inputQuery, andsourceDialect(targetDialect). Include optional parameters:BigQuery- : Cloud Storage URI to a metadata zip file (
metadataFilePath) for schema resolution.gs://.../metadata.zip - : Inline YAML configurations (e.g. object renaming, schema redirection, optimizer rules). Show the YAML you used in your response so the user can reuse it (e.g. in batch translation).
translationConfigs
-
Poll Status (): If the returned
get_translationistranslation_stateorRUNNING, pollPENDINGusing theget_translationID,translation, andprojectNumberuntillocationistranslation.stateorSUCCEEDED, at the interval given in the BigQuery Migration Service (BQMS) MCP tool descriptions. While the job runs, do work that doesn't need the result (e.g. read the dialect gotchas reference).FAILED -
Inspect Output & Logs:
- Examine for the converted GoogleSQL.
translated_query - Review each entry in : inspect
translation_logs(severity,ERROR),WARNING(e.g.category), andRelationNotFound(effect,CORRECTNESS,COMPLETENESS).COMPATIBILITY - Check the SQL itself, not only the logs: BigQuery Migration Service
(BQMS) emits for types it cannot map, sometimes with no log entry. Treat any
ERROR_TYPE(...)or inline error comment as an error, and when a BigQuery project is available, dry-run the result (ERROR_TYPE). Abq query --dry_run --use_legacy_sql=falseorNot founderror still means the SQL is syntactically valid; the referenced objects just don't exist in BigQuery yet. Say so and move on, even if the error suggests a fix: don't stub tables or change the SQL ormust be qualified with a datasetflags to get past it.bq - A error means BigQuery Migration Service (BQMS) copied that call unchanged, sometimes with the function name in backticks. Treat it as untranslated even if BigQuery has a function with that name: a dry run that stops at missing tables doesn't check it.
NoSuchFunction - If the source dialect has a gotchas reference (see Dialect-Specific Gotchas), read it and check the output for the patterns it lists. Some semantic differences produce no log entry and still pass a dry run.
- Examine
-
Iterate Until Clean: If the logs contain errors or warnings, the translated SQL containsor inline error comments, the dry run fails, or the output matches a dialect gotcha, resolve the issues as described in Resolving Translation Issues and re-translate.
ERROR_TYPE -
Explain & Compare Differences (): When requested to explain a translation, compare syntax differences, or understand how specific query logic was converted, call
explain_translationwithexplain_translation(the translation ID),translation, andprojectNumber. This returns a structured explanation of the query's purpose and a detailed breakdown of syntax, function, and behavioral differences between the source query and the translated GoogleSQL. The explanation doesn't validate the output, so check the translation as in step 3 rather than relying on it.location
-
提交查询():调用
translate_query,传入translate_query、projectNumber、location、inputQuery和sourceDialect(targetDialect)。可选择包含以下参数:BigQuery- :metadata zip 文件的 Cloud Storage URI(
metadataFilePath),用于解析 schema。gs://.../metadata.zip - :内联 YAML 配置(例如对象重命名、schema 重定向、optimizer rules)。在回复中展示你使用的 YAML,以便用户复用(例如用于 batch translation)。
translationConfigs
-
轮询状态():如果返回的
get_translation为translation_state或RUNNING,请使用 translation ID、PENDING和projectNumber轮询location,直至get_translation变为translation.state或SUCCEEDED;轮询间隔应遵循 BigQuery Migration Service (BQMS) MCP tool descriptions 中的规定。作业运行期间,可处理不需要结果的工作(例如阅读 dialect gotchas reference)。FAILED -
检查输出与日志:
- 检查 中的转换后 GoogleSQL。
translated_query - 审查 中的每一条记录:检查
translation_logs(severity、ERROR)、WARNING(例如category)和RelationNotFound(effect、CORRECTNESS、COMPLETENESS)。COMPATIBILITY - 不仅要检查日志,还要检查 SQL 本身:BigQuery Migration Service (BQMS) 会为无法映射的类型输出 ,有时不会生成日志条目。将任何
ERROR_TYPE(...)或内联错误注释都视为错误;如果有可用的 BigQuery project,请对结果执行 dry-run(ERROR_TYPE)。bq query --dry_run --use_legacy_sql=false或Not found错误仍表示 SQL 在语法上有效,只是引用的对象尚不存在于 BigQuery 中。即使错误提示了修复方法,也要说明这一点并继续操作:不要创建 stub tables,也不要修改 SQL 或must be qualified with a datasetflags 来绕过错误。bq - 错误表示 BigQuery Migration Service (BQMS) 原样复制了该调用,有时函数名会带有反引号。即使 BigQuery 中存在同名函数,也应将其视为未转换:因缺少 tables 而停止的 dry-run 并不会检查该函数。
NoSuchFunction - 如果 source dialect 有 gotchas reference(参见 Dialect-Specific Gotchas),请阅读该文档并检查输出中是否出现其列出的模式。部分语义差异不会生成日志条目,但仍能通过 dry-run。
- 检查
-
迭代直至无错误:如果 logs 中包含 errors 或 warnings、翻译后的 SQL 包含或内联错误注释、dry-run 失败,或者输出与某个 dialect gotcha 匹配,请按照 Resolving Translation Issues 中所述的方法解决问题并重新翻译。
ERROR_TYPE -
解释并比较差异():当用户要求解释翻译结果、比较语法差异,或了解特定 query logic 的转换方式时,使用
explain_translation(translation ID)、translation和projectNumber调用location。它会返回对查询用途的结构化说明,以及 source query 与翻译后 GoogleSQL 在语法、函数和行为方面的详细差异分解。该说明不会验证输出,因此仍需按照步骤 3 检查翻译结果,而不能依赖该说明。explain_translation
2. Batch Translation Workflow
2. Batch Translation 工作流
Batch translation handles multiple files staged in Cloud Storage. It resolves
cross-file dependencies and produces a consolidated translation summary report
().
translation_report.csvBatch translation 处理暂存在 Cloud Storage 中的多个文件。它能够解决跨文件依赖,并生成统一的翻译摘要报告()。
translation_report.csvSteps
步骤
-
Stage Input Files to Cloud Storage: Upload the raw SQL files, configuration files (), and any metadata file from the local workspace to a Cloud Storage input folder:
*.config.yamlbashgcloud storage cp input/*.sql gs://my-bucket/migration/input/ gcloud storage cp *.config.yaml gs://my-bucket/migration/input/Note: Do not pass configuration file paths in tool parameters if those files are already located inside.sourceBaseUriUse new, empty Cloud Storage input and output folders for each run (e.g. with a timestamp suffix). Stage only files from the local workspace or generated in this run. Never reuse or copy inputs, DDL, or outputs that are already in a bucket from earlier runs: they may come from different inputs or settings. -
Initiate Batch Translation:
- Translating SQL Queries (): Call
translate_batch_querieswithtranslate_batch_queries,projectNumber,location,sourceDialect,targetDialect, andsourceBaseUri.targetBaseUri - Translating Metadata Zip Files (): To aid batch translation by generating target BigQuery DDL schema definitions from an extracted database metadata zip file (
translate_metadatafrom Teradata, Oracle, etc.), callmetadata.zipspecifyingtranslate_metadata(Cloud Storage URI tosourceBaseUri) andmetadata.zip(destination Cloud Storage folder).targetBaseUri
- Translating SQL Queries (
-
Poll Completion (): Poll using the returned workflow ID until
fetch_batch_translationistranslation.stateorSUCCEEDED, at the interval given in the BigQuery Migration Service (BQMS) MCP tool descriptions.FAILED -
Retrieve & Inspect Outputs: Download the generated files from:
translation.target_base_uribashgcloud storage cp -r gs://my-bucket/migration/output/ ./local_output/Locate and inspectto review all warnings, errors, and line-level diagnostics. Then scan the generatedreport/translation_report.csvfiles for.sqland inline error comments, which the report can miss, and dry-run the generated SQL when a BigQuery project is available (handleERROR_TYPE(...)errors as in the interactive workflow). Check the output for the patterns in the source dialect's gotchas reference, if one exists. Resolve or clearly report every remaining issue; do not describe the output as clean until this check passes.Not found -
Iterate Until Clean: Resolve the issues found in step 4 as described in Resolving Translation Issues, and re-run from step 1 with the changed inputs.
-
Post-processing (Final Step Only): Only after resolving every compiler error that re-translation can fix, execute any application-level post-processing (e.g. AST re-injection) to restore host source files. effect_correctness.md lists the errors that may remain when re-injecting embedded SQL.
-
将输入文件暂存到 Cloud Storage:将本地 workspace 中的原始 SQL 文件、配置文件()及任何 metadata 文件上传到 Cloud Storage 输入文件夹:
*.config.yamlbashgcloud storage cp input/*.sql gs://my-bucket/migration/input/ gcloud storage cp *.config.yaml gs://my-bucket/migration/input/注意:如果配置文件已经位于中,请勿在 tool 参数中传递这些文件路径。sourceBaseUri每次运行都应使用新的空 Cloud Storage 输入和输出文件夹(例如添加 timestamp 后缀)。只暂存来自本地 workspace 或在本次运行中生成的文件。切勿复用或复制 bucket 中早期运行留下的 inputs、DDL 或 outputs:它们可能来自不同的输入或设置。 -
启动 Batch Translation:
- 翻译 SQL Queries():调用
translate_batch_queries,传入translate_batch_queries、projectNumber、location、sourceDialect、targetDialect和sourceBaseUri。targetBaseUri - 翻译 Metadata Zip Files():若要通过从解压后的数据库 metadata zip 文件(来自 Teradata、Oracle 等的
translate_metadata)生成目标 BigQuery DDL schema definitions 来辅助 batch translation,请调用metadata.zip,并指定translate_metadata(sourceBaseUri的 Cloud Storage URI)和metadata.zip(目标 Cloud Storage 文件夹)。targetBaseUri
- 翻译 SQL Queries(
-
轮询完成状态():使用返回的 workflow ID 轮询,直至
fetch_batch_translation变为translation.state或SUCCEEDED;轮询间隔应遵循 BigQuery Migration Service (BQMS) MCP tool descriptions 中的规定。FAILED -
检索并检查输出:从下载生成的文件:
translation.target_base_uribashgcloud storage cp -r gs://my-bucket/migration/output/ ./local_output/找到并检查,以审查所有 warnings、errors 和行级诊断信息。然后扫描生成的report/translation_report.csv文件,查找.sql和内联错误注释,这些问题可能会遗漏在报告之外;如果有可用的 BigQuery project,请对生成的 SQL 执行 dry-run(按照交互式工作流处理ERROR_TYPE(...)错误)。如果 source dialect 有 gotchas reference,请检查输出中是否出现其中的模式。解决所有遗留问题,或明确报告这些问题;在检查通过之前,不要将输出描述为无错误。Not found -
迭代直至无错误:按照步骤 4 和 Resolving Translation Issues 中所述的方法解决发现的问题,并从步骤 1 开始,使用修改后的输入重新运行。
-
后处理(仅最后一步):只有在解决所有可通过重新翻译修复的编译错误后,才能执行应用程序级后处理(例如 AST re-injection),以恢复宿主源文件。effect_correctness.md 列出了重新注入嵌入式 SQL 时可能遗留的错误。
Resolving Translation Issues
解决翻译问题
Both workflows resolve issues the same way:
- Identify the Effect (,
CORRECTNESS,COMPLETENESS) of each issue, and resolve them in order of compiler phase dependencies:COMPATIBILITY(parse errors) first, thenCORRECTNESS(missing objects), thenCOMPLETENESS(unsupported features).COMPATIBILITY - MUST Read the Reference Guide for each effect in Troubleshooting by Error Effect BEFORE applying fixes, and follow its resolution workflow.
- Fixes can preprocess the input (e.g. prepend DDL) or post-process the BigQuery Migration Service (BQMS) output (e.g. a rewrite that a dialect gotcha calls for). Run BigQuery Migration Service (BQMS) on the preprocessed input rather than translating it yourself.
- Keep the source system's dialect for every re-translation. Another dialect may accept the same SQL but translate it with different semantics (e.g. return types, numeric types, or identifier case), so don't switch dialects to get cleaner output.
- Rewrites of the BigQuery Migration Service (BQMS) output, including
optimizations the user asks for, must return the same rows as the source
query. For example, a filter that runs after a row limit (,
TOP) can't be merged into the query before the limit.LIMIT - Wait for each re-translation to finish, record its translation ID, error counts, and any pre- or post-processing in the tracker, and compare the counts with the previous iteration. Repeat until no errors remain or all options are exhausted.
- Report the final SQL from the tracked iteration, the pre- and post-processing you applied, and any errors that remain.
两种工作流使用相同的方法解决问题:
- 确定每个问题的 Effect(、
CORRECTNESS、COMPLETENESS),并按照编译器阶段的依赖顺序解决:先处理COMPATIBILITY(解析错误),然后处理CORRECTNESS(缺失对象),最后处理COMPLETENESS(不支持的功能)。COMPATIBILITY - 在应用修复之前,必须阅读 Troubleshooting by Error Effect 中每种 effect 对应的 Reference Guide,并遵循其中的解决工作流。
- 修复操作可以预处理输入(例如在开头添加 DDL),也可以后处理 BigQuery Migration Service (BQMS) 输出(例如按照 dialect gotcha 重写内容)。请让 BigQuery Migration Service (BQMS) 处理预处理后的输入,而不是自行翻译。
- 每次重新翻译时都应保留 source system 的 dialect。另一种 dialect 可能接受相同的 SQL,却以不同语义进行翻译(例如返回类型、数字类型或标识符大小写),因此不要为了让输出更简洁而切换 dialect。
- 对 BigQuery Migration Service (BQMS) 输出进行的重写,包括用户要求的 optimizations,必须返回与 source query 相同的行。例如,在行数限制(、
TOP)之后执行的 filter 不能在限制之前合并到查询中。LIMIT - 等待每次重新翻译完成,在 tracker 中记录其 translation ID、错误数量以及所有前处理或后处理操作,并将这些数量与上一次迭代进行比较。重复此过程,直至没有错误剩余或所有可行方案均已用尽。
- 报告最终追踪迭代生成的 SQL、所应用的前处理和后处理,以及仍然存在的任何错误。
Reference Lookup Index
Reference Lookup Index
Use these reference guides when troubleshooting errors or authoring
configuration files:
排查错误或编写配置文件时,请使用以下 reference guides:
Troubleshooting by Error Effect
按错误影响排查问题
| Effect | Common Causes | Reference Guide |
|---|---|---|
| Syntax errors, unescaped literals, application script wrappers, | effect_correctness.md |
| Missing tables, views, columns ( | effect_completeness.md |
| Functions or features without direct GoogleSQL mapping. | effect_compatibility.md |
| Effect | 常见原因 | Reference Guide |
|---|---|---|
| 语法错误、未转义的字面量、应用程序脚本包装、 | effect_correctness.md |
| 缺失的 tables、views、columns( | effect_completeness.md |
| 没有直接 GoogleSQL 映射的 functions 或 features。 | effect_compatibility.md |
Configuration & Setup Guides
配置与设置指南
| Task | Description | Reference Guide |
|---|---|---|
| YAML Configurations | Database/schema redirection, name mapping, table partitioning and clustering injection, optimizer transformations, statement rewriters, and environment settings. Schemas in | translation_yaml_config.md |
| MCP Server Setup | Configuring the remote BigQuery Migration Service (BQMS) MCP server, OAuth scopes, and IAM roles. | mcp_config.md |
| 任务 | 说明 | Reference Guide |
|---|---|---|
| YAML Configurations | Database/schema 重定向、名称映射、table partitioning 和 clustering 注入、optimizer transformations、statement rewriters 以及 environment settings。Schemas 位于 | translation_yaml_config.md |
| MCP Server Setup | 配置远程 BigQuery Migration Service (BQMS) MCP server、OAuth scopes 和 IAM roles。 | mcp_config.md |
Dialect-Specific Gotchas
Dialect-Specific Gotchas
| Dialect | Key Considerations | Reference Guide |
|---|---|---|
| Teradata | SET/MULTISET tables, | teradata_gotchas.md |
| Oracle | PL/SQL procedures, | oracle_gotchas.md |
| Snowflake | Dynamic templates, | snowflake_gotchas.md |
| Dialect | 关键注意事项 | Reference Guide |
|---|---|---|
| Teradata | SET/MULTISET tables、 | teradata_gotchas.md |
| Oracle | PL/SQL procedures、 | oracle_gotchas.md |
| Snowflake | Dynamic templates、 | snowflake_gotchas.md |