bigquery-observability
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseBigQuery Observability
BigQuery 可观测性
Tool Selection
工具选择
<!-- mdformat off -->
<!-- mdformat on -->
| Tool | Primary Use Cases | Strengths & Capabilities | When to Avoid / Limitations |
|---|---|---|---|
| Historical analysis, cohort comparison ( | Flexible SQL querying across | Avoid for high-frequency real-time polling or single-job point-lookups (can consume slots and take seconds to execute). |
REST API ( | Single-job point-lookup, real-time stage bottleneck diagnosis, automated pipeline status checks, reservation/capacity commitment configuration inspection ( | Zero-SQL overhead, fast REST/CLI point-lookups ( | Avoid for aggregate analysis across thousands of jobs, cross-project historical comparison, or system timeline aggregations. |
| Real-time alerting, fleet-wide dashboards, continuous slot utilization tracking, high-level SLA/SLO monitoring. | Out-of-the-box charts for slot utilization, query throughput, | Avoid for SQL-level debugging, individual query text inspection, or stage-level execution detail. |
<!-- mdformat off -->
<!-- mdformat on -->
| 工具 | 主要用例 | 优势与功能 | 避免使用场景/限制 |
|---|---|---|---|
| 历史分析、队列比较( | 支持跨 | 避免用于高频实时轮询或单作业点查询(会消耗槽位,执行可能需要数秒)。 |
REST API ( | 单作业点查询、实时阶段瓶颈诊断、自动化管道状态检查、预留/容量承诺配置检查( | 零SQL开销,快速REST/CLI点查询( | 避免用于跨数千个作业的聚合分析、跨项目历史比较或系统时间线聚合。 |
| 实时告警、全集群仪表盘、持续槽位利用率跟踪、高层级SLA/SLO监控。 | 开箱即用的槽位利用率、查询吞吐量、 | 避免用于SQL级调试、单个查询文本检查或阶段级执行详情分析。 |
Prerequisites & Environment Setup
前提条件与环境设置
Before retrieving telemetry or running observability queries, ensure the
Google Cloud environment and project are configured:
-
Google Cloud SDK: Ensure the Google Cloud SDK is installed and configured.
-
Project Selection: Set the active Google Cloud project:bash
gcloud config set project {project_id} -
API Enablement: Ensure the BigQuery and Cloud Monitoring APIs are enabled:bash
gcloud services enable bigquery.googleapis.com monitoring.googleapis.com -
Authentication: Authenticate the environment:
- CLI queries and commands:
bqgcloud auth login - SDKs and automated client tools:
gcloud auth application-default login - Service accounts: Set
GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
- CLI queries and
-
Billing & IAM Roles:
- Verify an active Google Cloud Billing account is attached to
.
{project_id} - Ensure appropriate IAM roles:
- : Running telemetry queries.
roles/bigquery.jobUser - or
roles/bigquery.resourceViewer: Organization-level jobs and reservation telemetry.roles/bigquery.admin - : Cloud Monitoring metrics.
roles/monitoring.viewer
- Verify an active Google Cloud Billing account is attached to
在检索遥测数据或运行可观测性查询之前,请确保Google Cloud环境和项目已配置完成:
-
Google Cloud SDK:确保已安装并配置Google Cloud SDK。
-
项目选择:设置当前活跃的Google Cloud项目:bash
gcloud config set project {project_id} -
API启用:确保已启用BigQuery和Cloud Monitoring API:bash
gcloud services enable bigquery.googleapis.com monitoring.googleapis.com -
身份认证:为环境进行身份认证:
- CLI查询和命令:
bqgcloud auth login - SDK和自动化客户端工具:
gcloud auth application-default login - 服务账号:设置
GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
- CLI查询和
-
计费与IAM角色:
- 确认已绑定有效的Google Cloud计费账号。
{project_id} - 确保拥有适当的IAM角色:
- :运行遥测查询。
roles/bigquery.jobUser - 或
roles/bigquery.resourceViewer:组织级作业和预留遥测。roles/bigquery.admin - :查看Cloud Monitoring指标。
roles/monitoring.viewer
- 确认
Workflow
工作流
-
Single-Job Point-Lookup (Zero-SQL Overhead): For single-job slowness or inspection, always prioritize the REST API or CLI () first. It provides zero-SQL overhead and fast point-lookups for internal stage bottlenecks (
bq show -j,performanceInsights, shuffle spill).queryPlanbashbq show --location={location} -j {project_id}:{job_id} -
Diagnostic Transition Logic: If no job-level issues are found (e.g. no clear internal bottlenecks), the investigation should transition to system-levelqueries (such as
INFORMATION_SCHEMAorJOBS_TIMELINE) to check for broader issues like slot contention, queueing delay, or noisy neighbors.RESERVATIONS_TIMELINE
-
单作业点查询(零SQL开销): 对于单作业变慢或检查场景,始终优先使用REST API或CLI()。它提供零SQL开销的快速点查询,可定位内部阶段瓶颈(
bq show -j、performanceInsights、shuffle溢出)。queryPlanbashbq show --location={location} -j {project_id}:{job_id} -
诊断转换逻辑: 如果未发现作业级问题(例如没有明确的内部瓶颈),应将调查转向系统级查询(例如
INFORMATION_SCHEMA或JOBS_TIMELINE),以检查更广泛的问题,如槽位争用、排队延迟或噪声邻居。RESERVATIONS_TIMELINE
Best Practices for Writing INFORMATION_SCHEMA
Queries
INFORMATION_SCHEMA编写INFORMATION_SCHEMA
查询的最佳实践
INFORMATION_SCHEMAEvery query against a BigQuery view must be qualified with
either a region qualifier or a dataset qualifier, optionally prefixed by
a project qualifier.
INFORMATION_SCHEMA针对BigQuery 视图的每个查询都必须使用区域限定符或数据集限定符进行限定,可选择添加项目限定符作为前缀。
INFORMATION_SCHEMAQualification Syntax & Scope Matching
限定语法与范围匹配
-
Region-Qualified Syntax:googlesql
`{project_id}`.`region-{region}`.INFORMATION_SCHEMA.{view}Example:`my-project`.`region-us`.INFORMATION_SCHEMA.JOBSApplies to: Regional telemetry views (,JOBS*,JOBS_TIMELINE*,RESERVATIONS*,CAPACITY_COMMITMENTS*,TABLE_STORAGE*). The client query execution location MUST match theSTREAMING_TIMELINE*qualifier (or BigQuery throws:region-{region}).Not found: Table {project_id}:region-{region}.INFORMATION_SCHEMA.{view} was not found in location {location} -
Dataset-Qualified Syntax:googlesql
`{project_id}`.`{dataset_id}`.INFORMATION_SCHEMA.{view}Example:`my-project`.`analytics`.INFORMATION_SCHEMA.TABLESApplies to: Dataset-scoped views (,PARTITIONS,SEARCH_INDEXES*). Never useROW_ACCESS_POLICIESwith dataset views.region- -
Dual-Scoped Views: Views like,
TABLES,COLUMNS,COLUMN_FIELD_PATHS,VIEWS, andROUTINEScan be qualified with eitherVECTOR_INDEXESor{dataset_id}depending on whether dataset or region-wide analysis is required.region-{region} -
Project Qualifier (): Optional. If omitted, queries default to the project in which the query is executing. Specifying a project qualifier on organization-level views (e.g.
{project_id}) has no impact on results.JOBS_BY_ORGANIZATION
-
区域限定语法:googlesql
`{project_id}`.`region-{region}`.INFORMATION_SCHEMA.{view}示例:`my-project`.`region-us`.INFORMATION_SCHEMA.JOBS适用范围: 区域级遥测视图(、JOBS*、JOBS_TIMELINE*、RESERVATIONS*、CAPACITY_COMMITMENTS*、TABLE_STORAGE*)。客户端查询执行位置必须与STREAMING_TIMELINE*限定符匹配(否则BigQuery会报错:region-{region})。Not found: Table {project_id}:region-{region}.INFORMATION_SCHEMA.{view} was not found in location {location} -
数据集限定语法:googlesql
`{project_id}`.`{dataset_id}`.INFORMATION_SCHEMA.{view}示例:`my-project`.`analytics`.INFORMATION_SCHEMA.TABLES适用范围: 数据集级视图(、PARTITIONS、SEARCH_INDEXES*)。数据集视图切勿使用ROW_ACCESS_POLICIES前缀。region- -
双范围视图: 像、
TABLES、COLUMNS、COLUMN_FIELD_PATHS、VIEWS和ROUTINES这类视图,可以根据需要使用VECTOR_INDEXES或{dataset_id}进行限定,分别对应数据集级或区域级分析。region-{region} -
项目限定符(): 可选。如果省略,查询默认使用当前执行查询的项目。对组织级视图(例如
{project_id})指定项目限定符不会影响查询结果。JOBS_BY_ORGANIZATION
Principle of Least Privilege & Scope Selection
最小权限原则与范围选择
When constructing queries, always select the scope and
view variant with the least IAM permission requirement that satisfies the
analytical need:
INFORMATION_SCHEMA- User-Level over Project-Level (): When diagnosing queries or sessions executed by the current user, use
_BY_USER(e.g._BY_USER,JOBS_BY_USER). This requires onlySESSIONS_BY_USER(granted viabigquery.jobs.listorroles/bigquery.user), avoiding the need forroles/bigquery.jobUserorbigquery.jobs.listAll.roles/bigquery.admin - Dataset-Level over Region/Project-Level: When querying table metadata,
columns, or views for a specific dataset, qualify with rather than
{dataset_id}when project-level metadata access is restricted. Dataset-scoped queries require permissions only on that target dataset.region-{region} - Project-Level over Org/Folder-Level (): Always start with project-scoped views before escalating to
_BY_PROJECTor_BY_FOLDER. Folder and organization queries require broad folder/org IAM permissions (_BY_ORGANIZATIONorbigquery.jobs.listAllat the Org/Folder node).bigquery.tables.list - Metadata Roles over Data Roles: For table and storage introspection,
prefer (which provides
roles/bigquery.metadataViewerandbigquery.tables.get) overbigquery.tables.listorroles/bigquery.dataViewerwhen data read access (roles/bigquery.dataOwner) is not needed. (Note:bigquery.tables.getDatauniquely requiresINFORMATION_SCHEMA.PARTITIONS).bigquery.tables.getData
构建查询时,始终选择满足分析需求所需IAM权限最少的范围和视图变体:
INFORMATION_SCHEMA- 用户级优先于项目级(): 诊断当前用户执行的查询或会话时,使用
_BY_USER后缀的视图(例如_BY_USER、JOBS_BY_USER)。这仅需要SESSIONS_BY_USER权限(通过bigquery.jobs.list或roles/bigquery.user授予),无需roles/bigquery.jobUser或bigquery.jobs.listAll权限。roles/bigquery.admin - 数据集级优先于区域/项目级: 查询特定数据集的表元数据、列或视图时,如果项目级元数据访问受限,请使用限定而非
{dataset_id}。数据集级查询仅需要目标数据集上的权限。region-{region} - 项目级优先于组织/文件夹级(): 始终先从项目级视图开始,再升级到
_BY_PROJECT或_BY_FOLDER级视图。文件夹和组织级查询需要广泛的文件夹/组织IAM权限(组织/文件夹节点上的_BY_ORGANIZATION或bigquery.jobs.listAll权限)。bigquery.tables.list - 元数据角色优先于数据角色: 对于表和存储内省操作,优先选择(提供
roles/bigquery.metadataViewer和bigquery.tables.get权限)而非bigquery.tables.list或roles/bigquery.dataViewer,前提是不需要数据读取权限(roles/bigquery.dataOwner)。(注意:bigquery.tables.getData是个例外,它需要INFORMATION_SCHEMA.PARTITIONS权限。)bigquery.tables.getData
Execution Guardrails & Query Invariants
执行防护与查询不变式
- Mandatory Partition & Time Filtering: Always filter on (e.g.,
creation_time) orcreation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 DAY)to avoid full metadata table scans.usage_date - Script Wrapper Exclusion: Add when aggregating compute spend to avoid double-counting parent scripts and child jobs.
AND (statement_type != 'SCRIPT' OR statement_type IS NULL) - Column Pruning: Never use against
SELECT *; only project required columns.INFORMATION_SCHEMA - Dry Run & Cost Estimation: Use a dry run (or API
bq query --dry_run --use_legacy_sql=false "{query}") before executing complex queries, multi-view joins, or large scans to validate syntax and estimatedryRun=trueat zero cost.totalBytesProcessed - Empty Regional Scope (0 Rows): If the execution location matches the qualifier, but the project has no datasets or jobs in that region, the query succeeds and returns 0 rows. Never assume 0 rows means 0 usage—always verify the target dataset locations.
- Non-Hierarchical Region Scope: Region qualifiers are not hierarchical.
Multi-regions do not encompass single regions (e.g. returns only multi-region
region-usmetadata and does not include single regions likeUS).region-us-central1 - No Multi-Region Aggregation in SQL: Region qualifiers cannot be joined
cross-region in a single query (e.g. cannot join
region-us).region-eu - Uncached Execution & Minimum Scan Size: query results are never cached. On-demand queries incur a minimum of 10 MB of data processing charges per execution.
INFORMATION_SCHEMA
- 强制分区与时间过滤: 始终按(例如
creation_time)或creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 DAY)进行过滤,避免全量元数据表扫描。usage_date - 排除脚本包装器: 聚合计算开销时添加,避免重复计算父脚本和子作业。
AND (statement_type != 'SCRIPT' OR statement_type IS NULL) - 列裁剪: 切勿对使用
INFORMATION_SCHEMA;仅投影所需的列。SELECT * - 试运行与成本估算: 执行复杂查询、多视图连接或大范围扫描前,使用试运行(或API的
bq query --dry_run --use_legacy_sql=false "{query}"参数)来验证语法并零成本估算dryRun=true。totalBytesProcessed - 空区域范围(0行结果): 如果执行位置与限定符匹配,但项目在该区域没有数据集或作业,查询会成功执行并返回0行。切勿假设0行意味着0使用量——务必验证目标数据集的位置。
- 非层级化区域范围: 区域限定符不具有层级关系。多区域不包含单区域(例如仅返回多区域
region-us的元数据,不包含US等单区域)。region-us-central1 - SQL中不支持跨区域聚合: 单个查询中不能跨区域连接不同的区域限定符(例如不能连接
region-us)。region-eu - 无缓存执行与最小扫描大小: 查询结果永远不会被缓存。按需查询每次执行至少产生10 MB的数据处理费用。
INFORMATION_SCHEMA
Domain References & SQL Queries
领域参考与SQL查询
Telemetry Query Guides
遥测查询指南
- On-Demand Compute: Billed Bytes
(): Authoritative Golden CTE (
references/compute_ondemand_billable.md), timezone-aligned billing date extraction (PST8PDT), BQML CREATE_MODEL 50x multiplier rules, script wrapper deduplication, and row-level security (RLS) masking checks.bytes_billed_cte - Capacity Compute: Billable Slots & Commitments
(): Query templates for auditing billable capacity hours across 1-Year/3-Year commitments, uncovered baseline PAYG slots, and dynamic autoscaling hours.
references/compute_capacity_billable.md - Storage Footprints & Usage (Bytes Stored)
(): Storage snapshot queries, compression ratio calculations, Time Travel / Fail-Safe churn, daily average GiB time-integrals, and billing model evaluation.
references/storage_footprints.md
- 按需计算:计费字节数():权威的黄金CTE(
references/compute_ondemand_billable.md)、与时区对齐的计费日期提取(PST8PDT)、BQML CREATE_MODEL 50倍乘数规则、脚本包装器去重、行级安全(RLS)掩码检查。bytes_billed_cte - 容量计算:计费槽位与承诺():用于审计跨1年/3年承诺、未覆盖的基准PAYG槽位和动态自动扩缩容小时数的计费容量小时数的查询模板。
references/compute_capacity_billable.md - 存储占用与使用情况(存储字节数)():存储快照查询、压缩率计算、时间旅行/故障安全变动量、日均GiB时间积分、计费模型评估。
references/storage_footprints.md
Performance & Troubleshooting Guides
性能与故障排查指南
- Job Performance Queries (): Queries for evaluating individual and aggregate job performance, stage bottleneck flags, comparable jobs via normalized literals (
references/job_performance_queries.md), BI Engine acceleration, metadata cache (cmeta) acceleration, and execution variance outliers.query_info.query_hashes.normalized_literals - Resource Contention Queries
(): Queries for diagnosing slot contention, queue latency, per-minute concurrency/queue timelines, and 1-second reservation slot saturation.
references/resource_contention_queries.md - Capacity & Configuration Queries
(): Queries for evaluating second-by-second baseline/max capacity ceilings, autoscaling saturation timelines, and auditing configuration changes (
references/capacity_and_configuration_queries.md,RESERVATION_CHANGES_BY_PROJECT).ASSIGNMENT_CHANGES_BY_PROJECT
- 作业性能查询():用于评估单个和聚合作业性能、阶段瓶颈标记、通过归一化字面量(
references/job_performance_queries.md)查找可比作业、BI Engine加速、元数据缓存(cmeta)加速、执行方差异常值的查询。query_info.query_hashes.normalized_literals - 资源争用查询():用于诊断槽位争用、队列延迟、每分钟并发/队列时间线、1秒级预留槽位饱和度的查询。
references/resource_contention_queries.md - 容量与配置查询():用于评估逐秒基准/最大容量上限、自动扩缩容饱和度时间线、审计配置变更(
references/capacity_and_configuration_queries.md、RESERVATION_CHANGES_BY_PROJECT)的查询。ASSIGNMENT_CHANGES_BY_PROJECT
Schema Dictionaries (Column Definitions & Units)
模式字典(列定义与单位)
- Compute & Capacity Schema Dictionary (): Complete column dictionary, physical units, and least-privilege IAM roles for all compute, job, session, reservation, capacity commitment, and assignment views (
references/schema_compute.md,JOBS*,JOBS_TIMELINE*,SESSIONS_BY_USER,SESSIONS_BY_PROJECT,RESERVATIONS*,RESERVATION_CHANGES*,RESERVATIONS_TIMELINE*,CAPACITY_COMMITMENTS*,CAPACITY_COMMITMENT_CHANGES_BY_PROJECT,ASSIGNMENTS*).ASSIGNMENT_CHANGES_BY_PROJECT - Storage & Data Catalog Schema Dictionary
(): Complete column dictionary, physical units, and least-privilege IAM roles for all table storage, partition, column, snapshot, dataset, constraint, and replication views (
references/schema_storage.md,TABLE_STORAGE*,TABLE_STORAGE_USAGE_TIMELINE*,TABLES*,TABLE_OPTIONS,COLUMNS,COLUMN_FIELD_PATHS,PARTITIONS,VIEWS,MATERIALIZED_VIEWS,TABLE_SNAPSHOTS*,TABLE_CONSTRAINTS,KEY_COLUMN_USAGE,SCHEMATA*,SCHEMATA_OPTIONS,SCHEMATA_REPLICAS*,SCHEMATA_LINKS).SHARED_DATASET_USAGE - Platform, Governance & Ingestion Schema Dictionary
(): Complete column dictionary, physical units, and least-privilege IAM roles for all remaining views including Access Control (
references/schema_others.md,OBJECT_PRIVILEGES,ROW_ACCESS_POLICIES), Streaming Ingestion (ROW_ACCESS_POLICY_OPTIONS,STREAMING_TIMELINE_BY_PROJECT*), Configuration Options (WRITE_API_TIMELINE_BY_PROJECT*,PROJECT_OPTIONS*,EFFECTIVE_PROJECT_OPTIONS,ORGANIZATION_OPTIONS*), Insights & Recommendations (ORGANIZATION_OPTIONS_CHANGES,RECOMMENDATIONS*), and Indexes/BI Engine/Routines (INSIGHTS,SEARCH_INDEXES*,SEARCH_INDEX_COLUMNS,SEARCH_INDEX_OPTIONS,VECTOR_INDEXES*,VECTOR_INDEX_COLUMNS,VECTOR_INDEX_OPTIONS,BI_CAPACITIES,BI_CAPACITY_CHANGES,ROUTINES*,ROUTINE_OPTIONS).PARAMETERS
- 计算与容量模式字典():所有计算、作业、会话、预留、容量承诺和分配视图(
references/schema_compute.md、JOBS*、JOBS_TIMELINE*、SESSIONS_BY_USER、SESSIONS_BY_PROJECT、RESERVATIONS*、RESERVATION_CHANGES*、RESERVATIONS_TIMELINE*、CAPACITY_COMMITMENTS*、CAPACITY_COMMITMENT_CHANGES_BY_PROJECT、ASSIGNMENTS*)的完整列字典、物理单位和最小权限IAM角色。ASSIGNMENT_CHANGES_BY_PROJECT - 存储与数据目录模式字典():所有表存储、分区、列、快照、数据集、约束和复制视图(
references/schema_storage.md、TABLE_STORAGE*、TABLE_STORAGE_USAGE_TIMELINE*、TABLES*、TABLE_OPTIONS、COLUMNS、COLUMN_FIELD_PATHS、PARTITIONS、VIEWS、MATERIALIZED_VIEWS、TABLE_SNAPSHOTS*、TABLE_CONSTRAINTS、KEY_COLUMN_USAGE、SCHEMATA*、SCHEMATA_OPTIONS、SCHEMATA_REPLICAS*、SCHEMATA_LINKS)的完整列字典、物理单位和最小权限IAM角色。SHARED_DATASET_USAGE - 平台、治理与摄入模式字典():所有其余视图的完整列字典、物理单位和最小权限IAM角色,包括访问控制(
references/schema_others.md、OBJECT_PRIVILEGES、ROW_ACCESS_POLICIES)、流摄入(ROW_ACCESS_POLICY_OPTIONS、STREAMING_TIMELINE_BY_PROJECT*)、配置选项(WRITE_API_TIMELINE_BY_PROJECT*、PROJECT_OPTIONS*、EFFECTIVE_PROJECT_OPTIONS、ORGANIZATION_OPTIONS*)、洞察与建议(ORGANIZATION_OPTIONS_CHANGES、RECOMMENDATIONS*)、索引/BI Engine/例程(INSIGHTS、SEARCH_INDEXES*、SEARCH_INDEX_COLUMNS、SEARCH_INDEX_OPTIONS、VECTOR_INDEXES*、VECTOR_INDEX_COLUMNS、VECTOR_INDEX_OPTIONS、BI_CAPACITIES、BI_CAPACITY_CHANGES、ROUTINES*、ROUTINE_OPTIONS)。PARAMETERS