pubsub-basics

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GCP Pub/Sub Basics

GCP Pub/Sub 基础知识

This skill provides safe, repeatable guidance for operating Google Cloud Pub/Sub from the
gcloud
CLI. It covers the core messaging concepts, the IAM roles required to act on resources, step-by-step recipes for creating topics and subscriptions, and the operational guardrails that prevent destructive or ambiguous actions. Follow it whenever you provision, inspect, or operate Pub/Sub infrastructure for event-driven systems, streaming pipelines, or asynchronous service integrations.
本技能提供了通过
gcloud
CLI操作Google Cloud Pub/Sub的安全、可重复的指导。它涵盖了核心消息传递概念、操作资源所需的IAM角色、创建主题和订阅的分步方案,以及防止破坏性或模糊操作的运维防护措施。当你为事件驱动系统、流式管道或异步服务集成配置、检查或操作Pub/Sub基础设施时,请遵循本指南。

Overview

概述

Google Cloud Pub/Sub is a fully managed, asynchronous messaging service that decouples message producers from message consumers. Producers publish messages without knowing who consumes them, and consumers receive messages without knowing who produced them. This decoupling enables resilient, horizontally scalable, event-driven architectures.
Google Cloud Pub/Sub是一款全托管的异步消息服务,可将消息生产者与消息消费者解耦。生产者无需知晓谁会消费消息即可发布消息,消费者也无需知晓消息来源即可接收消息。这种解耦特性支持构建弹性、可水平扩展的事件驱动架构。

Core concepts

核心概念

  • Topic — A named resource to which producers (publishers) send messages. A topic is the entry point for data flowing into the system. Topics may optionally enforce a schema on incoming messages.
  • Subscription — A named resource representing a stream of messages from a single topic to be delivered to a subscribing application. Each subscription receives its own independent copy of every message published to the topic after the subscription was created. A topic can have many subscriptions; this is the fan-out mechanism.
  • Pull delivery — The subscriber application explicitly requests messages from the subscription on its own schedule (using a client library, the CLI, or a streaming pull connection). Pull is well suited to high-throughput batch consumers and workers that control their own concurrency. Messages must be explicitly acknowledged (acked) after successful processing.
  • Push delivery — Pub/Sub proactively sends each message as an HTTPS POST request to a configured endpoint (for example a Cloud Run service, Cloud Functions function, or any publicly reachable HTTPS URL). The endpoint acknowledges a message by returning a success HTTP status (
    102
    ,
    200
    ,
    201
    ,
    202
    , or
    204
    ). Push is well suited to serverless consumers and webhook-style integrations.
  • Acknowledgement (ack) and ack deadline — A consumer must acknowledge a message to signal successful processing. If the message is not acked within the subscription's ack deadline, Pub/Sub redelivers it. This at-least-once delivery model means consumers must be designed to be idempotent.
  • Dead Letter Topic (DLT) — A topic to which messages are forwarded after a subscription fails to deliver them more than a configured maximum number of times (
    --max-delivery-attempts
    ). Dead lettering isolates "poison" messages so they stop blocking healthy traffic and can be inspected or reprocessed later. The Pub/Sub service account must have permission to publish to the dead letter topic and to subscribe to the source subscription.
  • Schema — An optional, versioned definition (Avro or Protocol Buffer) that constrains the structure of messages published to a topic. When a topic is associated with a schema and an encoding (
    JSON
    or
    BINARY
    ), Pub/Sub validates every published message and rejects non-conforming payloads. Schemas provide a contract between producers and consumers and prevent malformed data from entering the pipeline.
  • Topic(主题) — 生产者(发布者)向其发送消息的命名资源。主题是数据流入系统的入口点。主题可选择性地对传入消息强制执行Schema(模式)
  • Subscription(订阅) — 代表从单个主题流向订阅应用的消息流的命名资源。每个订阅都会独立接收订阅创建后发布到主题的每条消息的副本。一个主题可以有多个订阅,这就是扇出机制。
  • Pull delivery(拉取交付) — 订阅应用根据自身调度主动从订阅请求消息(使用客户端库、CLI或流式拉取连接)。拉取模式非常适合高吞吐量的批处理消费者和控制自身并发的工作负载。消息处理成功后必须显式确认(acked)。
  • Push delivery(推送交付) — Pub/Sub主动将每条消息作为HTTPS POST请求发送到配置的endpoint(端点)(例如Cloud Run服务、Cloud Functions函数或任何可公开访问的HTTPS URL)。端点通过返回成功HTTP状态码(
    102
    200
    201
    202
    204
    )来确认消息。推送模式非常适合无服务器消费者和webhook风格的集成。
  • Acknowledgement (ack) and ack deadline(确认与确认期限) — 消费者必须确认消息以表示处理成功。如果在订阅的确认期限内未确认消息,Pub/Sub会重新投递该消息。这种至少一次的交付模型意味着消费者必须设计为幂等的。
  • Dead Letter Topic (DLT,死信主题) — 当订阅无法投递消息超过配置的最大次数(
    --max-delivery-attempts
    )时,消息会被转发到该主题。死信机制可隔离“有毒”消息,使其不再阻塞正常流量,后续可对其进行检查或重新处理。Pub/Sub服务账号必须拥有向死信主题发布消息以及订阅源订阅的权限。
  • Schema(模式) — 可选的版本化定义(Avro或Protocol Buffer),用于约束发布到主题的消息结构。当主题关联模式和编码(
    JSON
    BINARY
    )时,Pub/Sub会验证每条发布的消息,并拒绝不符合要求的负载。模式为生产者和消费者提供了契约,可防止格式错误的数据进入管道。

Prerequisites

前提条件

Before performing any operation, confirm the following are in place.
  1. Enable the Pub/Sub API. The
    pubsub.googleapis.com
    service must be enabled on the target project.
    gcloud services enable pubsub.googleapis.com --project=PROJECT_ID
  2. Install and update the gcloud CLI. Operations in this skill assume a recent Google Cloud CLI. Confirm it is installed and current.
    gcloud version
    gcloud components update --quiet
  3. Authorize the CLI. Authenticate with an identity (user or service account) that holds the required roles described below.
    gcloud auth login
    gcloud config set project PROJECT_ID
    For automated or non-interactive environments, prefer Application Default Credentials backed by a dedicated service account:
    gcloud auth activate-service-account --key-file=KEY_FILE_PATH
  4. Confirm the active project and account. Always verify the execution context before mutating resources.
    gcloud config list --format="value(core.project,core.account)"
执行任何操作前,请确认以下条件已满足。
  1. 启用Pub/Sub API。目标项目必须启用
    pubsub.googleapis.com
    服务。
    gcloud services enable pubsub.googleapis.com --project=PROJECT_ID
  2. 安装并更新gcloud CLI。本技能中的操作假定使用最新版本的Google Cloud CLI。请确认已安装且为最新版本。
    gcloud version
    gcloud components update --quiet
  3. 授权CLI。使用拥有下文所述所需角色的身份(用户或服务账号)进行身份验证。
    gcloud auth login
    gcloud config set project PROJECT_ID
    对于自动化或非交互式环境,优先使用由专用服务账号支持的应用默认凭据:
    gcloud auth activate-service-account --key-file=KEY_FILE_PATH
  4. 确认活跃项目和账号。在修改资源前,请始终验证执行上下文。
    gcloud config list --format="value(core.project,core.account)"

Key IAM roles

关键IAM角色

Grant the least privilege required for the task. The principal acting through this skill should hold only the roles that match its responsibilities.
RoleRole IDPurpose
Pub/Sub Admin
roles/pubsub.admin
Full control: create, update, and delete topics, subscriptions, and schemas, and manage IAM policies. Restrict to provisioning identities.
Pub/Sub Publisher
roles/pubsub.publisher
Publish messages to topics. Grant to producer services.
Pub/Sub Subscriber
roles/pubsub.subscriber
Consume and acknowledge messages from subscriptions. Grant to consumer services.
Pub/Sub Viewer
roles/pubsub.viewer
Read-only access to view topics, subscriptions, schemas, and configuration. Grant for inspection and auditing.
Service Account User
roles/iam.serviceAccountUser
Allows an identity to act as (impersonate) a service account. Required when configuring authenticated push subscriptions that invoke a service using a service-account identity token.
Additional notes:
  • Push subscriptions that call private services (for example an authenticated Cloud Run endpoint) require that the push service account is granted the invoker role on the target service (for example
    roles/run.invoker
    ), in addition to
    roles/iam.serviceAccountUser
    on that service account for the configuring identity.
  • Dead letter delivery requires the Pub/Sub service agent (
    service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com
    ) to hold
    roles/pubsub.publisher
    on the dead letter topic and
    roles/pubsub.subscriber
    on the source subscription.
授予任务所需的最小权限。通过本技能执行操作的主体应仅拥有与其职责匹配的角色。
角色角色ID用途
Pub/Sub管理员
roles/pubsub.admin
完全控制权:创建、更新和删除主题、订阅和模式,以及管理IAM策略。仅限配置身份使用。
Pub/Sub发布者
roles/pubsub.publisher
向主题发布消息。授予生产者服务。
Pub/Sub订阅者
roles/pubsub.subscriber
从订阅消费并确认消息。授予消费者服务。
Pub/Sub查看者
roles/pubsub.viewer
只读访问权限,可查看主题、订阅、模式和配置。授予用于检查和审计的身份。
服务账号用户
roles/iam.serviceAccountUser
允许一个身份充当(模拟)服务账号。配置使用服务账号身份令牌调用服务的已认证推送订阅时需要此角色。
补充说明:
  • 调用私有服务(例如已认证的Cloud Run端点)的推送订阅要求推送服务账号拥有目标服务的调用者角色(例如
    roles/run.invoker
    ),此外配置身份还需拥有该服务账号的
    roles/iam.serviceAccountUser
    角色。
  • 死信交付要求Pub/Sub服务代理(
    service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com
    )拥有死信主题的
    roles/pubsub.publisher
    角色和源订阅的
    roles/pubsub.subscriber
    角色。

Composite resource deployment recipe

复合资源部署方案

Use this composite recipe when you need to provision a new schema-bound topic and subscription together. It orchestrates the configuration, validation, and creation of the dependent resources (Schema -> Topic -> Subscription) in a single execution block to minimize shell turns and prevent partial configurations.
bash
undefined
当你需要一起配置新的绑定模式的主题和订阅时,请使用此复合方案。它会在单个执行块中编排依赖资源(模式 -> 主题 -> 订阅)的配置、验证和创建,以减少shell操作次数并避免部分配置的情况。
bash
undefined

Run as a single copy-pasteable execution block

作为可直接复制粘贴的单个执行块运行

( set -e

1. Configuration Parameters

PROJECT_ID="PROJECT_ID" SCHEMA_ID="SCHEMA_ID" SCHEMA_TYPE="avro" # avro or protocol-buffer SCHEMA_FILE="PATH_TO_SCHEMA_FILE" # Local path to schema definition file ENCODING="json" # json or binary TOPIC_ID="TOPIC_ID" SUBSCRIPTION_ID="SUBSCRIPTION_ID" ACK_DEADLINE=60 # Acknowledgement deadline in seconds
echo "=== [1/5] Validating Schema Definition ===" gcloud pubsub schemas validate-schema
--type="${SCHEMA_TYPE}"
--definition-file="${SCHEMA_FILE}"
--project="${PROJECT_ID}"
echo "=== [2/5] Creating Schema ===" gcloud pubsub schemas create "${SCHEMA_ID}"
--type="${SCHEMA_TYPE}"
--definition-file="${SCHEMA_FILE}"
--project="${PROJECT_ID}"
--quiet
echo "=== [3/5] Creating Topic ===" gcloud pubsub topics create "${TOPIC_ID}"
--schema="${SCHEMA_ID}"
--message-encoding="${ENCODING}"
--project="${PROJECT_ID}"
--quiet
echo "=== [4/5] Creating Subscription ===" gcloud pubsub subscriptions create "${SUBSCRIPTION_ID}"
--topic="${TOPIC_ID}"
--ack-deadline="${ACK_DEADLINE}"
--project="${PROJECT_ID}"
--quiet
echo "=== [5/5] Verifying Resources ===" gcloud pubsub schemas describe "${SCHEMA_ID}" --project="${PROJECT_ID}" gcloud pubsub topics describe "${TOPIC_ID}" --project="${PROJECT_ID}" gcloud pubsub subscriptions describe "${SUBSCRIPTION_ID}" --project="${PROJECT_ID}" )
undefined
( set -e

1. 配置参数

PROJECT_ID="PROJECT_ID" SCHEMA_ID="SCHEMA_ID" SCHEMA_TYPE="avro" # avro 或 protocol-buffer SCHEMA_FILE="PATH_TO_SCHEMA_FILE" # 模式定义文件的本地路径 ENCODING="json" # json 或 binary TOPIC_ID="TOPIC_ID" SUBSCRIPTION_ID="SUBSCRIPTION_ID" ACK_DEADLINE=60 # 确认期限(秒)
echo "=== [1/5] 验证模式定义 ===" gcloud pubsub schemas validate-schema
--type="${SCHEMA_TYPE}"
--definition-file="${SCHEMA_FILE}"
--project="${PROJECT_ID}"
echo "=== [2/5] 创建模式 ===" gcloud pubsub schemas create "${SCHEMA_ID}"
--type="${SCHEMA_TYPE}"
--definition-file="${SCHEMA_FILE}"
--project="${PROJECT_ID}"
--quiet
echo "=== [3/5] 创建主题 ===" gcloud pubsub topics create "${TOPIC_ID}"
--schema="${SCHEMA_ID}"
--message-encoding="${ENCODING}"
--project="${PROJECT_ID}"
--quiet
echo "=== [4/5] 创建订阅 ===" gcloud pubsub subscriptions create "${SUBSCRIPTION_ID}"
--topic="${TOPIC_ID}"
--ack-deadline="${ACK_DEADLINE}"
--project="${PROJECT_ID}"
--quiet
echo "=== [5/5] 验证资源 ===" gcloud pubsub schemas describe "${SCHEMA_ID}" --project="${PROJECT_ID}" gcloud pubsub topics describe "${TOPIC_ID}" --project="${PROJECT_ID}" gcloud pubsub subscriptions describe "${SUBSCRIPTION_ID}" --project="${PROJECT_ID}" )
undefined

Schema enforcement: "JSON" is a message encoding, not a schema type

模式强制执行:"JSON"是消息编码,而非模式类型

This trips up almost everyone. Google Cloud Pub/Sub has two schema types:
avro
and
protocol-buffer
. There is no
json
schema type, and
gcloud pubsub schemas create --type=json
will fail.
When a task asks for a "JSON schema" or "strict JSON validation," it means the topic validates messages with JSON encoding against an attached schema. JSON is the
--message-encoding
you set on the topic, not the type of the schema. The schema definition itself is normally written in AVRO.
So a request for a "strict JSON schema" breaks into two parts:
  1. A schema attached to the topic (the data contract).
  2. The topic's
    schemaSettings.encoding
    set to
    json
    (messages validated as JSON).
这几乎会困扰所有人。Google Cloud Pub/Sub有两种模式类型
avro
protocol-buffer
。不存在
json
模式类型,执行
gcloud pubsub schemas create --type=json
会失败。
当任务要求“JSON模式”或“严格JSON验证”时,意味着主题会根据附加的模式对JSON编码的消息进行验证。JSON是你在主题上设置的
--message-encoding
,而非模式的类型。模式定义本身通常用AVRO编写。
因此,“严格JSON模式”的需求分为两部分:
  1. 附加到主题的模式(数据契约)。
  2. 主题的
    schemaSettings.encoding
    设置为
    json
    (消息以JSON格式验证)。

Correct recipe

正确方案

bash
SCHEMA="orders-schema"
TOPIC="orders-topic"
bash
SCHEMA="orders-schema"
TOPIC="orders-topic"

1. Create the schema. Type is avro; the definition is written in Avro JSON.

1. 创建模式。类型为avro;定义用Avro JSON编写。

gcloud pubsub schemas create "$SCHEMA"
--type=avro
--definition='{"type":"record","name":"Order","fields":[ {"name":"id","type":"string"}, {"name":"amount","type":"double"} ]}'
gcloud pubsub schemas create "$SCHEMA"
--type=avro
--definition='{"type":"record","name":"Order","fields":[ {"name":"id","type":"string"}, {"name":"amount","type":"double"} ]}'

2. Create the topic and bind the schema with JSON MESSAGE ENCODING.

2. 创建主题并绑定模式,同时设置JSON消息编码。

--message-encoding=json is the part that makes it a "JSON schema".

--message-encoding=json 是使其成为“JSON模式”的关键部分。

gcloud pubsub topics create "$TOPIC"
--schema="$SCHEMA"
--message-encoding=json
gcloud pubsub topics create "$TOPIC"
--schema="$SCHEMA"
--message-encoding=json

3. Verify the contract: encoding must be json.

3. 验证契约:编码必须为json。

gcloud pubsub topics describe "$TOPIC" --format="value(schemaSettings.encoding)" # -> json

If a topic already exists, bind the schema and encoding with
`gcloud pubsub topics update "$TOPIC" --schema="$SCHEMA" --message-encoding=json`.

> **Default to JSON encoding** unless a task explicitly asks for binary. Reach for
> `--message-encoding=binary` only when the requirement calls for it. Choosing
> binary (or omitting the encoding) is the most common reason a "JSON schema"
> requirement is scored as unmet, because the schema looks present but the
> messages are not JSON-validated.
gcloud pubsub topics describe "$TOPIC" --format="value(schemaSettings.encoding)" # -> json

如果主题已存在,可使用`gcloud pubsub topics update "$TOPIC" --schema="$SCHEMA" --message-encoding=json`绑定模式和编码。

> **默认使用JSON编码**,除非任务明确要求二进制编码。仅当需求明确指定时才使用`--message-encoding=binary`。选择二进制编码(或省略编码)是“JSON模式”需求未得到满足的最常见原因,因为模式看似存在,但消息并未经过JSON验证。

Troubleshooting

故障排除

  • "Schema is AVRO but I need JSON": That is a category error. AVRO is the schema type; JSON is the message encoding. Keep the AVRO schema and set the topic's
    --message-encoding=json
    . Do not try to change the schema type.
  • Encoding shows empty / BINARY: the topic was created without
    --message-encoding=json
    (or without
    --schema
    ). Re-bind with
    gcloud pubsub topics update --schema=... --message-encoding=json
    .
  • “模式是AVRO,但我需要JSON”:这是分类错误。AVRO是模式类型;JSON是消息编码。保留AVRO模式并将主题的
    --message-encoding
    设置为json。不要尝试更改模式类型。
  • 编码显示为空 / BINARY:主题创建时未设置
    --message-encoding=json
    (或未设置
    --schema
    )。使用
    gcloud pubsub topics update --schema=... --message-encoding=json
    重新绑定。

Creating topics and subscriptions

创建主题和订阅

All commands below use placeholders in CAPS. Replace
TOPIC_ID
,
SUBSCRIPTION_ID
,
PROJECT_ID
,
ENDPOINT
, and similar tokens with concrete values before execution. Every command explicitly scopes the project with
--project=PROJECT_ID
and runs non-interactively with
--quiet
.
以下所有命令均使用大写占位符。执行前请将
TOPIC_ID
SUBSCRIPTION_ID
PROJECT_ID
ENDPOINT
等占位符替换为具体值。每个命令都通过
--project=PROJECT_ID
显式指定项目,并通过
--quiet
以非交互模式运行。

1. Create a topic

1. 创建主题

gcloud pubsub topics create TOPIC_ID \
  --project=PROJECT_ID \
  --quiet
To create a topic that enforces a message-retention duration (so that even subscriptions can replay within the window):
gcloud pubsub topics create TOPIC_ID \
  --message-retention-duration=7d \
  --project=PROJECT_ID \
  --quiet
To associate a topic with an existing schema:
gcloud pubsub topics create TOPIC_ID \
  --schema=SCHEMA_ID \
  --message-encoding=json \
  --project=PROJECT_ID \
  --quiet
gcloud pubsub topics create TOPIC_ID \
  --project=PROJECT_ID \
  --quiet
要创建强制消息保留时长的主题(以便订阅可在该时间段内重放消息):
gcloud pubsub topics create TOPIC_ID \
  --message-retention-duration=7d \
  --project=PROJECT_ID \
  --quiet
要将主题与现有模式关联:
gcloud pubsub topics create TOPIC_ID \
  --schema=SCHEMA_ID \
  --message-encoding=json \
  --project=PROJECT_ID \
  --quiet

2. Create a Pull subscription

2. 创建拉取订阅

A pull subscription lets the consumer fetch messages on demand. The
--ack-deadline
controls how long a consumer has to acknowledge a message before redelivery.
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet
To attach a dead letter topic and bound redelivery attempts:
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --dead-letter-topic=DEAD_LETTER_TOPIC_ID \
  --max-delivery-attempts=5 \
  --project=PROJECT_ID \
  --quiet
To restrict which messages this subscription receives using a server-side filter on message attributes (see Data Reduction below):
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --message-filter='attributes.eventType = "ORDER_CREATED"' \
  --project=PROJECT_ID \
  --quiet
拉取订阅允许消费者按需获取消息。
--ack-deadline
控制消费者在消息被重新投递前的确认时长。
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet
要附加死信主题并限制重投次数:
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --dead-letter-topic=DEAD_LETTER_TOPIC_ID \
  --max-delivery-attempts=5 \
  --project=PROJECT_ID \
  --quiet
要使用基于消息属性的服务器端过滤器限制订阅接收的消息(参见下文的数据缩减):
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --message-filter='attributes.eventType = "ORDER_CREATED"' \
  --project=PROJECT_ID \
  --quiet

3. Create a Push subscription

3. 创建推送订阅

A push subscription delivers each message to an HTTPS
ENDPOINT
. The endpoint must return a success status to acknowledge a message.
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet
For an authenticated push subscription that targets a private service, attach a push-auth service account so Pub/Sub sends a signed OIDC token:
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --push-auth-service-account=PUSH_SERVICE_ACCOUNT_EMAIL \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet
推送订阅会将每条消息投递到HTTPS
ENDPOINT
。端点必须返回成功状态码以确认消息。
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet
对于目标为私有服务的已认证推送订阅,请附加推送认证服务账号,以便Pub/Sub发送签名的OIDC令牌:
gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --push-auth-service-account=PUSH_SERVICE_ACCOUNT_EMAIL \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet

4. Verify what you created

4. 验证创建的资源

After creation, confirm configuration before relying on it.
gcloud pubsub topics describe TOPIC_ID \
  --project=PROJECT_ID

gcloud pubsub subscriptions describe SUBSCRIPTION_ID \
  --project=PROJECT_ID
创建完成后,请在依赖资源前确认配置正确。
gcloud pubsub topics describe TOPIC_ID \
  --project=PROJECT_ID

gcloud pubsub subscriptions describe SUBSCRIPTION_ID \
  --project=PROJECT_ID

Resource naming and verification checklist

资源命名和验证清单

Before completing any Pub/Sub provisioning task, run through this checklist to ensure all resources match expectations:
  • Verify resource names match requirements: Cross-reference the exact names of the Schema, Topic, and Subscription in the prompt or spec. Do not assume names.
  • Verify labels are applied: If the prompt requests labels (e.g.
    windtunnel-eval
    ), verify they are present on both the Topic and Subscription.
  • Verify resource existence: Run
    describe
    commands for the Schema, Topic, and Subscription and confirm the outputs show
    state: ACTIVE
    (for schema) and correct configurations.
完成任何Pub/Sub配置任务前,请运行以下清单以确保所有资源符合预期:
  • 验证资源名称符合要求:对照提示或规范中的模式、主题和订阅的确切名称进行交叉检查。不要假设名称。
  • 验证标签已应用:如果提示要求添加标签(例如
    windtunnel-eval
    ),请验证主题和订阅上均已添加该标签。
  • 验证资源存在:对模式、主题和订阅运行
    describe
    命令,确认输出显示
    state: ACTIVE
    (针对模式)以及正确的配置。

Core principles

核心原则

These principles are mandatory. Apply them to every Pub/Sub operation.
这些原则是强制性的。请将其应用于所有Pub/Sub操作。

Explicit command validation (mandatory)

显式命令验证(强制性)

  • Before running any mutating command, restate in plain language what the command will do and to which resource. Confirm the resource name, the project, and the effect.
  • Use
    --dry-run
    where the subcommand supports it, or run the read-only
    describe
    /
    list
    equivalent first to confirm the current state.
  • Never execute a command whose effect you cannot precisely predict. If the outcome is ambiguous, stop and validate first.
  • 在运行任何修改命令前,用通俗易懂的语言说明命令将执行的操作以及作用的资源。确认资源名称、项目和影响。
  • 在子命令支持的情况下使用
    --dry-run
    ,或者先运行只读的
    describe
    /
    list
    命令确认当前状态。
  • 永远不要执行你无法准确预测其效果的命令。如果结果不明确,请先停止并验证。

Data reduction

数据缩减

Minimize the volume of data fetched, processed, and returned. Large, unfiltered reads are slow, costly, and noisy.
  • --limit
    — Cap the number of results when listing or pulling. For example,
    gcloud pubsub subscriptions pull SUBSCRIPTION_ID --limit=10
    .
  • Server-side filtering — Apply
    --message-filter
    at subscription creation so unwanted messages are never delivered, rather than filtering after the fact. Use
    --filter
    on
    list
    commands to narrow results server-side where supported.
  • Projection — Request only the fields you need with
    --format="value(...)"
    or
    --format="table(...)"
    instead of dumping full resource representations.
尽量减少获取、处理和返回的数据量。大量未过滤的读取操作速度慢、成本高且冗余。
  • --limit
    — 列出或拉取结果时限制数量。例如,
    gcloud pubsub subscriptions pull SUBSCRIPTION_ID --limit=10
  • 服务器端过滤 — 在创建订阅时应用
    --message-filter
    ,这样不需要的消息永远不会被投递,而不是事后过滤。在支持的情况下,对
    list
    命令使用
    --filter
    在服务器端缩小结果范围。
  • 投影 — 使用
    --format="value(...)"
    --format="table(...)"
    仅请求你需要的字段,而不是转储完整的资源表示。

Project and location scoping

项目和地域范围

  • Always append
    --project=PROJECT_ID
    to every command. Never rely solely on the ambient
    gcloud config
    project; explicit scoping prevents acting on the wrong project.
  • For resources that support location or regional endpoints, specify the location explicitly rather than depending on defaults.
  • 始终在每个命令后附加
    --project=PROJECT_ID
    。永远不要仅依赖环境中的
    gcloud config
    项目;显式指定范围可防止误操作错误的项目。
  • 对于支持地域或区域端点的资源,请显式指定地域,而不是依赖默认值。

Execution constraints

执行约束

  • Non-interactive mode — Append
    --quiet
    (or
    -q
    ) to suppress interactive prompts so commands are deterministic and automation-safe. This must not be used to bypass the human-approval requirements in the Safety section below.
  • Single commands — Execute one discrete
    gcloud
    command at a time. Do not chain unrelated operations into a single invocation.
  • No raw shell operators — Do not use shell pipes (
    |
    ), redirects (
    >
    ,
    >>
    ), command substitution (
    $(...)
    ), background operators (
    &
    ), or
    &&
    /
    ;
    chaining to compose Pub/Sub commands. Each command must stand alone and be independently auditable.
  • 非交互模式 — 附加
    --quiet
    (或
    -q
    )以抑制交互式提示,使命令具有确定性且适合自动化。但不得用于绕过下文安全部分中的人工审批要求。
  • 单个命令 — 一次执行一个独立的
    gcloud
    命令。不要将无关操作链接到单个调用中。
  • 禁止使用原始shell操作符 — 不要使用shell管道(
    |
    )、重定向(
    >
    >>
    )、命令替换(
    $(...)
    )、后台操作符(
    &
    )或
    &&
    /
    ;
    链接来组合Pub/Sub命令。每个命令必须独立且可单独审计。

Safety and guardrails

安全和防护措施

[!CAUTION] Pub/Sub operations can be destructive and irreversible. Deleting a topic or subscription permanently discards undelivered messages and breaks every producer and consumer bound to it. Changing IAM policy can silently cut off access for production services. Treat every mutating operation as production-impacting unless explicitly proven otherwise.
[!CAUTION] Pub/Sub操作可能具有破坏性且不可逆。删除主题或订阅会永久丢弃未投递的消息,并中断所有绑定到它的生产者和消费者。更改IAM策略可能会静默切断生产服务的访问权限。除非明确证明不会影响生产,否则请将每个修改操作视为会影响生产环境的操作。

Prohibited operations (denylist)

禁止操作(黑名单)

The following operations MUST NOT be executed without explicit, recorded human approval. When a task appears to require one of these, stop and request approval first; describe the exact command and its blast radius.
  • Deleting a topic
    gcloud pubsub topics delete ...
    . Destroys the topic and detaches all subscriptions.
  • Deleting a subscription
    gcloud pubsub subscriptions delete ...
    . Permanently drops all unacknowledged messages held by that subscription.
  • Deleting or modifying a schema
    gcloud pubsub schemas delete ...
    or schema revision changes that can invalidate in-flight producers.
  • Purging/seeking messages destructively
    gcloud pubsub subscriptions seek ...
    to a timestamp or snapshot that discards the current backlog.
  • Changing critical IAM bindings — Adding, removing, or replacing
    roles/pubsub.admin
    ,
    roles/pubsub.publisher
    , or
    roles/pubsub.subscriber
    bindings on production topics or subscriptions, or any
    set-iam-policy
    operation that replaces an entire policy.
  • Detaching a topic
    gcloud pubsub topics detach-subscription ...
    , which immediately stops delivery to the affected subscription.
以下操作必须获得明确的、已记录的人工审批才能执行。当任务似乎需要执行以下操作时,请先停止并请求审批;说明确切的命令及其影响范围。
  • 删除主题
    gcloud pubsub topics delete ...
    。销毁主题并分离所有订阅。
  • 删除订阅
    gcloud pubsub subscriptions delete ...
    。永久删除该订阅持有的所有未确认消息。
  • 删除或修改模式
    gcloud pubsub schemas delete ...
    或可能使正在运行的生产者失效的模式修订更改。
  • 破坏性清除/查找消息
    gcloud pubsub subscriptions seek ...
    到某个时间戳或快照,这会丢弃当前的消息积压。
  • 更改关键IAM绑定 — 在生产主题或订阅上添加、删除或替换
    roles/pubsub.admin
    roles/pubsub.publisher
    roles/pubsub.subscriber
    绑定,或任何替换整个策略的
    set-iam-policy
    操作。
  • 分离主题
    gcloud pubsub topics detach-subscription ...
    ,这会立即停止向受影响的订阅投递消息。

Always-safe operations

始终安全的操作

Read-only inspection commands (
describe
,
list
, and bounded
pull
without auto-ack for diagnostics) are safe and encouraged for validation, provided they respect the data-reduction principles above.
只读检查命令(
describe
list
以及用于诊断的非自动确认的有限
pull
)是安全的,建议用于验证,前提是遵守上述数据缩减原则。

Troubleshooting / What to do if...

故障排除 / 如果出现以下情况该怎么办...

...a command fails with a permission error (PERMISSION_DENIED / 403)

...命令因权限错误失败(PERMISSION_DENIED / 403)

  1. Confirm the active identity:
    gcloud config list --format="value(core.account)"
    .
  2. Confirm the role on the resource matches the action (see the IAM table). For example, publishing requires
    roles/pubsub.publisher
    ; consuming requires
    roles/pubsub.subscriber
    .
  3. For push to private endpoints, verify the push service account has the invoker role on the target service and that the configuring identity holds
    roles/iam.serviceAccountUser
    on that service account.
  4. Confirm the API is enabled:
    gcloud services list --enabled --filter="pubsub" --project=PROJECT_ID
    .
  1. 确认活跃身份:
    gcloud config list --format="value(core.account)"
  2. 确认资源上的角色与操作匹配(参见IAM表)。例如,发布需要
    roles/pubsub.publisher
    ;消费需要
    roles/pubsub.subscriber
  3. 对于推送至私有端点的情况,请验证推送服务账号拥有目标服务的调用者角色,且配置身份拥有该服务账号的
    roles/iam.serviceAccountUser
    角色。
  4. 确认API已启用:
    gcloud services list --enabled --filter="pubsub" --project=PROJECT_ID

...a subscription has a growing backlog (messages not being consumed fast enough)

...订阅的消息积压不断增加(消息消费速度不够快)

  1. Inspect the backlog and oldest unacked age in Cloud Monitoring metrics (
    subscription/num_undelivered_messages
    ,
    subscription/oldest_unacked_message_age
    ).
  2. For pull consumers, scale out the number of consumer instances or increase per-instance concurrency so total throughput exceeds the publish rate.
  3. Confirm consumers are acking successfully; failed processing leads to redelivery and inflated backlog.
  4. Consider whether a
    --message-filter
    should reduce delivered volume, or whether the topic is over-fanned-out.
  1. 在Cloud Monitoring指标中检查积压和最旧未确认消息的时长(
    subscription/num_undelivered_messages
    subscription/oldest_unacked_message_age
    )。
  2. 对于拉取消费者,增加消费者实例数量或提高每个实例的并发度,使总吞吐量超过发布速率。
  3. 确认消费者正在成功确认消息;处理失败会导致重投并增加积压。
  4. 考虑是否应使用
    --message-filter
    减少投递量,或者主题是否过度扇出。

...messages accumulate unacknowledged (repeated redelivery)

...消息未被确认(重复投递)

  1. Verify the consumer is calling acknowledge after successful processing.
  2. Check the
    --ack-deadline
    . If processing routinely takes longer than the deadline, increase it or extend the deadline programmatically; otherwise Pub/Sub redelivers before the consumer finishes.
  3. Ensure consumers are idempotent — at-least-once delivery guarantees that duplicates can occur, and non-idempotent handlers compound the problem.
  4. If specific "poison" messages never succeed, attach a dead letter topic with
    --max-delivery-attempts
    so they are isolated instead of redelivered forever.
  1. 验证消费者在处理成功后是否调用了确认操作。
  2. 检查
    --ack-deadline
    。如果处理通常需要超过该期限的时间,请增加期限或通过编程方式延长期限;否则Pub/Sub会在消费者完成处理前重新投递消息。
  3. 确保消费者是幂等的——至少一次的交付保证意味着可能会出现重复消息,非幂等处理程序会加剧问题。
  4. 如果特定“有毒”消息始终处理失败,请附加死信主题并设置
    --max-delivery-attempts
    ,以便将其隔离而不是永远重投。

...schema validation fails (published messages are rejected)

...模式验证失败(发布的消息被拒绝)

  1. Confirm the topic's bound schema and encoding:
    gcloud pubsub topics describe TOPIC_ID --project=PROJECT_ID
    .
  2. Validate a sample message against the schema before publishing:
    gcloud pubsub schemas validate-message ...
    .
  3. Confirm the producer serializes with the encoding the topic expects (
    JSON
    vs
    BINARY
    ); a mismatch causes rejection even for structurally valid data.
  4. If the schema legitimately needs to change, commit a new, compatible schema revision rather than editing producers ad hoc — and treat schema deletion as a prohibited operation requiring approval.
  1. 确认主题绑定的模式和编码:
    gcloud pubsub topics describe TOPIC_ID --project=PROJECT_ID
  2. 在发布前验证示例消息是否符合模式:
    gcloud pubsub schemas validate-message ...
  3. 确认生产者使用的序列化编码与主题期望的一致(
    JSON
    vs
    BINARY
    );即使数据结构有效,编码不匹配也会导致消息被拒绝。
  4. 如果模式确实需要更改,请提交新的兼容模式修订版,而不是临时修改生产者——并将模式删除视为需要审批的禁止操作。

Reference directory

参考目录

Consult these reference documents for deeper, task-specific detail:
  • references/cli-usage.md
    — Comprehensive
    gcloud pubsub
    command reference, including publishing, pulling, acknowledging, seeking, snapshots, and output formatting recipes.
  • references/iam-security.md
    — Detailed IAM role and policy guidance, least-privilege patterns, service-account configuration for push delivery, and dead-letter permission setup.
  • references/client-libraries.md
    — Pointers to the official Google Cloud client libraries for programmatic publishing and consuming, with notes on when to prefer SDK code over the CLI.
如需更深入的任务特定细节,请查阅以下参考文档:
  • references/cli-usage.md
    — 全面的
    gcloud pubsub
    命令参考,包括发布、拉取、确认、查找、快照和输出格式方案。
  • references/iam-security.md
    — 详细的IAM角色和策略指南、最小权限模式、推送交付的服务账号配置以及死信权限设置。
  • references/client-libraries.md
    — 指向官方Google Cloud客户端库的链接,用于编程式发布和消费,并说明何时优先使用SDK代码而非CLI。