google-cloud-filestore-log-troubleshooting

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
<!-- disableFinding(LINE_OVER_80) -->
<!-- disableFinding(LINE_OVER_80) -->

Google Cloud Filestore Log-Based Troubleshooting

Google Cloud Filestore 基于日志的故障排查

Diagnoses, troubleshoots, and remediates Google Cloud Filestore client mount failures, permission errors (
EACCES
), and network timeouts (
ETIMEDOUT
) across projects.
用于跨项目诊断、排查和修复Google Cloud Filestore客户端挂载失败、权限错误(
EACCES
)及网络超时(
ETIMEDOUT
)问题。

Prerequisites & Quick Start

前提条件与快速开始

Required IAM roles on target project(s) (and Shared VPC host project if applicable):
  • Read:
    roles/file.viewer
    (instance & export ACLs),
    roles/compute.networkViewer
    (VPC firewall rules),
    roles/logging.viewer
    (Cloud Audit & GKE CSI logs),
    roles/mcp.toolUser
    (if using MCP tools).
  • Write (Remediation only):
    roles/file.editor
    (export ACL updates),
    roles/compute.securityAdmin
    (firewall rule creation).
Authenticate, verify billing/APIs, and configure your environment:
bash
gcloud auth login && gcloud auth application-default login
gcloud billing projects describe {project_id} --format="value(billingEnabled)"
gcloud services enable file.googleapis.com compute.googleapis.com logging.googleapis.com --quiet
gcloud config set project {project_id} && gcloud config set compute/region {region}
目标项目(如适用,还包括共享VPC宿主项目)所需的IAM角色:
  • 读取权限:
    roles/file.viewer
    (实例与导出ACL)、
    roles/compute.networkViewer
    (VPC防火墙规则)、
    roles/logging.viewer
    (Cloud Audit与GKE CSI日志)、
    roles/mcp.toolUser
    (如使用MCP工具)。
  • 写入权限(仅用于修复):
    roles/file.editor
    (导出ACL更新)、
    roles/compute.securityAdmin
    (防火墙规则创建)。
完成身份验证、验证计费/API并配置环境:
bash
gcloud auth login && gcloud auth application-default login
gcloud billing projects describe {project_id} --format="value(billingEnabled)"
gcloud services enable file.googleapis.com compute.googleapis.com logging.googleapis.com --quiet
gcloud config set project {project_id} && gcloud config set compute/region {region}

Attribution

归属标注

Prefix every
gcloud
command provided or executed with the skill metrics environment:
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud filestore instances describe ...
On direct REST API calls, append HTTP header:
User-Agent: gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)
.
在所有提供或执行的
gcloud
命令前添加技能指标环境变量:
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud filestore instances describe ...
直接调用REST API时,添加HTTP头:
User-Agent: gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)
。

Conceptual & Informational Queries (CRITICAL)

概念性与信息查询(关键)

For purely conceptual, architectural, or educational questions (e.g., "What causes EACCES on Filestore?", "Why does GKE Node IP appear instead of Pod IP?", "What ports does Filestore require?", "Explain root squash"):
  • Rule: Answer immediately using pre-trained knowledge and the workflow rules below. Answering directly minimizes tool latency and token usage when the user only seeks architectural guidance.
  • Constraint: Do not execute external tool calls or API requests for basic knowledge questions.
对于纯概念性、架构性或教学类问题(例如“Filestore上出现EACCES的原因是什么?”、“为什么显示GKE节点IP而非Pod IP?”、“Filestore需要哪些端口?”、“解释root squash的作用”):
  • 规则:使用预训练知识和以下工作流规则立即回答。 当用户仅需要架构指导时,直接回答可最大程度减少工具延迟和Token消耗。
  • 约束:对于基础知识问题,不得执行外部工具调用或API请求。

Handling "No-Command" Constraints (CRITICAL)

“禁止执行命令”约束的处理(关键)

If the user prompt contains constraints like "Do not execute commands", "without executing", or "read-only":
  • Rule: Strictly avoid calling
    run_command
    to execute any shell, Python, or
    gcloud
    commands.
  • Discovery:
    1. First, check if Filestore MCP tools (
      get_instance
      ,
      list_instances
      ) are available and use them (API calls, not command executions).
    2. If MCP tools are unavailable, read
      references/mock-fleet-data.md
      only if the requested instance matches one of the evaluation scenarios (
      finance-share
      ,
      shared-nfs
      ,
      ml-data
      ,
      data-hub
      ,
      prod-share
      ). Never report mock data as live production state. If the instance is not listed there, state that live access is required and provide the exact commands for the user to run.
    3. Fast-Path Stop Rule: Once you locate the target instance in
      references/mock-fleet-data.md
      , stop reading additional files immediately and formulate your response. Do NOT read
      scripts/quick_diagnose.py
      ,
      scripts/diagnose_lib.py
      ,
      _internal/quick_diagnose_test.py
      , or
      EVAL.*
      files when command execution is disabled, as inspecting code/test files wastes turns and triggers timeouts.
    4. Explain the required diagnostic steps and output the exact attributed commands for manual execution.
  • Mandatory User Confirmation Requirement: Even when command execution is disabled or the user asks only for recommendations, your response MUST STILL end with a clear question prompting the user for explicit confirmation before applying any remediation (e.g., "Would you like me to proceed with creating the VPC ingress firewall rule
    [rule_name]
    ? Please confirm to proceed."
    ).
如果用户提示包含“不得执行命令”、“无需执行”或“只读”等约束:
  • 规则:**严格禁止调用
    run_command
    **执行任何Shell、Python或
    gcloud
    命令。
  • 排查方式:
    1. 首先检查Filestore MCP工具(
      get_instance
      、
      list_instances
      )是否可用并使用它们(是API调用,而非命令执行)。
    2. 如果MCP工具不可用,仅当请求的实例匹配评估场景(
      finance-share
      、
      shared-nfs
      、
      ml-data
      、
      data-hub
      、
      prod-share
      )之一时,才读取
      references/mock-fleet-data.md
      。严禁将模拟数据报告为实时生产状态。如果实例未在其中列出,说明需要实时访问权限,并提供供用户运行的精确命令。
    3. 快速停止规则:在
      references/mock-fleet-data.md
      中找到目标实例后,立即停止读取其他文件并组织回复。当命令执行被禁用时,请勿读取
      scripts/quick_diagnose.py
      、
      scripts/diagnose_lib.py
      、
      _internal/quick_diagnose_test.py
      或
      EVAL.*
      文件,因为检查代码/测试文件会浪费轮次并导致超时。
    4. 说明所需的诊断步骤,并输出精确的带归属标识的命令供手动执行。
  • 强制用户确认要求:即使命令执行被禁用或用户仅要求建议,你的回复仍必须以明确的问题结尾,在应用任何修复前请求用户明确确认(例如:“你是否希望我继续创建VPC入站防火墙规则
    [rule_name]
    ?请确认以继续。”
    )。

Multi-Runtime Execution Options

多运行时执行选项

Option 1: Bundled Python CLI Script (Recommended for CLI / Terminal Agents)

选项1:捆绑的Python CLI脚本(推荐用于CLI/终端Agent)

bash
undefined
bash
undefined

Single instance diagnosis

Single instance diagnosis

python3 scripts/quick_diagnose.py --instance="<INSTANCE_ID>" --location="<LOCATION_OR_ZONE>"
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]
python3 scripts/quick_diagnose.py --instance="<INSTANCE_ID>" --location="<LOCATION_OR_ZONE>"
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]

Bulk project-wide fleet diagnosis

Bulk project-wide fleet diagnosis

python3 scripts/quick_diagnose.py --all --project="<PROJECT_ID>" [--json]

| Flag | Purpose |
| :--- | :--- |
| `--instance`, `--location` | Filestore instance ID and region/zone (`--zone` is a legacy alias). Required unless `--all`. |
| `--project` | GCP project ID (defaults to active `gcloud` project). |
| `--client-ip` / `--client-subnet` | Client IP or CIDR to evaluate against export ACLs and ingress firewall rules. |
| `--json` | Emit machine-readable JSON on stdout (narrative report goes to stderr). |
| `--apply-fix` | Execute generated remediation commands. **Only pass after explicit user confirmation.** |
python3 scripts/quick_diagnose.py --all --project="<PROJECT_ID>" [--json]

| 标志 | 用途 |
| :--- | :--- |
| `--instance`, `--location` | Filestore实例ID和区域/可用区(`--zone`是旧版别名)。除非使用`--all`,否则为必填项。 |
| `--project` | GCP项目ID(默认为当前激活的`gcloud`项目)。 |
| `--client-ip` / `--client-subnet` | 用于评估导出ACL和入站防火墙规则的客户端IP或CIDR。 |
| `--json` | 在标准输出输出机器可读的JSON(叙述性报告输出到标准错误)。 |
| `--apply-fix` | 执行生成的修复命令。**仅在获得用户明确确认后添加此参数。** |

Option 2: Filestore MCP Tools / REST API /
gcloud
CLI

选项2:Filestore MCP工具 / REST API /
gcloud
CLI

  • MCP Tools: Call
    get_instance(name='projects/{project_id}/locations/{location}/instances/{instance_id}')
    and inspect
    fileShares[0].nfsExportOptions
    and
    networks[0].network
    .
  • Native REST API (
    call_gcp_api
    )
    : Invoke
    service="file"
    ,
    version="v1"
    ,
    resource_path="projects/{project_id}/locations/{location}/instances/{instance_id}"
    .
  • Standard
    gcloud
    :
    bash
    CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
    gcloud filestore instances describe {instance_id} --location={location} --project={project_id} --format=json

  • MCP工具:调用
    get_instance(name='projects/{project_id}/locations/{location}/instances/{instance_id}')
    并检查
    fileShares[0].nfsExportOptions
    和
    networks[0].network
    。
  • 原生REST API(
    call_gcp_api
    )
    :调用
    service="file"
    、
    version="v1"
    、
    resource_path="projects/{project_id}/locations/{location}/instances/{instance_id}"
    。
  • 标准
    gcloud
    :
    bash
    CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
    gcloud filestore instances describe {instance_id} --location={location} --project={project_id} --format=json

Core Operational Workflow

核心操作工作流

Step 1: Parameter Extraction & GKE Node Architecture

步骤1:参数提取与GKE节点架构

  • Extract
    instance
    ,
    location
    (region/zone),
    project
    , and
    client_ip
    /
    client_subnet
    . If target parameters are missing, list instances or ask the user to confirm.
  • GKE Architecture Rule: PersistentVolumes are mounted by the Linux kernel on the GKE Worker Node, not inside the Pod network namespace. Even if Pod IPs (
    10.4.0.0/14
    ) are allowlisted, the NFS server sees traffic originating from the GKE Node Internal IP. Always evaluate and allowlist the GKE Node Subnet CIDR in
    nfsExportOptions
    and VPC firewall rules.
  • 提取
    instance
    、
    location
    (区域/可用区)、
    project
    以及
    client_ip
    /
    client_subnet
    。如果缺少目标参数,列出实例或要求用户确认。
  • GKE架构规则:PersistentVolume由Linux内核挂载在GKE工作节点上,而非Pod网络命名空间内。即使Pod IP(
    10.4.0.0/14
    )在允许列表中,NFS服务器看到的流量也来自GKE节点内部IP。始终在
    nfsExportOptions
    和VPC防火墙规则中评估并加入GKE节点子网CIDR的允许列表。

Step 2: Instance Metadata & Shared VPC Resolution

步骤2:实例元数据与共享VPC解析

  1. Verify instance
    state
    is
    READY
    (report state blocker if
    CREATING
    ,
    DELETING
    , or
    ERROR
    ).
  2. Extract primary Filestore IP (
    networks[0].ipAddresses[0]
    ) and VPC network URI.
  3. If
    networks[0].network
    references
    projects/{host_project}/global/networks/{network}
    , resolve
    {host_project}
    as the Shared VPC host project for firewall queries.
  1. 验证实例
    state
    为
    READY
    (如果状态为
    CREATING
    、
    DELETING
    或
    ERROR
    ,报告状态阻塞问题)。
  2. 提取Filestore主IP(
    networks[0].ipAddresses[0]
    )和VPC网络URI。
  3. 如果
    networks[0].network
    引用
    projects/{host_project}/global/networks/{network}
    ,则将
    {host_project}
    解析为共享VPC宿主项目以用于防火墙查询。

Step 3: Export ACL Evaluation (
EACCES
vs
EROFS
)

步骤3:导出ACL评估(
EACCES
与
EROFS
)

  1. Inspect
    fileShares[0].nfsExportOptions
    (if empty, default
    0.0.0.0/0
    READ_WRITE
    NO_ROOT_SQUASH
    applies).
  2. EACCES (Permission Denied by Server)
    : Triggered when the client IP or subnet CIDR is not covered by any
    ipRanges
    entry.
    • Remediation: Non-destructively append the client IP/subnet CIDR to
      nfsExportOptions
      , preserving all existing export rules to avoid breaking active mounts.
  3. EROFS (Read-only file system)
    : Triggered when
    accessMode
    is
    READ_ONLY
    but client writes are attempted.
  1. 检查
    fileShares[0].nfsExportOptions
    (如果为空,则应用默认的
    0.0.0.0/0
    READ_WRITE
    NO_ROOT_SQUASH
    规则)。
  2. EACCES(服务器拒绝权限)
    :当客户端IP或子网CIDR未被任何
    ipRanges
    条目覆盖时触发。
    • 修复方案:非破坏性地将客户端IP/子网CIDR追加到
      nfsExportOptions
      中,保留所有现有导出规则以避免中断正在运行的挂载。
  3. EROFS(文件系统只读)
    :当
    accessMode
    为
    READ_ONLY
    但客户端尝试写入时触发。

Step 4: VPC Ingress Firewall Inspection (
ETIMEDOUT
)

步骤4:VPC入站防火墙检查(
ETIMEDOUT
)

  1. Query ingress firewall rules in the VPC network (in the host project if Shared VPC).
  2. ETIMEDOUT (Connection Timed Out)
    : Triggered when priority-sorted ingress rules block or fail to allow TCP port
    2049
    (NFS) and TCP/UDP port
    111
    (rpcbind) from the client CIDR.
    • Remediation: Create an ingress firewall rule (
      ALLOW
      tcp:2049,udp:2049,tcp:111,udp:111
      ) from the client subnet CIDR.
  1. 查询VPC网络中的入站防火墙规则(如果是共享VPC,则在宿主项目中查询)。
  2. ETIMEDOUT(连接超时)
    :当按优先级排序的入站规则阻止或未允许来自客户端CIDR的TCP端口
    2049
    (NFS)和TCP/UDP端口
    111
    (rpcbind)的流量时触发。
    • 修复方案:创建一条入站防火墙规则,允许来自客户端子网CIDR的
      tcp:2049,udp:2049,tcp:111,udp:111
      流量。

Step 5: Cloud Audit Logs & Configuration Drift Scanning

步骤5:Cloud Audit日志与配置漂移扫描

Query Cloud Audit Admin Activity logs (
file.googleapis.com
) over the past 24 hours to check if an
UpdateInstance
operation modified export ACLs or networks:
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud logging read 'logName="projects/{project_id}/logs/cloudaudit.googleapis.com%2Factivity" AND protoPayload.serviceName="file.googleapis.com" AND protoPayload.resourceName=~".*{instance}.*"' \
    --project="{project_id}" --freshness="1d" --limit=5 --format="json"
  • Scope to Admin Activity: Always include
    cloudaudit.googleapis.com%2Factivity
    in
    logName
    so read-only
    GetInstance
    /
    ListInstances
    calls (
    data_access
    ) are not falsely flagged as administrative drift.
  • Drift is Informational: Recent
    UpdateInstance
    events explain when and by whom (
    principalEmail
    ,
    timestamp
    ) configuration changed, but only a failed export ACL or firewall check constitutes a mount blocker.
  • Single Bulk Query Rule: When scanning multiple instances (
    --all
    ), issue a single project-wide audit log query rather than per-instance queries in a loop.
  • GKE CSI Driver Logs: Optionally inspect
    resource.labels.container_name="gcp-filestore-driver"
    (
    severity>=ERROR
    ) for client-side mount errors.

查询过去24小时内的Cloud Audit管理活动日志(
file.googleapis.com
),检查是否有
UpdateInstance
操作修改了导出ACL或网络:
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud logging read 'logName="projects/{project_id}/logs/cloudaudit.googleapis.com%2Factivity" AND protoPayload.serviceName="file.googleapis.com" AND protoPayload.resourceName=~".*{instance}.*"' \
    --project="{project_id}" --freshness="1d" --limit=5 --format="json"
  • 限定管理活动范围:始终在
    logName
    中包含
    cloudaudit.googleapis.com%2Factivity
    ,这样只读的
    GetInstance
    /
    ListInstances
    调用(
    data_access
    )不会被误判为管理配置漂移。
  • 漂移仅作参考:最近的
    UpdateInstance
    事件可说明配置变更的时间和操作者(
    principalEmail
    、
    timestamp
    ),但只有导出ACL或防火墙检查失败才是挂载阻塞的原因。
  • 单次批量查询规则:扫描多个实例(
    --all
    )时,执行一次项目级的审计日志查询,而非循环逐个实例查询。
  • GKE CSI驱动日志:可选择性检查
    resource.labels.container_name="gcp-filestore-driver"
    (
    severity>=ERROR
    )以查找客户端侧的挂载错误。

Output Format & Mandatory Confirmation Gate

输出格式与强制确认关卡

Every diagnostic report MUST include a structured summary table and root cause analysis:
markdown
undefined
每份诊断报告必须包含结构化的汇总表和根因分析:
markdown
undefined

Filestore Diagnostic Report:
[instance-name]

Filestore诊断报告:
[instance-name]

ParameterValue
Instance ID
[instance-name]
(
[location]
, Tier:
[tier]
, State:
READY
)
Filestore IP & Network
[filestore-ip]
on VPC
[network-name]
(Host Project:
[host-project]
)
Port 2049 & Export ACLPort 2049:
OPEN / BLOCKED
| Export ACL:
PASS / REJECTED
Diagnostic VerdictHEALTHY or BLOCKED: [Root Cause]
参数值
实例ID
[instance-name]
(
[location]
,层级:
[tier]
,状态:
READY
)
Filestore IP与网络VPC
[network-name]
上的
[filestore-ip]
(宿主项目:
[host-project]
)
2049端口与导出ACL2049端口:
OPEN / BLOCKED
| 导出ACL:
PASS / REJECTED
诊断结论健康或阻塞:[根因]

Root Cause Analysis & Recommended Remediation

根因分析与建议修复方案

  • [Explanation of ETIMEDOUT (missing firewall rule on port 2049) vs EACCES (missing export ACL rule) and any recent UpdateInstance audit drift]
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud compute firewall-rules create ...
undefined
  • [解释ETIMEDOUT(2049端口缺少防火墙规则)与EACCES(缺少导出ACL规则)的区别,以及任何近期的UpdateInstance审计漂移]
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud compute firewall-rules create ...
undefined

Interactive Remediation Confirmation Gate (MANDATORY)

交互式修复确认关卡(强制)

  • NEVER execute write or remediation commands without explicit user confirmation.
  • Always conclude your response with an explicit confirmation question:
    "Would you like me to proceed with executing this remediation command for you? Please confirm to proceed."
  • 严禁在未获得用户明确确认的情况下执行写入或修复命令。
  • 回复末尾必须始终包含明确的确认问题:
    “你是否希望我继续为你执行这条修复命令?请确认以继续。”

References & Bundled Scripts

参考资料与捆绑脚本

  • Error Signatures & Architecture Matrix
  • Cloud Logging & Audit Drift Queries
  • VPC Network & Firewall Specification
  • Mock Fleet Data for Evaluations
  • scripts/quick_diagnose.py
    &
    scripts/diagnose_lib.py
    : Zero-dependency CLI runner and pure-Python evaluation engine.
  • 错误特征与架构矩阵
  • Cloud Logging与审计漂移查询
  • VPC网络与防火墙规范
  • 评估用模拟实例集数据
  • scripts/quick_diagnose.py
    与
    scripts/diagnose_lib.py
    :零依赖CLI运行器和纯Python评估引擎。