google-cloud-filestore-log-troubleshooting
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chinese<!-- disableFinding(LINE_OVER_80) -->
<!-- disableFinding(LINE_OVER_80) -->
Google Cloud Filestore Log-Based Troubleshooting
Google Cloud Filestore 基于日志的故障排查
Diagnoses, troubleshoots, and remediates Google Cloud Filestore client mount failures, permission errors (), and network timeouts () across projects.
EACCESETIMEDOUT用于跨项目诊断、排查和修复Google Cloud Filestore客户端挂载失败、权限错误()及网络超时()问题。
EACCESETIMEDOUTPrerequisites & Quick Start
前提条件与快速开始
Required IAM roles on target project(s) (and Shared VPC host project if applicable):
- Read: (instance & export ACLs),
roles/file.viewer(VPC firewall rules),roles/compute.networkViewer(Cloud Audit & GKE CSI logs),roles/logging.viewer(if using MCP tools).roles/mcp.toolUser - Write (Remediation only): (export ACL updates),
roles/file.editor(firewall rule creation).roles/compute.securityAdmin
Authenticate, verify billing/APIs, and configure your environment:
bash
gcloud auth login && gcloud auth application-default login
gcloud billing projects describe {project_id} --format="value(billingEnabled)"
gcloud services enable file.googleapis.com compute.googleapis.com logging.googleapis.com --quiet
gcloud config set project {project_id} && gcloud config set compute/region {region}目标项目(如适用,还包括共享VPC宿主项目)所需的IAM角色:
- 读取权限:(实例与导出ACL)、
roles/file.viewer(VPC防火墙规则)、roles/compute.networkViewer(Cloud Audit与GKE CSI日志)、roles/logging.viewer(如使用MCP工具)。roles/mcp.toolUser - 写入权限(仅用于修复):(导出ACL更新)、
roles/file.editor(防火墙规则创建)。roles/compute.securityAdmin
完成身份验证、验证计费/API并配置环境:
bash
gcloud auth login && gcloud auth application-default login
gcloud billing projects describe {project_id} --format="value(billingEnabled)"
gcloud services enable file.googleapis.com compute.googleapis.com logging.googleapis.com --quiet
gcloud config set project {project_id} && gcloud config set compute/region {region}Attribution
归属标注
Prefix every command provided or executed with the skill metrics environment:
gcloudbash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud filestore instances describe ...On direct REST API calls, append HTTP header: .
User-Agent: gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)在所有提供或执行的命令前添加技能指标环境变量:
gcloudbash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud filestore instances describe ...直接调用REST API时,添加HTTP头:。
User-Agent: gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)Conceptual & Informational Queries (CRITICAL)
概念性与信息查询(关键)
For purely conceptual, architectural, or educational questions (e.g., "What causes EACCES on Filestore?", "Why does GKE Node IP appear instead of Pod IP?", "What ports does Filestore require?", "Explain root squash"):
- Rule: Answer immediately using pre-trained knowledge and the workflow rules below. Answering directly minimizes tool latency and token usage when the user only seeks architectural guidance.
- Constraint: Do not execute external tool calls or API requests for basic knowledge questions.
对于纯概念性、架构性或教学类问题(例如“Filestore上出现EACCES的原因是什么?”、“为什么显示GKE节点IP而非Pod IP?”、“Filestore需要哪些端口?”、“解释root squash的作用”):
- 规则:使用预训练知识和以下工作流规则立即回答。 当用户仅需要架构指导时,直接回答可最大程度减少工具延迟和Token消耗。
- 约束:对于基础知识问题,不得执行外部工具调用或API请求。
Handling "No-Command" Constraints (CRITICAL)
“禁止执行命令”约束的处理(关键)
If the user prompt contains constraints like "Do not execute commands", "without executing", or "read-only":
- Rule: Strictly avoid calling to execute any shell, Python, or
run_commandcommands.gcloud - Discovery:
- First, check if Filestore MCP tools (,
get_instance) are available and use them (API calls, not command executions).list_instances - If MCP tools are unavailable, read only if the requested instance matches one of the evaluation scenarios (
references/mock-fleet-data.md,finance-share,shared-nfs,ml-data,data-hub). Never report mock data as live production state. If the instance is not listed there, state that live access is required and provide the exact commands for the user to run.prod-share - Fast-Path Stop Rule: Once you locate the target instance in , stop reading additional files immediately and formulate your response. Do NOT read
references/mock-fleet-data.md,scripts/quick_diagnose.py,scripts/diagnose_lib.py, or_internal/quick_diagnose_test.pyfiles when command execution is disabled, as inspecting code/test files wastes turns and triggers timeouts.EVAL.* - Explain the required diagnostic steps and output the exact attributed commands for manual execution.
- First, check if Filestore MCP tools (
- Mandatory User Confirmation Requirement: Even when command execution is disabled or the user asks only for recommendations, your response MUST STILL end with a clear question prompting the user for explicit confirmation before applying any remediation (e.g., "Would you like me to proceed with creating the VPC ingress firewall rule ? Please confirm to proceed.").
[rule_name]
如果用户提示包含“不得执行命令”、“无需执行”或“只读”等约束:
- 规则:**严格禁止调用**执行任何Shell、Python或
run_command命令。gcloud - 排查方式:
- 首先检查Filestore MCP工具(、
get_instance)是否可用并使用它们(是API调用,而非命令执行)。list_instances - 如果MCP工具不可用,仅当请求的实例匹配评估场景(、
finance-share、shared-nfs、ml-data、data-hub)之一时,才读取prod-share。严禁将模拟数据报告为实时生产状态。如果实例未在其中列出,说明需要实时访问权限,并提供供用户运行的精确命令。references/mock-fleet-data.md - 快速停止规则:在中找到目标实例后,立即停止读取其他文件并组织回复。当命令执行被禁用时,请勿读取
references/mock-fleet-data.md、scripts/quick_diagnose.py、scripts/diagnose_lib.py或_internal/quick_diagnose_test.py文件,因为检查代码/测试文件会浪费轮次并导致超时。EVAL.* - 说明所需的诊断步骤,并输出精确的带归属标识的命令供手动执行。
- 首先检查Filestore MCP工具(
- 强制用户确认要求:即使命令执行被禁用或用户仅要求建议,你的回复仍必须以明确的问题结尾,在应用任何修复前请求用户明确确认(例如:“你是否希望我继续创建VPC入站防火墙规则?请确认以继续。”)。
[rule_name]
Multi-Runtime Execution Options
多运行时执行选项
Option 1: Bundled Python CLI Script (Recommended for CLI / Terminal Agents)
选项1:捆绑的Python CLI脚本(推荐用于CLI/终端Agent)
bash
undefinedbash
undefinedSingle instance diagnosis
Single instance diagnosis
python3 scripts/quick_diagnose.py --instance="<INSTANCE_ID>" --location="<LOCATION_OR_ZONE>"
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]
python3 scripts/quick_diagnose.py --instance="<INSTANCE_ID>" --location="<LOCATION_OR_ZONE>"
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]
[--project="<PROJECT_ID>"] [--client-ip="<CLIENT_IP>"] [--client-subnet="<CLIENT_SUBNET_CIDR>"]
Bulk project-wide fleet diagnosis
Bulk project-wide fleet diagnosis
python3 scripts/quick_diagnose.py --all --project="<PROJECT_ID>" [--json]
| Flag | Purpose |
| :--- | :--- |
| `--instance`, `--location` | Filestore instance ID and region/zone (`--zone` is a legacy alias). Required unless `--all`. |
| `--project` | GCP project ID (defaults to active `gcloud` project). |
| `--client-ip` / `--client-subnet` | Client IP or CIDR to evaluate against export ACLs and ingress firewall rules. |
| `--json` | Emit machine-readable JSON on stdout (narrative report goes to stderr). |
| `--apply-fix` | Execute generated remediation commands. **Only pass after explicit user confirmation.** |python3 scripts/quick_diagnose.py --all --project="<PROJECT_ID>" [--json]
| 标志 | 用途 |
| :--- | :--- |
| `--instance`, `--location` | Filestore实例ID和区域/可用区(`--zone`是旧版别名)。除非使用`--all`,否则为必填项。 |
| `--project` | GCP项目ID(默认为当前激活的`gcloud`项目)。 |
| `--client-ip` / `--client-subnet` | 用于评估导出ACL和入站防火墙规则的客户端IP或CIDR。 |
| `--json` | 在标准输出输出机器可读的JSON(叙述性报告输出到标准错误)。 |
| `--apply-fix` | 执行生成的修复命令。**仅在获得用户明确确认后添加此参数。** |Option 2: Filestore MCP Tools / REST API / gcloud
CLI
gcloud选项2:Filestore MCP工具 / REST API / gcloud
CLI
gcloud- MCP Tools: Call and inspect
get_instance(name='projects/{project_id}/locations/{location}/instances/{instance_id}')andfileShares[0].nfsExportOptions.networks[0].network - Native REST API (): Invoke
call_gcp_api,service="file",version="v1".resource_path="projects/{project_id}/locations/{location}/instances/{instance_id}" - Standard :
gcloudbashCLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \ gcloud filestore instances describe {instance_id} --location={location} --project={project_id} --format=json
- MCP工具:调用并检查
get_instance(name='projects/{project_id}/locations/{location}/instances/{instance_id}')和fileShares[0].nfsExportOptions。networks[0].network - 原生REST API():调用
call_gcp_api、service="file"、version="v1"。resource_path="projects/{project_id}/locations/{location}/instances/{instance_id}" - 标准:
gcloudbashCLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \ gcloud filestore instances describe {instance_id} --location={location} --project={project_id} --format=json
Core Operational Workflow
核心操作工作流
Step 1: Parameter Extraction & GKE Node Architecture
步骤1:参数提取与GKE节点架构
- Extract ,
instance(region/zone),location, andproject/client_ip. If target parameters are missing, list instances or ask the user to confirm.client_subnet - GKE Architecture Rule: PersistentVolumes are mounted by the Linux kernel on the GKE Worker Node, not inside the Pod network namespace. Even if Pod IPs () are allowlisted, the NFS server sees traffic originating from the GKE Node Internal IP. Always evaluate and allowlist the GKE Node Subnet CIDR in
10.4.0.0/14and VPC firewall rules.nfsExportOptions
- 提取、
instance(区域/可用区)、location以及project/client_ip。如果缺少目标参数,列出实例或要求用户确认。client_subnet - GKE架构规则:PersistentVolume由Linux内核挂载在GKE工作节点上,而非Pod网络命名空间内。即使Pod IP()在允许列表中,NFS服务器看到的流量也来自GKE节点内部IP。始终在
10.4.0.0/14和VPC防火墙规则中评估并加入GKE节点子网CIDR的允许列表。nfsExportOptions
Step 2: Instance Metadata & Shared VPC Resolution
步骤2:实例元数据与共享VPC解析
- Verify instance is
state(report state blocker ifREADY,CREATING, orDELETING).ERROR - Extract primary Filestore IP () and VPC network URI.
networks[0].ipAddresses[0] - If references
networks[0].network, resolveprojects/{host_project}/global/networks/{network}as the Shared VPC host project for firewall queries.{host_project}
- 验证实例为
state(如果状态为READY、CREATING或DELETING,报告状态阻塞问题)。ERROR - 提取Filestore主IP()和VPC网络URI。
networks[0].ipAddresses[0] - 如果引用
networks[0].network,则将projects/{host_project}/global/networks/{network}解析为共享VPC宿主项目以用于防火墙查询。{host_project}
Step 3: Export ACL Evaluation (EACCES
vs EROFS
)
EACCESEROFS步骤3:导出ACL评估(EACCES
与EROFS
)
EACCESEROFS- Inspect (if empty, default
fileShares[0].nfsExportOptions0.0.0.0/0READ_WRITEapplies).NO_ROOT_SQUASH - : Triggered when the client IP or subnet CIDR is not covered by any
EACCES (Permission Denied by Server)entry.ipRanges- Remediation: Non-destructively append the client IP/subnet CIDR to , preserving all existing export rules to avoid breaking active mounts.
nfsExportOptions
- Remediation: Non-destructively append the client IP/subnet CIDR to
- : Triggered when
EROFS (Read-only file system)isaccessModebut client writes are attempted.READ_ONLY
- 检查(如果为空,则应用默认的
fileShares[0].nfsExportOptions0.0.0.0/0READ_WRITE规则)。NO_ROOT_SQUASH - :当客户端IP或子网CIDR未被任何
EACCES(服务器拒绝权限)条目覆盖时触发。ipRanges- 修复方案:非破坏性地将客户端IP/子网CIDR追加到中,保留所有现有导出规则以避免中断正在运行的挂载。
nfsExportOptions
- 修复方案:非破坏性地将客户端IP/子网CIDR追加到
- :当
EROFS(文件系统只读)为accessMode但客户端尝试写入时触发。READ_ONLY
Step 4: VPC Ingress Firewall Inspection (ETIMEDOUT
)
ETIMEDOUT步骤4:VPC入站防火墙检查(ETIMEDOUT
)
ETIMEDOUT- Query ingress firewall rules in the VPC network (in the host project if Shared VPC).
- : Triggered when priority-sorted ingress rules block or fail to allow TCP port
ETIMEDOUT (Connection Timed Out)(NFS) and TCP/UDP port2049(rpcbind) from the client CIDR.111- Remediation: Create an ingress firewall rule (
ALLOW) from the client subnet CIDR.tcp:2049,udp:2049,tcp:111,udp:111
- Remediation: Create an ingress firewall rule (
- 查询VPC网络中的入站防火墙规则(如果是共享VPC,则在宿主项目中查询)。
- :当按优先级排序的入站规则阻止或未允许来自客户端CIDR的TCP端口
ETIMEDOUT(连接超时)(NFS)和TCP/UDP端口2049(rpcbind)的流量时触发。111- 修复方案:创建一条入站防火墙规则,允许来自客户端子网CIDR的流量。
tcp:2049,udp:2049,tcp:111,udp:111
- 修复方案:创建一条入站防火墙规则,允许来自客户端子网CIDR的
Step 5: Cloud Audit Logs & Configuration Drift Scanning
步骤5:Cloud Audit日志与配置漂移扫描
Query Cloud Audit Admin Activity logs () over the past 24 hours to check if an operation modified export ACLs or networks:
file.googleapis.comUpdateInstancebash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud logging read 'logName="projects/{project_id}/logs/cloudaudit.googleapis.com%2Factivity" AND protoPayload.serviceName="file.googleapis.com" AND protoPayload.resourceName=~".*{instance}.*"' \
--project="{project_id}" --freshness="1d" --limit=5 --format="json"- Scope to Admin Activity: Always include in
cloudaudit.googleapis.com%2Factivityso read-onlylogName/GetInstancecalls (ListInstances) are not falsely flagged as administrative drift.data_access - Drift is Informational: Recent events explain when and by whom (
UpdateInstance,principalEmail) configuration changed, but only a failed export ACL or firewall check constitutes a mount blocker.timestamp - Single Bulk Query Rule: When scanning multiple instances (), issue a single project-wide audit log query rather than per-instance queries in a loop.
--all - GKE CSI Driver Logs: Optionally inspect (
resource.labels.container_name="gcp-filestore-driver") for client-side mount errors.severity>=ERROR
查询过去24小时内的Cloud Audit管理活动日志(),检查是否有操作修改了导出ACL或网络:
file.googleapis.comUpdateInstancebash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud logging read 'logName="projects/{project_id}/logs/cloudaudit.googleapis.com%2Factivity" AND protoPayload.serviceName="file.googleapis.com" AND protoPayload.resourceName=~".*{instance}.*"' \
--project="{project_id}" --freshness="1d" --limit=5 --format="json"- 限定管理活动范围:始终在中包含
logName,这样只读的cloudaudit.googleapis.com%2Factivity/GetInstance调用(ListInstances)不会被误判为管理配置漂移。data_access - 漂移仅作参考:最近的事件可说明配置变更的时间和操作者(
UpdateInstance、principalEmail),但只有导出ACL或防火墙检查失败才是挂载阻塞的原因。timestamp - 单次批量查询规则:扫描多个实例()时,执行一次项目级的审计日志查询,而非循环逐个实例查询。
--all - GKE CSI驱动日志:可选择性检查(
resource.labels.container_name="gcp-filestore-driver")以查找客户端侧的挂载错误。severity>=ERROR
Output Format & Mandatory Confirmation Gate
输出格式与强制确认关卡
Every diagnostic report MUST include a structured summary table and root cause analysis:
markdown
undefined每份诊断报告必须包含结构化的汇总表和根因分析:
markdown
undefinedFilestore Diagnostic Report: [instance-name]
[instance-name]Filestore诊断报告:[instance-name]
[instance-name]| Parameter | Value |
|---|---|
| Instance ID | |
| Filestore IP & Network | |
| Port 2049 & Export ACL | Port 2049: |
| Diagnostic Verdict | HEALTHY or BLOCKED: [Root Cause] |
| 参数 | 值 |
|---|---|
| 实例ID | |
| Filestore IP与网络 | VPC |
| 2049端口与导出ACL | 2049端口: |
| 诊断结论 | 健康或阻塞:[根因] |
Root Cause Analysis & Recommended Remediation
根因分析与建议修复方案
- [Explanation of ETIMEDOUT (missing firewall rule on port 2049) vs EACCES (missing export ACL rule) and any recent UpdateInstance audit drift]
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud compute firewall-rules create ...undefined- [解释ETIMEDOUT(2049端口缺少防火墙规则)与EACCES(缺少导出ACL规则)的区别,以及任何近期的UpdateInstance审计漂移]
bash
CLOUDSDK_METRICS_ENVIRONMENT="gcs-skills gcs-skills/1.0 (skill:google-cloud-filestore-log-troubleshooting)" \
gcloud compute firewall-rules create ...undefinedInteractive Remediation Confirmation Gate (MANDATORY)
交互式修复确认关卡(强制)
- NEVER execute write or remediation commands without explicit user confirmation.
- Always conclude your response with an explicit confirmation question:
"Would you like me to proceed with executing this remediation command for you? Please confirm to proceed."
- 严禁在未获得用户明确确认的情况下执行写入或修复命令。
- 回复末尾必须始终包含明确的确认问题:
“你是否希望我继续为你执行这条修复命令?请确认以继续。”
References & Bundled Scripts
参考资料与捆绑脚本
- Error Signatures & Architecture Matrix
- Cloud Logging & Audit Drift Queries
- VPC Network & Firewall Specification
- Mock Fleet Data for Evaluations
- &
scripts/quick_diagnose.py: Zero-dependency CLI runner and pure-Python evaluation engine.scripts/diagnose_lib.py
- 错误特征与架构矩阵
- Cloud Logging与审计漂移查询
- VPC网络与防火墙规范
- 评估用模拟实例集数据
- 与
scripts/quick_diagnose.py:零依赖CLI运行器和纯Python评估引擎。scripts/diagnose_lib.py