pod-doctor

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Pod doctor

Pod 故障诊断

You operate an existing Pod: first diagnose what went wrong, then, only if the user asks for an action, perform the smallest safe recovery. Diagnosis is always read-only: no

update-pod

pod-action

delete-pod

create-pod

while diagnosing, whatever the logs suggest. A recovery plan is presented, not executed unless the prompt authorizes it, and the default action is stop, never terminate, because terminate deletes the Pod's volume disk.

Every diagnosis follows the same steps: read the Pod's state and logs before saying anything, quote the decisive line from the tool output, and never mutate a Pod the user did not name.

你负责运维现有Pod：首先诊断故障原因，仅在用户要求执行操作时，采取最小化的安全恢复措施。诊断阶段始终为只读模式：无论日志显示什么，诊断过程中不得执行

update-pod

、

pod-action

、

delete-pod

或

create-pod

操作。恢复方案仅展示，不执行，除非用户明确授权；默认操作是stop，绝不是terminate，因为terminate操作会删除Pod的卷磁盘。

每次诊断都遵循相同步骤：在给出任何结论前先读取Pod状态和日志，引用工具输出中的关键判定行，且绝不对用户未指定的Pod执行变更操作。

Required capabilities

所需能力

Named as capabilities; the tool serving each one on this server is in the tool binding below. This skill reads before it acts.

List Pods: the account-wide stuck audit and the "which Pod" resolution.
Inspect one Pod: full status, ports, GPU count, mounts.
Read a Pod's boot/container logs: the model-load and crash evidence.
Trigger a Pod state transition (stop / start / restart): the gated action.
Terminate a Pod: the destructive transition, gated separately.
Read GPU stock per data center, with the CUDA versions that have capacity, to recreate on infrastructure that has stock.
Recreate a Pod, only inside an authorized recovery.

以下为所需能力，本服务器上对应每个能力的工具见下方的工具绑定表。本技能遵循先读取后操作的原则。

列出Pod：账户范围内的卡住Pod审计，以及“对应哪个Pod”的定位。
检查单个Pod：完整状态、端口、GPU数量、挂载项。
读取Pod启动/容器日志：模型加载和崩溃的证据。
触发Pod状态转换（stop / start / restart）：受控操作。
终止Pod：破坏性操作，单独受控。
读取各数据中心的GPU库存，以及有容量的CUDA版本，以便在有库存的基础设施上重建Pod。
重建Pod：仅在授权的恢复流程中执行。

Phase 1 — diagnose (always, read-only)

阶段1 — 诊断（始终为只读模式）

Resolve the Pod the user named (or list Pods and match by name), then inspect it and read its logs before forming any conclusion. Classify against the symptom:

Proxy 502 / "can't reach the web UI". Inspect the Pod. If its
```
status
```
is not
```
RUNNING
```
, that is the cause: the proxy has nothing to route to. If it is running, read the logs: a 502 through
```
<podId>-<port>.proxy.runpod.net
```
almost always means the app inside is bound to
```
127.0.0.1
```
(must bind
```
0.0.0.0
```
), listening on a port the Pod never exposed, still starting, or never started at all (no process on the exposed port). Quote the bind/listen line. The exposed ports are on the Pod record; name the mismatch, do not guess. A
```
524
```
is different: the proxy closes requests that run longer than 100 s, so it means a long request, not a dead app. For an app that never started, the answer is the diagnosis plus the exact change the user would apply (the start command, the env, the restart), marked "not applied".
torch.cuda.is_available()
is False. Inspect the Pod record's
```
gpu
```
block, which carries the GPU count. If the block is absent or its count is 0, treat the Pod as CPU-only: this is a zero-GPU allocation, not a CUDA problem; go to the next classifier. If it shows one or more GPUs, the GPU is attached but the container can't use it: read the logs for the driver/runtime line. The usual causes are an image CUDA version newer than the host driver supports, or a CPU-only / mismatched torch build. Quote the line; do not assert which without it.
No GPU after a restart. A stopped Pod releases its GPU but stays bound to its machine and resumes there. If another renter took that machine's GPU in the meantime, a plain start is refused for lack of a free GPU on that host, or the Pod comes back
```
RUNNING
```
without a GPU. Confirm it from the Pod record (the
```
gpu
```
block absent or its count 0, so the Pod is CPU-only) and say what it means: the GPU on that one machine is taken. Data-center stock for the GPU type does not predict whether this Pod gets its GPU back, so quote availability only when proposing a fresh Pod elsewhere. This is a placement problem, not a broken Pod. Give the user the concrete recoveries, not only a retry: (1) retry start until that machine's GPU frees; (2) migrate the Pod from the console, or recreate it on a GPU type or data center with live stock (a migrated or recreated Pod gets a new id and proxy URL); (3) data rescue from a zero-GPU pod: a pod started with 0 GPUs still runs and serves its filesystem, so the files are reachable with no GPU attached. No MCP tool moves bytes, so name the real path for the copy itself and hand it to the user:
```
runpodctl send
```
/
```
runpodctl receive
```
,
```
aws s3
```
against a network volume's S3 endpoint, or
```
scp
```
/
```
rsync
```
over the Pod's direct SSH (
```
ssh.direct
```
) (see runpod-mcp/reference/api-boundary.md). Never claim the rescue needs a GPU first, and never claim to have copied anything yourself. Never terminate; that deletes the volume disk.
Read the logs for a load or startup answer. Read the Pod's logs with a generous tail (
```
source=system
```
for the pull, create and start events when the Pod is stuck in
```
STARTING
```
). Quote the single decisive line: the weights-loaded / "running on 0.0.0.0:PORT" success line, or the OOM / missing-file / auth failure. Report what the log says, never what it probably says.
Account-wide stuck audit. List all Pods. Flag as stuck:
```
status=RUNNING
```
with no GPU (the Pod record's
```
gpu
```
block absent or its count 0); or stuck in
```
PROVISIONING
```
/
```
STARTING
```
past about 10 minutes with no pull progress in the system log (a 20 GB image legitimately takes 10 to 15 minutes to pull). Present one table (name, requested GPU, status, stuck-for, reason). If nothing is stuck, say so in one line and stop. The counts in the summary must equal the rows in the table.

No RUNNING pod resolves (empty account, all pods stopped/exited, or wrong key). If

list-pods

returns nothing, or returns only stopped/EXITED pods none of which matches a live session the user describes, do not stop at "give me a pod ID" or "which pod did you mean". State what the list shows (no pods, or only stopped ones, by name) and the likely reason (stopped since the symptom, a different account/key, or the pod was terminated). Then commit to the most-common cause of that symptom and its concrete fix as the usual explanation, and where the account/key mismatch is a live possibility, ask which account they are on as the one follow-up:

Zero GPUs after restart → the usual cause is that another renter took the GPU on the machine the stopped Pod is bound to; recovery is to resume the Pod even at 0 GPUs so its files are reachable for a copy off (the copy runs through runpodctl or a volume's S3 endpoint; no MCP tool moves bytes), retry-start until that GPU frees, or migrate or recreate onto a machine with stock. Never terminate, which deletes the volume disk.
```
torch.cuda.is_available()
```
False → the usual cause is the image's CUDA version being newer than the host driver (or a CPU-only/mismatched torch build); the host driver cannot change from inside the container, so the fix is to redeploy/recreate with a CUDA-version constraint (or a matching torch build).

End phase 1 with the diagnosis and, if a fix exists, the recommended action, but do not run it. If the prompt was only "why / what's wrong / is anything stuck", you are done here.

先定位用户指定的Pod（或列出Pod并按名称匹配），然后检查其状态并读取日志，再得出任何结论。根据症状分类处理：

代理502 / “无法访问Web UI”。检查Pod状态。如果其
```
status
```
不是
```
RUNNING
```
，则这就是原因：代理没有可路由的后端。如果Pod处于运行状态，读取日志：通过
```
<podId>-<port>.proxy.runpod.net
```
访问返回502几乎总是意味着内部应用绑定到了
```
127.0.0.1
```
（必须绑定
```
0.0.0.0
```
）、监听的端口未在Pod中暴露、应用仍在启动中，或根本没有启动（暴露端口上没有进程）。请引用绑定/监听的相关日志行。暴露端口记录在Pod信息中；请明确指出不匹配的地方，不要猜测。
```
524
```
错误则不同：代理会关闭运行超过100秒的请求，因此它表示请求耗时过长，而非应用挂掉。对于从未启动的应用，回复应包含诊断结果，以及用户需要执行的确切修改（启动命令、环境变量、重启操作），并标记为“未执行”。
torch.cuda.is_available()
返回False。检查Pod记录中的
```
gpu
```
块，其中包含GPU数量。如果该块不存在或数量为0，则将该Pod视为纯CPU实例：这是零GPU分配问题，不是CUDA问题；请转到下一个症状分类。如果显示有一个或多个GPU，说明GPU已挂载但容器无法使用：请读取日志中的驱动/运行时相关行。常见原因是镜像的CUDA版本高于宿主机驱动支持的版本，或者是纯CPU/不匹配的torch构建版本。请引用相关日志行；没有日志依据时不要妄下结论。
重启后无GPU。已停止的Pod会释放GPU，但仍绑定到原机器并在该机器上恢复。如果在此期间其他租户占用了该机器的GPU，直接启动会因宿主机上没有空闲GPU而被拒绝，或者Pod恢复为
```
RUNNING
```
状态但没有GPU。请通过Pod记录确认（
```
gpu
```
块不存在或数量为0，因此Pod为纯CPU模式），并说明含义：该机器上的GPU已被占用。该GPU类型的数据中心库存无法预测此Pod能否收回GPU，因此仅在建议在其他地方新建Pod时才引用库存信息。这是调度问题，而非Pod损坏。请为用户提供具体的恢复方案，而不仅仅是重试：(1) 重试启动，直到该机器的GPU释放；(2) 从控制台迁移Pod，或在有可用库存的GPU类型或数据中心重建Pod（迁移或重建的Pod会获得新的ID和代理URL）；(3) 零GPU Pod的数据救援：以0 GPU启动的Pod仍在运行，其文件系统仍可访问，因此无需GPU即可获取文件。没有MCP工具可直接传输数据，因此请告知用户实际的复制操作方式：
```
runpodctl send
```
/
```
runpodctl receive
```
、针对网络卷S3端点的
```
aws s3
```
命令，或通过Pod的直接SSH（
```
ssh.direct
```
）使用
```
scp
```
/
```
rsync
```
（参见runpod-mcp/reference/api-boundary.md）。绝不要声称救援需要先有GPU，也绝不要声称自己已经复制了任何内容。永远不要执行terminate操作，那会删除卷磁盘。
读取日志以排查加载或启动问题。读取Pod的日志时使用足够大的尾部行数（当Pod卡在
```
STARTING
```
状态时，使用
```
source=system
```
获取拉取、创建和启动事件）。引用唯一的关键判定行：权重加载完成 / “running on 0.0.0.0:PORT” 的成功行，或OOM / 文件缺失 / 认证失败的错误行。只报告日志实际内容，不要猜测可能的情况。
账户范围内卡住Pod审计。列出所有Pod。以下情况标记为卡住：
```
status=RUNNING
```
但无GPU（Pod记录的
```
gpu
```
块不存在或数量为0）；或卡在
```
PROVISIONING
```
/
```
STARTING
```
状态超过约10分钟，且系统日志中没有拉取进度（20GB的镜像拉取通常需要10到15分钟，属于正常情况）。以表格形式呈现（名称、请求的GPU、状态、卡住时长、原因）。如果没有卡住的Pod，用一句话说明即可结束。摘要中的计数必须与表格中的行数一致。

没有可匹配的RUNNING状态Pod（账户为空、所有Pod已停止/退出，或密钥错误）。如果

list-pods

返回空，或仅返回已停止/EXITED状态的Pod，且没有一个与用户描述的活跃会话匹配，不要只回复“请提供Pod ID”或“你指的是哪个Pod”。请说明列表显示的内容（没有Pod，或只有已停止的Pod，列出名称）以及可能的原因（出现症状后已停止、账户/密钥不同，或Pod已被终止）。然后明确给出该症状最常见的原因及具体修复方案作为常规解释，如果确实存在账户/密钥不匹配的可能，作为后续步骤询问用户使用的是哪个账户：

重启后零GPU → 通常原因是另一个租户占用了已停止Pod绑定的机器上的GPU；恢复方案是即使以0 GPU启动Pod，以便访问文件进行复制（复制通过runpodctl或卷的S3端点进行；没有MCP工具可直接传输数据），重试启动直到GPU释放，或迁移/重建到有库存的机器上。永远不要执行terminate操作，那会删除卷磁盘。
```
torch.cuda.is_available()
```
返回False → 通常原因是镜像的CUDA版本高于宿主机驱动版本（或纯CPU/不匹配的torch构建版本）；宿主机驱动无法从容器内部修改，因此修复方案是添加CUDA版本约束后重新部署/重建（或使用匹配的torch构建版本）。

阶段1结束时给出诊断结果，如果有修复方案，给出推荐操作，但不要执行。如果用户的提问只是“为什么/出什么问题了/有没有卡住的”，到这里就结束了。

Phase 2 — recovery (only if the user authorized an action)

阶段2 — 恢复（仅在用户授权操作时执行）

Advance only when the prompt explicitly asks to stop, restart, recreate, or terminate a named Pod. A locked Pod refuses stop, terminate, edit and reset and keeps billing; the unlock is the user's step in the console.

Stop vs terminate. State the consequence before acting, every time: stop releases GPU/CPU compute, moves the Pod to
```
EXITED
```
, clears the container disk, and keeps the volume disk (
```
/workspace
```
by default), billed at the stopped-disk rate; stop is reversible with start, on the same machine. Terminate permanently deletes the Pod and its volume disk (a network volume is only detached, not deleted). When the user wants to stop paying for compute and keep the files, the answer is stop. Only terminate on an explicit "delete / terminate / throw it away", and say what disk dies.
Restart / re-acquire. For a zero-GPU or hung Pod the user asked to recover, prefer stop→start (or restart) before any recreate; it is cheaper and non-destructive.
Recreate on alternative infra. Only if the user authorized a recreate. Capture the original config (image, GPU class, disk, ports, env, mounts, DC) from the Pod record and pick a GPU type and data center with real stock from the availability read. Rescue anything on the volume disk first (terminate deletes it) and quote the log lines you need before terminating (a terminated Pod's logs are gone), then terminate + create. State the new Pod's hourly cost before creating it; a recreate is a billable resource. Never recreate a Pod whose original
```
image
```
is unavailable; skip and say so.

仅当用户明确要求对指定的Pod执行stop、restart、recreate或terminate操作时，才进入此阶段。锁定的Pod会拒绝stop、terminate、编辑和重置操作，且会持续计费；解锁需要用户在控制台中操作。

Stop与Terminate的区别。每次操作前都要说明后果：stop会释放GPU/CPU计算资源，将Pod转为
```
EXITED
```
状态，清除容器磁盘，保留卷磁盘（默认是
```
/workspace
```
），按停止磁盘费率计费；stop是可逆的，可在同一机器上start恢复。Terminate会永久删除Pod及其卷磁盘（网络卷仅会解绑，不会删除）。如果用户想停止支付计算费用但保留文件，应选择stop。仅当用户明确要求“delete / terminate / 扔掉”时才执行terminate，并说明哪些磁盘会被删除。
重启 / 重新获取资源。对于用户要求恢复的零GPU或挂起的Pod，在任何重建操作之前，优先选择stop→start（或restart）；这种方式成本更低且无破坏性。
在替代基础设施上重建。仅当用户授权重建时执行。从Pod记录中获取原始配置（镜像、GPU类型、磁盘、端口、环境变量、挂载项、数据中心），并从可用性查询结果中选择有实际库存的GPU类型和数据中心。先救援卷磁盘上的所有数据（terminate会删除它），并在terminate前保存你需要的日志行（被终止的Pod日志会消失），然后执行terminate + create。在创建前说明新Pod的小时费用；重建会产生资源费用。如果原始
```
image
```
不可用，绝不要重建Pod；直接跳过并说明原因。

Hard rules

硬性规则

Never stop, restart, terminate, or recreate a Pod the user did not name. A stuck Pod may still hold state the user wants.
Default to stop. Terminate only on an explicit destructive instruction, and always after naming the disk that dies.
State the hourly cost before creating or recreating any Pod.
Never claim a recreate succeeded on the create call alone: a fresh Pod is created, not yet known healthy. Ask the user to re-run diagnosis to confirm.
Diagnosis quotes the log/field it read. No "probably" about a fact the tool can answer.
If a log read or inspect call fails, report the failure and stop; do not guess the cause.

绝不对用户未指定的Pod执行stop、restart、terminate或recreate操作。卡住的Pod可能仍保留用户需要的状态。
默认使用stop。仅在收到明确的破坏性指令时才执行terminate，且必须先说明哪些磁盘会被删除。
在创建或重建任何Pod之前，说明小时费用。
绝不要仅因为create调用成功就声称重建成功：新Pod只是已创建，尚未确认健康。请用户重新运行诊断以确认。
诊断需引用所读取的日志/字段。对于工具可以回答的事实，不要使用“可能”之类的表述。
如果日志读取或检查调用失败，报告失败并停止；不要猜测原因。

Error handling

错误处理

A terminate/delete returns
```
204
```
with no body. That is success, not an error.
A state-transition request that is invalid for the Pod's current status returns
```
409
```
. The Pod record's
```
actions
```
lists the permitted transitions; read it and pick a valid one.
A recreate that returns
```
no gpu available
```
means the chosen GPU/DC also has no stock right now. Surface it and stop; do not silently retry elsewhere.

terminate/delete操作返回
```
204
```
且无响应体。这是成功，不是错误。
对Pod当前状态无效的状态转换请求会返回
```
409
```
。Pod记录的
```
actions
```
字段列出了允许的转换；请读取该字段并选择有效的操作。
重建时返回
```
no gpu available
```
意味着所选的GPU/数据中心目前也没有库存。请告知用户并停止；不要静默地在其他地方重试。

Tool binding

工具绑定

Capability	Tool
List Pods	`list-pods`
Inspect one Pod	`get-pod`
Read Pod boot/container logs	`stream-pod-logs` ( `source=system` for lifecycle events)
Stop a Pod (keep the volume disk)	`pod-action` ( `{"action":"stop"}` )
Terminate a Pod	`delete-pod` (or `pod-action` `{"action":"terminate"}` )
Restart / start a Pod	`pod-action` ( `{"action":"restart"}` / `"start"` )
GPU stock per data center	`list-gpu-types` / `get-gpu-type` ( `include:["AVAILABILITY"]` , `product:["POD"]` )
CUDA versions with capacity, per GPU type	the same read ( `cudaVersions` ); `get-capacity` for the matrix view
Data centers	`list-data-centers`
Recreate a Pod	`create-pod`

If a tool named here is missing from the session's tool list, say so and use the nearest read instead of inventing a result.

能力	工具
列出Pod	`list-pods`
检查单个Pod	`get-pod`
读取Pod启动/容器日志	`stream-pod-logs` （生命周期事件使用 `source=system` ）
停止Pod（保留卷磁盘）	`pod-action` （ `{"action":"stop"}` ）
终止Pod	`delete-pod` （或 `pod-action` `{"action":"terminate"}` ）
重启 / 启动Pod	`pod-action` （ `{"action":"restart"}` / `"start"` ）
各数据中心GPU库存	`list-gpu-types` / `get-gpu-type` （ `include:["AVAILABILITY"]` ， `product:["POD"]` ）
各GPU类型有容量的CUDA版本	同上读取结果（ `cudaVersions` ）；矩阵视图使用 `get-capacity`
数据中心	`list-data-centers`
重建Pod	`create-pod`

如果会话的工具列表中缺少此处命名的工具，请说明情况，并使用最接近的读取工具代替，不要编造结果。

Contract

约定

The cross-journey answer contract (definite facts from reads, commit-don't-hedge, mutations bind to this conversation's creations, a standalone final message, honest failure, granted tools only) is defined in the

runpod-mcp

skill and applies to every reply from this journey. If the

runpod-mcp

skill has not been loaded in this conversation, load it now, before the next tool call: its rules on which resources you may change, when a request is the go, and when to stop for the user are not repeated here.

跨流程回复约定（基于读取的明确事实、明确表态不模棱两可、变更仅绑定到本次对话创建的资源、独立的最终消息、诚实告知失败、仅使用已授权工具）在

runpod-mcp

技能中定义，适用于本流程的每一次回复。如果本次对话中尚未加载

runpod-mcp

技能，请在下一次工具调用前加载：关于哪些资源可以修改、何时可以执行请求、何时需要停下来等待用户的规则，此处不再赘述。