runpod-vllm-claude-code

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

RunPod vLLM + Claude Code bridge

RunPod vLLM + Claude Code 桥接方案

Two things are being solved together here, and it's worth being clear about which is which:

Getting a model running on RunPod. Straightforward — pick a GPU, start a vLLM container, wait for health.
Getting Claude Code to actually talk to it. Not straightforward — Claude Code speaks Anthropic's Messages API (
```
/v1/messages
```
, content blocks, SSE events). vLLM speaks OpenAI's
```
/v1/chat/completions
```
. Nothing bridges these two by default.
```
ANTHROPIC_BASE_URL
```
lets Claude Code point at any server that implements the Messages API shape — so a small translation proxy in the middle is what makes "point Claude Code at my RunPod box" actually work.
```
scripts/anthropic-openai-proxy.js
```
is that proxy, already fixed for the two protocol mismatches that will otherwise burn an hour of retries (see Gotchas below). Use it as-is; don't re-derive it from scratch.

This is exploratory/disposable infra by nature — a GPU pod costs real money per hour the moment it exists. Before creating one, tell the user which GPU and roughly $/hr, unless they already named the machine and accepted the cost.

这里同时解决了两个问题，有必要先明确区分：

在RunPod上运行模型。流程很直接——选择GPU，启动vLLM容器，等待健康检查通过。
让Claude Code真正与之通信。这并不简单——Claude Code使用Anthropic的Messages API（
```
/v1/messages
```
、内容块、SSE事件），而vLLM使用OpenAI的
```
/v1/chat/completions
```
接口。默认情况下没有组件能桥接这两套API。
```
ANTHROPIC_BASE_URL
```
允许Claude Code指向任何实现了Messages API格式的服务器——因此中间加一个小型转换代理，才能真正实现“让Claude Code连接到我的RunPod实例”。
```
scripts/anthropic-openai-proxy.js
```
就是这个代理，已经修复了两处会浪费一小时重试的协议不兼容问题（见下文“已知坑点”）。直接使用即可，不要从头重新实现。

这本质上是探索性/一次性的基础设施——GPU Pod一旦创建就会按小时产生实际费用。在创建之前，需要告知用户使用的GPU型号和大致的每小时费用，除非用户已经指定了机器并接受了成本。

Step 0: get real numbers on the model before guessing hardware

步骤0：在猜测硬件之前，先获取模型的真实参数

Don't trust generic "VRAM requirements" blog posts for a model name — the GPU-rental SEO space is full of content-farm sites (things like "willitrunai.com", "canitrun.dev", "orcarouter.ai") that auto-generate plausible-looking spec pages for any model name, including ones that barely exist yet. They will confidently disagree with each other.

Get ground truth from the HuggingFace API instead:

bash

undefined

不要轻信网上关于某款模型的通用「VRAM需求」博客文章——GPU租赁领域的SEO内容农场网站比比皆是（比如"willitrunai.com"、"canitrun.dev"、"orcarouter.ai"这类），它们会为任何模型名自动生成看似合理的规格页面，甚至包括那些几乎还不存在的模型。这些网站的说法还会互相矛盾。

请改用HuggingFace API获取真实数据：

bash

undefined

Real param count / architecture (don't trust the repo name's number):

curl -s "https://huggingface.co/<org>/<model>/raw/main/config.json"

Real total weight size, which is what actually has to fit in VRAM:

curl -s "https://huggingface.co/<org>/<model>/raw/main/model.safetensors.index.json"
| python3 -c "import json,sys; print(json.load(sys.stdin)['metadata']['total_size'] / 1e9, 'GB')"


bf16 weight size in GB ≈ params(B) × 2. Budget the GPU so weights leave at least
~20-30GB of headroom for KV cache + activations — more if the model is a dense
transformer (full KV cache scales with context), less if it's a hybrid/linear-attention
architecture (Mamba/GDN layers keep fixed-size state, so long context is much cheaper
than a same-size dense model).

If the model is a known problem architecture (very new hybrid attention, a fresh
release), search for `"<architecture class name>" vllm github issues` — e.g.
`Qwen3_5ForConditionalGeneration vllm issues` — before committing to a GPU choice,
since brand-new architectures sometimes have open correctness bugs in vLLM
(garbled output, tensor-format mismatches). Flag this risk to the user rather than
silently hoping it works.

curl -s "https://huggingface.co/<org>/<model>/raw/main/model.safetensors.index.json"
| python3 -c "import json,sys; print(json.load(sys.stdin)['metadata']['total_size'] / 1e9, 'GB')"


bf16权重大小（GB）≈ 参数数量（B）× 2。规划GPU显存时，权重加载后需要至少预留20-30GB的空间给KV cache和激活值——如果是密集Transformer模型需要预留更多（完整KV cache大小随上下文长度增长），如果是混合/线性注意力架构则可以少留一些（Mamba/GDN层维持固定大小的状态，因此长上下文的显存开销远小于同参数量的密集模型）。

如果该模型属于已知有问题的架构（非常新的混合注意力、刚发布的模型），在确定GPU选型之前，先搜索`"<架构类名>" vllm github issues`——比如`Qwen3_5ForConditionalGeneration vllm issues`——因为全新架构在vLLM中有时存在未修复的正确性bug（输出乱码、张量格式不匹配）。需要向用户告知这个风险，而不是默默祈祷它能正常运行。

Step 1: pick and check GPU availability/price (don't guess the type string)

步骤1：选择并检查GPU可用性/价格（不要猜测类型字符串）

RunPod's REST API 404s on

/v1/gputypes

and

/v1/datacenters

— those aren't real endpoints despite being an obvious guess. Use the GraphQL API instead:

bash

curl -s -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"query":"query { gpuTypes(input: {id: \"NVIDIA H100 80GB HBM3\"}) { id lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice stockStatus } } }"}'

Valid

gpuTypeIds

include

"NVIDIA H100 80GB HBM3"

(SXM),

"NVIDIA H100 PCIe"

"NVIDIA H100 NVL"

"NVIDIA A100 80GB PCIe"

"NVIDIA A100-SXM4-80GB"

— get the full list with

query { gpuTypes { id displayName memoryInGb } }

if unsure. Pass a short list of acceptable types to pod creation (see below) so RunPod falls back automatically if your first choice is out of stock, rather than failing the whole provision.

Where the key lives: check

$RUNPOD_API_KEY

in the environment first; if it is not set, export it (create one in the RunPod console under Settings -> API Keys).

RunPod的REST API在访问

/v1/gputypes

和

/v1/datacenters

时会返回404——尽管这两个端点是很容易猜到的，但它们并不是真实存在的接口。请改用GraphQL API：

bash

curl -s -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"query":"query { gpuTypes(input: {id: \"NVIDIA H100 80GB HBM3\"}) { id lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice stockStatus } } }"}'

有效的

gpuTypeIds

包括

"NVIDIA H100 80GB HBM3"

（SXM）、

"NVIDIA H100 PCIe"

、

"NVIDIA H100 NVL"

、

"NVIDIA A100 80GB PCIe"

、

"NVIDIA A100-SXM4-80GB"

——如果不确定，可以用

query { gpuTypes { id displayName memoryInGb } }

获取完整列表。创建Pod时传入一个可接受的GPU类型短列表（见下文），这样如果首选型号缺货，RunPod会自动回退到其他型号，而不会导致整个部署失败。

密钥位置：首先检查环境变量中的

$RUNPOD_API_KEY

；如果未设置，需要导出该变量（可在RunPod控制台的Settings -> API Keys页面创建）。

Step 2: create the pod

步骤2：创建Pod

bash

curl -s -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-<model-slug>-test",
    "gpuTypeIds": ["NVIDIA H100 PCIe", "NVIDIA H100 80GB HBM3", "NVIDIA H100 NVL"],
    "imageName": "vllm/vllm-openai:latest",
    "containerDiskInGb": <weights_GB * 1.5, minimum 80>,
    "ports": ["8000/http"],
    "dockerStartCmd": [
      "--model", "<org>/<model>",
      "--host", "0.0.0.0", "--port", "8000",
      "--trust-remote-code",
      "--enforce-eager",
      "--gpu-memory-utilization", "0.90",
      "--max-model-len", "65536",
      "--served-model-name", "<served-name>",
      "--enable-auto-tool-choice",
      "--tool-call-parser", "hermes"
    ],
    "env": { "HF_HOME": "/root/.cache/huggingface" }
  }'

Notes on the flags:

--max-model-len
: go big, bigger than feels necessary. 65536 sounded generous but a real multi-hour coding session (git surgery, lots of tool output) reached ~127K tokens and hit the wall — and worse,
```
/compact
```
itself needs input + a real output budget for the summary, so if the conversation is already near the ceiling, compact fails with the same overflow error at the exact moment you need it most. Prefer the model's own native context length — check
```
max_position_embeddings
```
in the HF
```
config.json
```
from Step 0 — and use that if GPU memory allows (a 262144-token window on an 80GB H100 was comfortable for a ~28B model). Going from 65536 straight to the model's real max avoided a second round of the same problem.
--tool-call-parser
: pick the one matching the model family (
```
hermes
```
is a reasonable default for most fine-tunes;
```
qwen3_coder
```
+
```
--reasoning-parser qwen3
```
for Qwen3.x-coder-style models). Wrong parser choice won't error loudly — the model will just never emit tool_calls, only text. If Claude Code isn't calling tools, check the parser first.
--enforce-eager
: keep it for the first smoke test — it avoids CUDA-graph compile time and any related instability while you're just checking the model responds sanely. Drop it once correctness is confirmed and you care about speed.
Poll
```
GET /v1/pods
```
first and reuse a
```
RUNNING
```
pod with a matching
```
name
```
if one already exists, instead of creating a duplicate.

The vLLM URL is always

https://<podId>-8000.proxy.runpod.net

. Poll

/health

(expect 404 then a string of 502s while the image pulls and weights download, then 200) — budget several minutes for a 50GB+ model. Run the poll loop in the background (

run_in_background: true

) rather than blocking the conversation on it.

Once healthy, sanity-check with a raw curl before involving Claude Code at all:

bash

curl -s "$VLLM_URL/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "model": "<served-name>", "messages": [{"role":"user","content":"What is 2+2? One word."}],
  "max_tokens": 20, "temperature": 0
}'

Coherent output here proves the model+vLLM combo works before adding the Claude Code layer's own complexity on top. If this step gives garbled/repeating tokens, it's a model or vLLM version issue, not a bridging issue — don't proceed to step 3 yet.

bash

curl -s -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-<model-slug>-test",
    "gpuTypeIds": ["NVIDIA H100 PCIe", "NVIDIA H100 80GB HBM3", "NVIDIA H100 NVL"],
    "imageName": "vllm/vllm-openai:latest",
    "containerDiskInGb": <weights_GB * 1.5, minimum 80>,
    "ports": ["8000/http"],
    "dockerStartCmd": [
      "--model", "<org>/<model>",
      "--host", "0.0.0.0", "--port", "8000",
      "--trust-remote-code",
      "--enforce-eager",
      "--gpu-memory-utilization", "0.90",
      "--max-model-len", "65536",
      "--served-model-name", "<served-name>",
      "--enable-auto-tool-choice",
      "--tool-call-parser", "hermes"
    ],
    "env": { "HF_HOME": "/root/.cache/huggingface" }
  }'

参数说明：

--max-model-len
：设大一点，比你觉得必要的更大。 65536听起来已经很充裕，但在实际数小时的编码会话中（复杂git操作、大量工具输出）token数会达到约127K并触达上限——更糟的是，
```
/compact
```
命令本身需要输入空间和真实的输出来生成摘要，因此如果会话已经接近上限，compact会在你最需要它的时候因为同样的溢出错误而失败。优先使用模型自身的原生上下文长度——查看步骤0中HF的
```
config.json
```
里的
```
max_position_embeddings
```
——如果显存允许就用这个值（在80GB H100上，28B左右的模型用262144 token的窗口是完全没问题的）。直接从65536跳到模型的真实最大长度，可以避免再次遇到同样的问题。
--tool-call-parser
：选择与模型系列匹配的解析器（对于大多数微调模型，
```
hermes
```
是合理的默认值；对于Qwen3.x-coder风格的模型，使用
```
qwen3_coder
```
+
```
--reasoning-parser qwen3
```
）。解析器选错不会有明显的报错——模型只会输出文本，永远不会生成tool_calls。如果Claude Code没有调用工具，首先检查解析器配置。
--enforce-eager
：首次冒烟测试时保留这个参数——在你只是验证模型能否正常响应时，它可以避免CUDA图编译时间和相关的不稳定性。确认模型正确性后，如果关注速度，可以去掉这个参数。
先轮询
```
GET /v1/pods
```
，如果已经有同名的
```
RUNNING
```
状态Pod，直接复用，不要创建重复实例。

vLLM的URL始终是

https://<podId>-8000.proxy.runpod.net

。轮询

/health

接口（镜像拉取和权重下载期间会先返回404，然后是一系列502，最后返回200）——50GB以上的模型预计需要几分钟时间。将轮询循环放在后台运行（

run_in_background: true

），不要阻塞会话。

健康检查通过后，先通过原生curl做一次基础可用性校验，再接入Claude Code：

bash

curl -s "$VLLM_URL/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "model": "<served-name>", "messages": [{"role":"user","content":"What is 2+2? One word."}],
  "max_tokens": 20, "temperature": 0
}'

如果输出逻辑通顺，就证明模型+vLLM的组合是正常的，之后再加入Claude Code层的复杂度。如果这一步出现乱码/重复token，那是模型或vLLM版本的问题，不是桥接的问题——暂时不要进入步骤3。

Step 3: bridge Claude Code to it

步骤3：将Claude Code桥接到vLLM服务

Copy

scripts/anthropic-openai-proxy.js

next to where you're working and run it:

bash

VLLM_URL="https://<podId>-8000.proxy.runpod.net" PORT=<free-port> SERVED_MODEL="<served-name>" \
  node anthropic-openai-proxy.js

Pick a genuinely free local port first —

ss -ltn | grep :<port>

— this box runs many Claude sessions in parallel and a common port like 8787 is often already taken by someone else's process. Don't just assume your port is free.

Then run an isolated Claude Code invocation against it — isolated meaning it must not fall back to the user's real, billed Anthropic account if the override doesn't take:

bash

env -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
  ANTHROPIC_BASE_URL="http://localhost:<port>" \
  ANTHROPIC_API_KEY="sk-local-dummy-key" \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=<max-model-len used above> \
  claude --bare -p "<test task>" \
  --dangerously-skip-permissions \
  --model <served-name> \
  --output-format text

--bare
is what makes this safe: it forces auth to be strictly
```
ANTHROPIC_API_KEY
```
/
```
apiKeyHelper
```
, never OAuth or keychain. Without it, a misconfigured override can silently fall through to the user's real logged-in account and bill it —
```
--bare
```
closes that off structurally rather than relying on the env override being perfect.
CLAUDE_CODE_MAX_CONTEXT_TOKENS
: without this, an unrecognized
```
--model
```
name makes Claude Code assume a 200k window for its own budgeting, which causes it to request far more
```
max_tokens
```
than the vLLM server's
```
--max-model-len
```
allows. Set it to the same number you passed vLLM.
CLAUDE_CODE_MAX_OUTPUT_TOKENS
: a separate, easy-to-miss ceiling. When
```
/compact
```
has to summarize a very large conversation, Claude Code computes how big an output budget it needs and can hit an internal hard limit ("Claude's response exceeded the 128000 output token maximum"), refusing client-side before ever contacting the backend — no proxy log entry, nothing to debug remotely. Set this to something reasonable (e.g.
```
8192
```
) alongside
```
CLAUDE_CODE_MAX_CONTEXT_TOKENS
```
.
Watch the proxy's own stdout log while this runs — it prints incoming message/tool counts per turn, which is the fastest way to see whether requests are actually reaching the remote model versus dying in the bridge.
Give it 2-3+ minutes for a real coding task; a
```
--dangerously-skip-permissions -p
```
run that hasn't returned in 180s is normal, not stuck — let it background rather than assuming failure.

将

scripts/anthropic-openai-proxy.js

复制到你的工作目录下并运行：

bash

VLLM_URL="https://<podId>-8000.proxy.runpod.net" PORT=<free-port> SERVED_MODEL="<served-name>" \
  node anthropic-openai-proxy.js

首先选择一个真正空闲的本地端口——用

ss -ltn | grep :<端口号>

检查——这台机器会并行运行很多Claude会话，像8787这种常用端口通常已经被其他进程占用了。不要想当然地认为端口是空闲的。

然后针对它运行一个隔离的Claude Code调用——隔离的意思是，如果配置的覆盖不生效，也不能回退到用户真实的、会产生费用的Anthropic账户：

bash

env -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
  ANTHROPIC_BASE_URL="http://localhost:<port>" \
  ANTHROPIC_API_KEY="sk-local-dummy-key" \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=<max-model-len used above> \
  claude --bare -p "<test task>" \
  --dangerously-skip-permissions \
  --model <served-name> \
  --output-format text

参数说明：

--bare
是安全性的保障：它强制认证严格使用
```
ANTHROPIC_API_KEY
```
/
```
apiKeyHelper
```
，绝不会使用OAuth或钥匙串。如果没有这个参数，配置错误的覆盖可能会静默回退到用户真实的登录账户并产生费用——
```
--bare
```
从结构上杜绝了这种情况，而不是依赖环境变量覆盖的完美性。
CLAUDE_CODE_MAX_CONTEXT_TOKENS
：如果不设置这个参数，无法识别的
```
--model
```
名称会让Claude Code默认按200k的上下文窗口来规划预算，导致它请求的
```
max_tokens
```
远超过vLLM服务器
```
--max-model-len
```
允许的上限。将其设置为你传给vLLM的相同数值。
CLAUDE_CODE_MAX_OUTPUT_TOKENS
：这是另一个容易被忽略的上限。当
```
/compact
```
需要总结一个非常大的会话时，Claude Code会计算所需的输出预算，可能会触达内部硬限制（“Claude的响应超过了128000输出token上限”），在联系后端之前就在客户端拒绝请求——代理日志里没有任何记录，远程也无法调试。在设置
```
CLAUDE_CODE_MAX_CONTEXT_TOKENS
```
的同时，把这个参数也设为一个合理的值（比如
```
8192
```
）。
运行期间观察代理的标准输出日志——它会打印每轮收到的消息/工具调用数量，这是判断请求是否真正到达远程模型、还是死在桥接层的最快方法。
真实的编码任务需要2-3分钟甚至更久；
```
--dangerously-skip-permissions -p
```
的运行在180秒内没返回是正常的，不是卡住了——让它在后台运行，不要直接认为失败了。

Mid-session: swapping the pod under a live conversation

会话中途：在不中断对话的情况下更换Pod

If a real session outgrows the context window it was started with, you don't need to lose the conversation to fix it — the chat's history lives locally in Claude Code's own transcript, not on the pod. The pod is disposable and swappable:

Terminate the old pod, create a new one with a bigger
```
--max-model-len
```
(same image, same served-model-name).
Poll the new pod's
```
/health
```
same as Step 2.
Kill the running proxy process and restart it with the same PORT, pointing
```
VLLM_URL
```
at the new pod. Nothing on the Claude Code side needs to change — same
```
ANTHROPIC_BASE_URL
```
, same port.

Have the user resume:

claude --bare --resume <session-id> ...

with

CLAUDE_CODE_MAX_CONTEXT_TOKENS

bumped to match the new window.

Restarting the proxy while a request is in flight will error that one request out (Claude Code shows an API error and usually auto-retries) — tell the user before doing this if they're actively mid-turn, don't just do it silently.

如果实际会话的上下文超出了启动时的窗口大小，你不需要丢失对话来修复这个问题——聊天历史保存在Claude Code本地的会话记录中，而不是Pod上。Pod是一次性的、可替换的：

终止旧Pod，创建一个
```
--max-model-len
```
更大的新Pod（使用相同的镜像、相同的served-model-name）。
按照步骤2的方法轮询新Pod的
```
/health
```
接口。
杀掉正在运行的代理进程，用相同的端口重启它，将
```
VLLM_URL
```
指向新Pod。Claude Code端不需要做任何改动——保持相同的
```
ANTHROPIC_BASE_URL
```
和端口即可。
让用户恢复会话：运行
```
claude --bare --resume <session-id> ...
```
，同时将
```
CLAUDE_CODE_MAX_CONTEXT_TOKENS
```
调高到与新窗口匹配。

如果在请求处理过程中重启代理，该请求会报错（Claude Code会显示API错误，通常会自动重试）——如果用户正在对话中途，要先告知用户再操作，不要偷偷重启。

Known gotchas (already fixed in the bundled proxy — know them if you have to debug further)

已知坑点（已在附带的代理中修复——如果需要进一步调试可以了解）

max_tokens
overflow. Claude Code's default requested
```
max_tokens
```
can be ~32000 regardless of the actual
```
--max-model-len
```
. If prompt + max_tokens exceeds the server's context window, vLLM 400s on every single turn and Claude Code just retries forever, burning GPU time for nothing. Fix: clamp the forwarded
```
max_tokens
```
to something sane (the bundled proxy defaults to 4096, override with
```
MAX_TOKENS_CAP
```
).
Interleaved
role:"system"
messages. Claude Code sends
```
<system-reminder>
```
content as separate
```
role:"system"
```
turns mid-conversation (not just the one top-level
```
system
```
field). Anthropic's own API is fine with this, but vLLM's chat templates enforce "system must be the first message" and 400 on anything else. Fix: fold any non-leading system-role message into
```
user
```
before forwarding.
A
pkill -f <pattern>
that matches your own shell kills your current command (exit 144). When killing a stuck nested
```
claude
```
process, find its exact PID with
```
ps aux | grep
```
and
```
kill <pid>
```
— don't
```
pkill -f "claude"
```
from inside a shell whose own invocation string also contains "claude". This isn't just a
```
claude
```
-specific gotcha:
```
pkill -f "node proxy.js"
```
self-matched too, because the shell wrapper's own command line contained that same string. Always kill by exact PID, never by a pattern that could describe the command you're running it from.
A single long blocking backend response can exceed RunPod's edge-proxy timeout. Regular RunPod Pods have no configurable timeout setting exposed via the API (
```
executionTimeoutMs
```
/
```
idleTimeout
```
exist in RunPod's API but are Serverless-only, not Pods) — so a
```
/compact
```
that has to reason over 100K+ tokens before answering can silently exceed it, and the proxy gets back an HTML gateway error page instead of JSON (
```
Unexpected token '<', "<!DOCTYPE"
```
). Fix: always request
```
stream: true
```
(+
```
stream_options: {include_usage: true}
```
) from vLLM and relay chunks as they arrive — continuous bytes flowing means the idle timeout never triggers, no matter how long generation takes. The bundled proxy already does this.
vLLM can report
prompt_tokens: 0
in the final streaming usage chunk even when
```
completion_tokens
```
comes back correct and real generation happened (seen after heavy tool-call turns). Since Claude Code uses the reported input token count for its own auto-compact timing, a false 0 can make it think there's far more headroom than there really is — silently setting up the exact same overflow this whole skill exists to avoid. The bundled proxy falls back to a rough character-count estimate (
```
fallbackInputTokens
```
) whenever the backend's number is falsy; don't strip that out.

max_tokens
溢出。不管实际的
```
--max-model-len
```
是多少，Claude Code默认请求的
```
max_tokens
```
可能达到约32000。如果提示词+max_tokens超过了服务器的上下文窗口，vLLM每一轮都会返回400，而Claude Code会一直重试，白白浪费GPU时间。修复方法：将转发的
```
max_tokens
```
限制在合理范围内（附带的代理默认值为4096，可通过
```
MAX_TOKENS_CAP
```
覆盖）。
交错的
role:"system"
消息。 Claude Code会在对话中途将
```
<system-reminder>
```
内容作为单独的
```
role:"system"
```
轮次发送（而不只是顶层的一个
```
system
```
字段）。Anthropic自己的API支持这种做法，但vLLM的聊天模板强制要求“system消息必须是第一条”，其他情况都会返回400。修复方法：在转发前，将所有非开头的system角色消息合并到
```
user
```
角色中。
pkill -f <模式>
匹配到当前shell会杀掉你正在运行的命令（退出码144）。要杀掉卡住的嵌套
```
claude
```
进程时，先用
```
ps aux | grep
```
找到它的精确PID，然后用
```
kill <pid>
```
杀掉——不要在自身调用字符串包含“claude”的shell中运行
```
pkill -f "claude"
```
。这不只是
```
claude
```
特有的坑：
```
pkill -f "node proxy.js"
```
也会匹配到自身，因为shell包装器自己的命令行包含相同的字符串。始终用精确的PID杀进程，不要用可能匹配到当前命令的模式。
单个长时间阻塞的后端响应可能超过RunPod的边缘代理超时时间。 普通的RunPod Pod没有通过API暴露可配置的超时设置（
```
executionTimeoutMs
```
/
```
idleTimeout
```
在RunPod API中存在，但仅适用于Serverless服务，不适用于Pod）——因此，需要先推理100K+ token再回答的
```
/compact
```
请求可能会静默超时，代理收到的是HTML网关错误页面而不是JSON（报错
```
Unexpected token '<', "<!DOCTYPE"
```
）。修复方法：始终向vLLM请求
```
stream: true
```
（加上
```
stream_options: {include_usage: true}
```
），并在收到数据块时立即转发——持续的字节流意味着空闲超时永远不会触发，不管生成需要多长时间。附带的代理已经实现了这一点。
vLLM可能在最终的流式使用量块中返回
prompt_tokens: 0
——即使
```
completion_tokens
```
返回正确且确实发生了生成（在大量工具调用轮次后会出现这种情况）。由于Claude Code使用报告的输入token数来决定自动压缩的时机，错误的0值会让它认为剩余空间比实际大得多——悄悄埋下了本方案要避免的同样的溢出问题。附带的代理在后端返回的数值为假值时，会回退到粗略的字符数估算（
```
fallbackInputTokens
```
）；不要去掉这个逻辑。

Watching a live session's context usage from outside

从外部监控运行中会话的上下文使用情况

You can monitor how full a running

--resume

session's context is without touching it, by reading its own transcript for the latest assistant turn's usage — useful when the user wants periodic "what % full am I" updates while they work.

bash

undefined

你可以在不干扰会话的情况下，通过读取会话记录中最新助手轮次的使用量，监控正在运行的

--resume

会话的上下文占满程度——当用户在工作中想要定期了解“我现在用了百分之多少的上下文”时，这个方法很有用。

bash

undefined

--bare sessions land under ~/.claude/projects/<cwd-slug>/<session-id>.jsonl —

NOT the account-specific ~/.claude-<acct>/ dirs. find it with:

find ~/.claude -iname "<session-id>"

python3 -c " import json with open('<transcript.jsonl>') as f: lines = f.readlines() for line in reversed(lines): line = line.strip() if not line: continue try: o = json.loads(line) except: continue if o.get('type') == 'assistant': usage = o.get('message', {}).get('usage') if usage and usage.get('input_tokens'): print(o['timestamp'], usage['input_tokens'], usage['input_tokens']/<max_model_len>*100) break "


Scan the file in *reverse from the end*, not a fixed `tail -N` window — a single turn
with many tool round-trips can push the latest usage-bearing line much further back
than a small tail captures, making it look like nothing happened when it did.

find ~/.claude -iname "<session-id>"


要*从末尾倒序*扫描文件，而不是用固定的`tail -N`窗口——单轮包含多次工具往返的会话，可能会把最新的带使用量的行推到比小tail窗口更靠后的位置，导致看起来好像没有更新，但实际上已经有了。

Recovering from a bad

/compact

从糟糕的

/compact

结果中恢复

/compact

produces a summary so lossy the model loses track of the task (or worse, takes destructive action out of confusion), the raw pre-compact messages are still sitting in the transcript file — compact just tells future turns to use the summary instead of replaying everything before it. That means you can surgically remove the compact and resume from before it happened. This is genuinely easy to get wrong twice in a row (it was, here) — read

references/transcript-surgery.md

before attempting it, don't wing it from first principles.

如果

/compact

生成的摘要丢失了太多信息，导致模型忘记了任务（更糟的是，因为困惑而执行了破坏性操作），压缩前的原始消息仍然保存在会话记录文件中——压缩只是告诉未来的轮次使用摘要，而不是重放之前的所有内容。这意味着你可以精准地移除压缩操作，从压缩发生前的状态恢复会话。这真的很容易连续搞错两次（亲身体验）——在尝试之前，请先阅读

references/transcript-surgery.md

，不要凭第一感觉瞎试。

Step 4: teardown — "sin dejar rastro" means verify, not assume

步骤4：清理销毁——“sin dejar rastro”的意思是要验证，而非假设

When the user says kill/delete everything, that means confirm the negative, not just issue the delete calls:

bash

undefined

当用户要求终止/删除所有资源时，需要确认确实没有任何残留，而不只是发出删除命令：

bash

undefined

kill exact PIDs (not pkill -f, see gotcha 3)

ps aux | grep -E "anthropic-openai-proxy|claude --bare" | grep -v grep kill <pid> <pid>

terminate the pod

curl -s -X DELETE "https://rest.runpod.io/v1/pods/<podId>" -H "Authorization: Bearer $RUNPOD_API_KEY" -w "\n%{http_code}\n"

confirm nothing is left running or costing money

curl -s https://rest.runpod.io/v1/pods -H "Authorization: Bearer $RUNPOD_API_KEY" # expect [] curl -s https://rest.runpod.io/v1/networkvolumes -H "Authorization: Bearer $RUNPOD_API_KEY" # expect [] unless one was deliberately kept curl -s -X POST https://api.runpod.io/graphql -H "Authorization: Bearer $RUNPOD_API_KEY"
-H "Content-Type: application/json" -d '{"query":"query { myself { clientBalance currentSpendPerHr } }"}'

currentSpendPerHr should read 0

delete scratch files (proxy copy, test project, logs) from wherever they were put —

the session scratchpad directory, never the repo

rm -rf <scratch-dir>


A network volume (persistent disk) is a *separate* billable resource from a pod —
terminating the pod does not delete an attached network volume. If one was created for
this session (rather than pre-existing), delete it explicitly:
`DELETE /v1/networkvolumes/<id>`.

Report the final state back with the actual numbers (balance, `currentSpendPerHr`,
empty pod/volume lists) rather than just saying "done" — that's what "sin dejar
rastro" is actually asking to see proof of.

rm -rf <scratch-dir>


网络卷（持久化磁盘）是与Pod*独立的*计费资源——终止Pod不会删除挂载的网络卷。如果是为本会话创建的网络卷（而非预先存在的），需要显式删除：`DELETE /v1/networkvolumes/<id>`。

要向用户反馈最终状态和实际数据（余额、`currentSpendPerHr`、空的Pod/卷列表），而不是只说“完成了”——这才是“sin dejar rastro”真正要求的：提供不留痕迹的证据。

runpod-vllm-claude-code

Original

Translation

RunPod vLLM + Claude Code bridge

RunPod vLLM + Claude Code 桥接方案

Step 0: get real numbers on the model before guessing hardware

步骤0：在猜测硬件之前，先获取模型的真实参数

Real param count / architecture (don't trust the repo name's number):

Real param count / architecture (don't trust the repo name's number):

Real total weight size, which is what actually has to fit in VRAM:

Real total weight size, which is what actually has to fit in VRAM:

Step 1: pick and check GPU availability/price (don't guess the type string)

步骤1：选择并检查GPU可用性/价格（不要猜测类型字符串）

Step 2: create the pod

步骤2：创建Pod

Step 3: bridge Claude Code to it

步骤3：将Claude Code桥接到vLLM服务

Mid-session: swapping the pod under a live conversation

会话中途：在不中断对话的情况下更换Pod

Known gotchas (already fixed in the bundled proxy — know them if you have to debug further)

已知坑点（已在附带的代理中修复——如果需要进一步调试可以了解）

Watching a live session's context usage from outside

从外部监控运行中会话的上下文使用情况

--bare sessions land under ~/.claude/projects/<cwd-slug>/<session-id>.jsonl —

--bare sessions land under ~/.claude/projects/<cwd-slug>/<session-id>.jsonl —

NOT the account-specific ~/.claude-<acct>/ dirs. find it with:

NOT the account-specific ~/.claude-<acct>/ dirs. find it with:

Recovering from a bad /compact

从糟糕的/compact结果中恢复

Step 4: teardown — "sin dejar rastro" means verify, not assume

步骤4：清理销毁——“sin dejar rastro”的意思是要验证，而非假设

kill exact PIDs (not pkill -f, see gotcha 3)

kill exact PIDs (not pkill -f, see gotcha 3)

terminate the pod

terminate the pod

confirm nothing is left running or costing money

confirm nothing is left running or costing money

currentSpendPerHr should read 0

currentSpendPerHr should read 0

delete scratch files (proxy copy, test project, logs) from wherever they were put —

delete scratch files (proxy copy, test project, logs) from wherever they were put —

the session scratchpad directory, never the repo

the session scratchpad directory, never the repo

Recovering from a bad
`/compact`

从糟糕的
`/compact`
结果中恢复