RunPod vLLM + Claude Code bridge

Two things are being solved together here, and it's worth being clear about which is which:

Getting a model running on RunPod. Straightforward — pick a GPU, start a vLLM container, wait for health.
Getting Claude Code to actually talk to it. Not straightforward — Claude Code speaks Anthropic's Messages API (
```
/v1/messages
```
, content blocks, SSE events). vLLM speaks OpenAI's
```
/v1/chat/completions
```
. Nothing bridges these two by default.
```
ANTHROPIC_BASE_URL
```
lets Claude Code point at any server that implements the Messages API shape — so a small translation proxy in the middle is what makes "point Claude Code at my RunPod box" actually work.
```
scripts/anthropic-openai-proxy.js
```
is that proxy, already fixed for the two protocol mismatches that will otherwise burn an hour of retries (see Gotchas below). Use it as-is; don't re-derive it from scratch.

This is exploratory/disposable infra by nature — a GPU pod costs real money per hour the moment it exists. Before creating one, tell the user which GPU and roughly $/hr, unless they already named the machine and accepted the cost.

Step 0: get real numbers on the model before guessing hardware

Don't trust generic "VRAM requirements" blog posts for a model name — the GPU-rental SEO space is full of content-farm sites (things like "willitrunai.com", "canitrun.dev", "orcarouter.ai") that auto-generate plausible-looking spec pages for any model name, including ones that barely exist yet. They will confidently disagree with each other.

Get ground truth from the HuggingFace API instead:

bash

# Real param count / architecture (don't trust the repo name's number):
curl -s "https://huggingface.co/<org>/<model>/raw/main/config.json"

# Real total weight size, which is what actually has to fit in VRAM:
curl -s "https://huggingface.co/<org>/<model>/raw/main/model.safetensors.index.json" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['metadata']['total_size'] / 1e9, 'GB')"

bf16 weight size in GB ≈ params(B) × 2. Budget the GPU so weights leave at least ~20-30GB of headroom for KV cache + activations — more if the model is a dense transformer (full KV cache scales with context), less if it's a hybrid/linear-attention architecture (Mamba/GDN layers keep fixed-size state, so long context is much cheaper than a same-size dense model).

If the model is a known problem architecture (very new hybrid attention, a fresh release), search for

"<architecture class name>" vllm github issues

— e.g.

Qwen3_5ForConditionalGeneration vllm issues

— before committing to a GPU choice, since brand-new architectures sometimes have open correctness bugs in vLLM (garbled output, tensor-format mismatches). Flag this risk to the user rather than silently hoping it works.

Step 1: pick and check GPU availability/price (don't guess the type string)

RunPod's REST API 404s on

/v1/gputypes

and

/v1/datacenters

— those aren't real endpoints despite being an obvious guess. Use the GraphQL API instead:

bash

curl -s -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"query":"query { gpuTypes(input: {id: \"NVIDIA H100 80GB HBM3\"}) { id lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice stockStatus } } }"}'

Valid

gpuTypeIds

include

"NVIDIA H100 80GB HBM3"

(SXM),

"NVIDIA H100 PCIe"

"NVIDIA H100 NVL"

"NVIDIA A100 80GB PCIe"

"NVIDIA A100-SXM4-80GB"

— get the full list with

query { gpuTypes { id displayName memoryInGb } }

if unsure. Pass a short list of acceptable types to pod creation (see below) so RunPod falls back automatically if your first choice is out of stock, rather than failing the whole provision.

Where the key lives: check

$RUNPOD_API_KEY

in the environment first; if it is not set, export it (create one in the RunPod console under Settings -> API Keys).

Step 2: create the pod

bash

curl -s -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-<model-slug>-test",
    "gpuTypeIds": ["NVIDIA H100 PCIe", "NVIDIA H100 80GB HBM3", "NVIDIA H100 NVL"],
    "imageName": "vllm/vllm-openai:latest",
    "containerDiskInGb": <weights_GB * 1.5, minimum 80>,
    "ports": ["8000/http"],
    "dockerStartCmd": [
      "--model", "<org>/<model>",
      "--host", "0.0.0.0", "--port", "8000",
      "--trust-remote-code",
      "--enforce-eager",
      "--gpu-memory-utilization", "0.90",
      "--max-model-len", "65536",
      "--served-model-name", "<served-name>",
      "--enable-auto-tool-choice",
      "--tool-call-parser", "hermes"
    ],
    "env": { "HF_HOME": "/root/.cache/huggingface" }
  }'

Notes on the flags:

--max-model-len
: go big, bigger than feels necessary. 65536 sounded generous but a real multi-hour coding session (git surgery, lots of tool output) reached ~127K tokens and hit the wall — and worse,
```
/compact
```
itself needs input + a real output budget for the summary, so if the conversation is already near the ceiling, compact fails with the same overflow error at the exact moment you need it most. Prefer the model's own native context length — check
```
max_position_embeddings
```
in the HF
```
config.json
```
from Step 0 — and use that if GPU memory allows (a 262144-token window on an 80GB H100 was comfortable for a ~28B model). Going from 65536 straight to the model's real max avoided a second round of the same problem.
--tool-call-parser
: pick the one matching the model family (
```
hermes
```
is a reasonable default for most fine-tunes;
```
qwen3_coder
```
+
```
--reasoning-parser qwen3
```
for Qwen3.x-coder-style models). Wrong parser choice won't error loudly — the model will just never emit tool_calls, only text. If Claude Code isn't calling tools, check the parser first.
--enforce-eager
: keep it for the first smoke test — it avoids CUDA-graph compile time and any related instability while you're just checking the model responds sanely. Drop it once correctness is confirmed and you care about speed.
Poll
```
GET /v1/pods
```
first and reuse a
```
RUNNING
```
pod with a matching
```
name
```
if one already exists, instead of creating a duplicate.

The vLLM URL is always

https://<podId>-8000.proxy.runpod.net

. Poll

/health

(expect 404 then a string of 502s while the image pulls and weights download, then 200) — budget several minutes for a 50GB+ model. Run the poll loop in the background (

run_in_background: true

) rather than blocking the conversation on it.

Once healthy, sanity-check with a raw curl before involving Claude Code at all:

bash

curl -s "$VLLM_URL/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "model": "<served-name>", "messages": [{"role":"user","content":"What is 2+2? One word."}],
  "max_tokens": 20, "temperature": 0
}'

Coherent output here proves the model+vLLM combo works before adding the Claude Code layer's own complexity on top. If this step gives garbled/repeating tokens, it's a model or vLLM version issue, not a bridging issue — don't proceed to step 3 yet.

Step 3: bridge Claude Code to it

Copy

scripts/anthropic-openai-proxy.js

next to where you're working and run it:

bash

VLLM_URL="https://<podId>-8000.proxy.runpod.net" PORT=<free-port> SERVED_MODEL="<served-name>" \
  node anthropic-openai-proxy.js

Pick a genuinely free local port first —

ss -ltn | grep :<port>

— this box runs many Claude sessions in parallel and a common port like 8787 is often already taken by someone else's process. Don't just assume your port is free.

Then run an isolated Claude Code invocation against it — isolated meaning it must not fall back to the user's real, billed Anthropic account if the override doesn't take:

bash

env -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
  ANTHROPIC_BASE_URL="http://localhost:<port>" \
  ANTHROPIC_API_KEY="sk-local-dummy-key" \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=<max-model-len used above> \
  claude --bare -p "<test task>" \
  --dangerously-skip-permissions \
  --model <served-name> \
  --output-format text

--bare
is what makes this safe: it forces auth to be strictly
```
ANTHROPIC_API_KEY
```
/
```
apiKeyHelper
```
, never OAuth or keychain. Without it, a misconfigured override can silently fall through to the user's real logged-in account and bill it —
```
--bare
```
closes that off structurally rather than relying on the env override being perfect.
CLAUDE_CODE_MAX_CONTEXT_TOKENS
: without this, an unrecognized
```
--model
```
name makes Claude Code assume a 200k window for its own budgeting, which causes it to request far more
```
max_tokens
```
than the vLLM server's
```
--max-model-len
```
allows. Set it to the same number you passed vLLM.
CLAUDE_CODE_MAX_OUTPUT_TOKENS
: a separate, easy-to-miss ceiling. When
```
/compact
```
has to summarize a very large conversation, Claude Code computes how big an output budget it needs and can hit an internal hard limit ("Claude's response exceeded the 128000 output token maximum"), refusing client-side before ever contacting the backend — no proxy log entry, nothing to debug remotely. Set this to something reasonable (e.g.
```
8192
```
) alongside
```
CLAUDE_CODE_MAX_CONTEXT_TOKENS
```
.
Watch the proxy's own stdout log while this runs — it prints incoming message/tool counts per turn, which is the fastest way to see whether requests are actually reaching the remote model versus dying in the bridge.
Give it 2-3+ minutes for a real coding task; a
```
--dangerously-skip-permissions -p
```
run that hasn't returned in 180s is normal, not stuck — let it background rather than assuming failure.

Mid-session: swapping the pod under a live conversation

If a real session outgrows the context window it was started with, you don't need to lose the conversation to fix it — the chat's history lives locally in Claude Code's own transcript, not on the pod. The pod is disposable and swappable:

Terminate the old pod, create a new one with a bigger
```
--max-model-len
```
(same image, same served-model-name).
Poll the new pod's
```
/health
```
same as Step 2.
Kill the running proxy process and restart it with the same PORT, pointing
```
VLLM_URL
```
at the new pod. Nothing on the Claude Code side needs to change — same
```
ANTHROPIC_BASE_URL
```
, same port.

Have the user resume:

claude --bare --resume <session-id> ...

with

CLAUDE_CODE_MAX_CONTEXT_TOKENS

bumped to match the new window.

Restarting the proxy while a request is in flight will error that one request out (Claude Code shows an API error and usually auto-retries) — tell the user before doing this if they're actively mid-turn, don't just do it silently.

Known gotchas (already fixed in the bundled proxy — know them if you have to debug further)

max_tokens
overflow. Claude Code's default requested
```
max_tokens
```
can be ~32000 regardless of the actual
```
--max-model-len
```
. If prompt + max_tokens exceeds the server's context window, vLLM 400s on every single turn and Claude Code just retries forever, burning GPU time for nothing. Fix: clamp the forwarded
```
max_tokens
```
to something sane (the bundled proxy defaults to 4096, override with
```
MAX_TOKENS_CAP
```
).
Interleaved
role:"system"
messages. Claude Code sends
```
<system-reminder>
```
content as separate
```
role:"system"
```
turns mid-conversation (not just the one top-level
```
system
```
field). Anthropic's own API is fine with this, but vLLM's chat templates enforce "system must be the first message" and 400 on anything else. Fix: fold any non-leading system-role message into
```
user
```
before forwarding.
A
pkill -f <pattern>
that matches your own shell kills your current command (exit 144). When killing a stuck nested
```
claude
```
process, find its exact PID with
```
ps aux | grep
```
and
```
kill <pid>
```
— don't
```
pkill -f "claude"
```
from inside a shell whose own invocation string also contains "claude". This isn't just a
```
claude
```
-specific gotcha:
```
pkill -f "node proxy.js"
```
self-matched too, because the shell wrapper's own command line contained that same string. Always kill by exact PID, never by a pattern that could describe the command you're running it from.
A single long blocking backend response can exceed RunPod's edge-proxy timeout. Regular RunPod Pods have no configurable timeout setting exposed via the API (
```
executionTimeoutMs
```
/
```
idleTimeout
```
exist in RunPod's API but are Serverless-only, not Pods) — so a
```
/compact
```
that has to reason over 100K+ tokens before answering can silently exceed it, and the proxy gets back an HTML gateway error page instead of JSON (
```
Unexpected token '<', "<!DOCTYPE"
```
). Fix: always request
```
stream: true
```
(+
```
stream_options: {include_usage: true}
```
) from vLLM and relay chunks as they arrive — continuous bytes flowing means the idle timeout never triggers, no matter how long generation takes. The bundled proxy already does this.
vLLM can report
prompt_tokens: 0
in the final streaming usage chunk even when
```
completion_tokens
```
comes back correct and real generation happened (seen after heavy tool-call turns). Since Claude Code uses the reported input token count for its own auto-compact timing, a false 0 can make it think there's far more headroom than there really is — silently setting up the exact same overflow this whole skill exists to avoid. The bundled proxy falls back to a rough character-count estimate (
```
fallbackInputTokens
```
) whenever the backend's number is falsy; don't strip that out.

Watching a live session's context usage from outside

You can monitor how full a running

--resume

session's context is without touching it, by reading its own transcript for the latest assistant turn's usage — useful when the user wants periodic "what % full am I" updates while they work.

bash

# --bare sessions land under ~/.claude/projects/<cwd-slug>/<session-id>.jsonl —
# NOT the account-specific ~/.claude-<acct>/ dirs. find it with:
find ~/.claude -iname "*<session-id>*"

python3 -c "
import json
with open('<transcript.jsonl>') as f:
    lines = f.readlines()
for line in reversed(lines):
    line = line.strip()
    if not line: continue
    try: o = json.loads(line)
    except: continue
    if o.get('type') == 'assistant':
        usage = o.get('message', {}).get('usage')
        if usage and usage.get('input_tokens'):
            print(o['timestamp'], usage['input_tokens'], usage['input_tokens']/<max_model_len>*100)
            break
"

Scan the file in reverse from the end, not a fixed

tail -N

window — a single turn with many tool round-trips can push the latest usage-bearing line much further back than a small tail captures, making it look like nothing happened when it did.

Recovering from a bad

/compact

/compact

produces a summary so lossy the model loses track of the task (or worse, takes destructive action out of confusion), the raw pre-compact messages are still sitting in the transcript file — compact just tells future turns to use the summary instead of replaying everything before it. That means you can surgically remove the compact and resume from before it happened. This is genuinely easy to get wrong twice in a row (it was, here) — read

references/transcript-surgery.md

before attempting it, don't wing it from first principles.

Step 4: teardown — "sin dejar rastro" means verify, not assume

When the user says kill/delete everything, that means confirm the negative, not just issue the delete calls:

bash

# kill exact PIDs (not pkill -f, see gotcha 3)
ps aux | grep -E "anthropic-openai-proxy|claude --bare" | grep -v grep
kill <pid> <pid>

# terminate the pod
curl -s -X DELETE "https://rest.runpod.io/v1/pods/<podId>" -H "Authorization: Bearer $RUNPOD_API_KEY" -w "\n%{http_code}\n"

# confirm nothing is left running or costing money
curl -s https://rest.runpod.io/v1/pods -H "Authorization: Bearer $RUNPOD_API_KEY"          # expect []
curl -s https://rest.runpod.io/v1/networkvolumes -H "Authorization: Bearer $RUNPOD_API_KEY" # expect [] unless one was deliberately kept
curl -s -X POST https://api.runpod.io/graphql -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" -d '{"query":"query { myself { clientBalance currentSpendPerHr } }"}'
# currentSpendPerHr should read 0

# delete scratch files (proxy copy, test project, logs) from wherever they were put —
# the session scratchpad directory, never the repo
rm -rf <scratch-dir>

A network volume (persistent disk) is a separate billable resource from a pod — terminating the pod does not delete an attached network volume. If one was created for this session (rather than pre-existing), delete it explicitly:

DELETE /v1/networkvolumes/<id>

Report the final state back with the actual numbers (balance,

currentSpendPerHr

, empty pod/volume lists) rather than just saying "done" — that's what "sin dejar rastro" is actually asking to see proof of.

runpod-vllm-claude-code

NPX Install

Tags

SKILL.md Content

RunPod vLLM + Claude Code bridge

Step 0: get real numbers on the model before guessing hardware

Step 1: pick and check GPU availability/price (don't guess the type string)

Step 2: create the pod

Step 3: bridge Claude Code to it

Mid-session: swapping the pod under a live conversation

Known gotchas (already fixed in the bundled proxy — know them if you have to debug further)

Watching a live session's context usage from outside

Recovering from a bad
`/compact`

Step 4: teardown — "sin dejar rastro" means verify, not assume