Benchmark $/token for a model on vLLM in Chalk

Produces a defensible cost-per-token number: serve the model on a GPU, drive it to saturation, confirm the GPU is actually the limiter, then divide authoritative cloud pricing by measured throughput.

The order matters. A throughput number without a saturation check is unfalsifiable, and a $/hr figure pulled from a blog post is usually wrong (see step 5).

1. Serve the model

chalk container run --dir <chalk-project> --json --no-spinners \
  -i vllm/vllm-openai:latest -n bench-server \
  --gpu nvidia-l4:1 --cpu 4 --memory 16Gi --port 8000 --lifetime 2h \
  --env "HF_TOKEN=$HF_TOKEN" --env HF_HOME=/root/.cache/huggingface \
  --await-running --await-timeout 20m \
  -e vllm -- serve <MODEL> --port 8000 \
  --dtype bfloat16 --max-model-len 16384 --gpu-memory-utilization 0.90

chalk container get --name bench-server

returns

webUrl

(publicly reachable) and

podName

. Poll

GET $webUrl/health

— it returns 503 until weights are loaded, then 200. Expect several minutes for a 7B download.

GPU selection. Only GPU pools labelled

chalk.ai/workload-type=compute

can run container/sandbox workloads. On the staging cluster that is

l4-gpus

(L4, 24GB, native bf16), capped at 2 nodes x 1 GPU. A pool that lacks the label (e.g.

t4-gpus

) will leave the pod Unschedulable and the controller gives up after 300s — shorter than a cold GPU node autoscale, so a valid request can fail purely on capacity. Free a GPU and retry.

Turing (T4) has no native bf16; use

--dtype float16

there.

2. Serving flags that change the answer

Benchmark the config you intend to run, and say which flags you set. These materially move the result:

Flag	Why it matters
`--max-num-seqs`	Caps concurrent sequences. Too high wastes scheduler time on short requests — throughput can fall as you raise it. Sweep it.
`--max-num-batched-tokens`	Prefill chunk budget. The binding constraint for prompt-heavy / short-output workloads.
`--enable-prefix-caching`	On by default in vLLM V1. Inflates apparent prompt throughput (see pitfalls).
`--limit-mm-per-prompt '{"image":1}'`	Multimodal models profile peak activation memory against worst-case images per prompt, stealing KV cache. Pin it.
`--dtype`	bf16 on Ada/Hopper; fp16 on Turing.
`--enforce-eager`	Skips CUDA graph capture: faster startup, slower decode. Fine for 1-token outputs, wrong for general serving. Don't leave it on in a benchmark you intend to generalize.
`--api-key`	The container `webUrl` is publicly resolvable. An unauthenticated LLM endpoint is exposed. Pair with `chalk container run --authenticated` .

3. Drive load

Use

scripts/bench.py

(bundled with this skill). It sweeps prompt length and concurrency, and reads throughput from vLLM's own Prometheus counters (

vllm:prompt_tokens_total

vllm:generation_tokens_total

vllm:request_success_total

) across a steady-state window.

./scripts/bench.py --base-url "$URL" --model "$MODEL" \
  --input-lens 128,512 --concurrency 8,32,64 --duration 30

Methodology rules, each of which fixes a real failure mode:

Sustained windows, not fixed request counts. Sub-10s windows are dominated by jitter. Use >=20s and a warm ramp before the measured window.
Server-side counters, not client timing. Removes client overhead and WAN latency from the measurement.
Sweep concurrency until throughput stops rising. One concurrency level tells you nothing about where saturation is.
Unique prompt prefixes. Identical prompts hit the prefix cache and inflate results (bundled script seeds each request).
Watch for client-side bottlenecks. If throughput drops at high concurrency, suspect the load generator's CPU before the server. Re-run from a bigger box, or from inside the cluster.

Where to run the driver. From a laptop is fine to start. To rule out the network path, run it as a container inside the cluster (

--cpu 6

, the vLLM image already has Python + httpx) and point it at the server's pod IP (

kubectl get pod <podName> -o jsonpath='{.status.podIP}'

), then repeat against the public

webUrl

. If the two agree, the ingress is not your bottleneck — measure this rather than assuming it.

4. Confirm the GPU is the limiter

A throughput number is only a hardware number if the hardware was saturated. Check DCGM on the node running the model:

NODE=$(kubectl -n ns-<env> get pod <podName> -o jsonpath='{.spec.nodeName}')
DCGM=$(kubectl -n chalk-telemetry get pods -o wide | grep "dcgm.*$NODE" | awk '{print $1}')
kubectl -n chalk-telemetry port-forward "$DCGM" 19400:9400 &
curl -s localhost:19400/metrics | grep -E '^DCGM_FI_DEV_(GPU_UTIL|POWER_USAGE|FB_USED)'

Sample it during the run.

GPU_UTIL

at 100 and

POWER_USAGE

at the card's TDP (72W for L4, 300W for A100 40GB) means you are GPU-bound and the number is real. Utilization well under 100% means you measured your client, the network, or a scheduler cap — not the GPU. Note the exporter scrapes on an interval, so identical consecutive samples are expected.

5. Get the real $/hr

Do not trust third-party pricing pages or a web search summary. For GPU machine families the accelerator is often a separate SKU from vCPU and RAM, and aggregator sites frequently omit it — which understates the node by 3x.

Query GCP's Cloud Billing Catalog directly (Compute Engine service

6F81-5844-456A

), summing the SKUs that make up the node:

# paginate skus?pageSize=5000&currencyCode=USD with an Authorization: Bearer token
# from `gcloud auth print-access-token`, filter serviceRegions for your region,
# then match description against your machine family and accelerator.

Worked example,

g2-standard-8

(8 vCPU, 32 GiB, 1x L4) in

us-central1

SKU	Rate	Qty	Subtotal
G2 Instance Core running in Americas	$0.024988 /hr	8	$0.199904
G2 Instance Ram running in Americas	$0.002927 /GiB-hr	32	$0.093664
Nvidia L4 GPU running in Americas	$0.560040 /hr	1	$0.560040
			$0.853608 /hr

Omitting the GPU SKU yields $0.2936/hr — a number that appears on public pricing aggregators and is wrong. Sanity-check by computing spot the same way and confirming it lands at the expected discount.

Also pull Spot and 1yr/3yr CUD rates; they are separate SKUs and are what a production deployment would actually pay.

6. Compute the cost

$/sec        = node_hourly / 3600
$/1M in-tok  = $/sec / prompt_tok_per_s * 1e6
$/1M req     = $/sec / rps * 1e6

Report $/1M input tokens and $/1M requests, plus the on-demand / spot / CUD variants.

Pick the right unit. For a classifier or guardrail that emits a single token, output tokens are identical to requests and "$/output token" is a degenerate metric — lead with $/classification. For a chat or coding model, decode dominates and $/output token is the meaningful figure; set

--output-tokens

to a realistic completion length rather than 1.

Pitfalls

Prefix caching inflates $/input-token. A shared system prompt gets cached, so
```
prompt_tokens_total
```
counts tokens that were never recomputed. The bundled script reports the hit rate. It is a fair comparison against a vendor who bills you for every input token, but it is not your real compute headroom — say which one you are quoting.
Saturation numbers are a floor, not a forecast. These are 100%-duty-cycle figures. At 20% real utilization, multiply by 5. Idle GPU time is the dominant cost in most production deployments.
Tool-calling models need the right parser and the right checkpoint.
```
--enable-auto-tool-choice --tool-call-parser hermes
```
plus an instruct model. Some Coder-tuned checkpoints emit tool calls as fenced JSON text that the parser ignores, so
```
tool_calls
```
returns null under
```
tool_choice: auto
```
while
```
tool_choice: required
```
works (guided decoding forces the shape).
Completed container pods are garbage-collected, taking their logs. End a one-shot driver script with
```
sleep
```
so you can still
```
kubectl logs
```
it.
chalk container exec
is unimplemented for k8s containers (
```
only implemented for host containers
```
). Drive work via the entrypoint, or use
```
--compute-class host
```
/ the sandbox API.
Piping a script into
bash
breaks silently if it invokes anything that reads stdin — the child eats the rest of the script. Write it to a file and add
```
exec </dev/null
```
.
Sandboxes have no network egress and no flag to grant it; even public IPs are blocked. Use
```
chalk container run --route <CIDR>
```
when the workload must reach anything.

Reporting

State the model, GPU, every non-default serving flag, the measured throughput, the DCGM saturation evidence, and the SKU breakdown behind the $/hr. A cost-per-token figure without those is not reproducible and should not be quoted.

benchmark-vllm-model

NPX Install

Tags

SKILL.md Content