Back to Details

benchmark-vllm-model

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Benchmark $/token for a model on vLLM in Chalk

在 Chalk 中对 vLLM 模型的 $/token 进行基准测试

Produces a defensible cost-per-token number: serve the model on a GPU, drive it to saturation, confirm the GPU is actually the limiter, then divide authoritative cloud pricing by measured throughput.
The order matters. A throughput number without a saturation check is unfalsifiable, and a $/hr figure pulled from a blog post is usually wrong (see step 5).
得出一项经得起质疑的每 token 成本数据：在 GPU 上服务该模型，将负载推至饱和状态，确认真正的瓶颈确实是 GPU，然后用权威云定价除以实测吞吐量。
顺序很重要。未经饱和度检查的吞吐量数字无法证伪，而从博客文章中摘取的 $/hr 数字通常不正确（见步骤 5）。

1. Serve the model

1. 服务模型

sh
chalk container run --dir <chalk-project> --json --no-spinners \
  -i vllm/vllm-openai:latest -n bench-server \
  --gpu nvidia-l4:1 --cpu 4 --memory 16Gi --port 8000 --lifetime 2h \
  --env "HF_TOKEN=$HF_TOKEN" --env HF_HOME=/root/.cache/huggingface \
  --await-running --await-timeout 20m \
  -e vllm -- serve <MODEL> --port 8000 \
  --dtype bfloat16 --max-model-len 16384 --gpu-memory-utilization 0.90
chalk container get --name bench-server
returns 
webUrl
(publicly reachable) and 
podName
. Poll 
GET $webUrl/health
— it returns 503 until weights are loaded, then 200. Expect several minutes for a 7B download.
GPU selection. Only GPU pools labelled 
chalk.ai/workload-type=compute
can run container/sandbox workloads. On the staging cluster that is 
l4-gpus
(L4, 24GB, native bf16), capped at 2 nodes x 1 GPU. A pool that lacks the label (e.g. 
t4-gpus
) will leave the pod Unschedulable and the controller gives up after 300s — shorter than a cold GPU node autoscale, so a valid request can fail purely on capacity. Free a GPU and retry.
Turing (T4) has no native bf16; use 
--dtype float16
there.
sh
chalk container run --dir <chalk-project> --json --no-spinners \
  -i vllm/vllm-openai:latest -n bench-server \
  --gpu nvidia-l4:1 --cpu 4 --memory 16Gi --port 8000 --lifetime 2h \
  --env "HF_TOKEN=$HF_TOKEN" --env HF_HOME=/root/.cache/huggingface \
  --await-running --await-timeout 20m \
  -e vllm -- serve <MODEL> --port 8000 \
  --dtype bfloat16 --max-model-len 16384 --gpu-memory-utilization 0.90
chalk container get --name bench-server
会返回 
webUrl
（公开可达）和 
podName
。轮询 
GET $webUrl/health
——权重加载完成前返回 503，之后返回 200。下载 7B 模型预计需要数分钟。
GPU 选择。 只有带有 
chalk.ai/workload-type=compute
标签的 GPU 池才能运行 container/sandbox 工作负载。在 staging 集群中，该池为 
l4-gpus
（L4、24GB、原生支持 bf16），上限为 2 个节点 x 1 个 GPU。如果池缺少该标签（例如 
t4-gpus
），pod 会保持 Unschedulable 状态，controller 会在 300s 后放弃——这比冷 GPU 节点自动扩缩容所需的时间更短，因此一个有效请求可能纯粹因为容量不足而失败。请释放 GPU 后重试。
Turing（T4）不支持原生 bf16；在该架构上应使用 
--dtype float16
。

2. Serving flags that change the answer

2. 会影响结果的服务参数

Benchmark the config you intend to run, and say which flags you set. These materially move the result:
FlagWhy it matters
--max-num-seqs
Caps concurrent sequences. Too high wastes scheduler time on short requests — throughput can fall as you raise it. Sweep it.
--max-num-batched-tokens
Prefill chunk budget. The binding constraint for prompt-heavy / short-output workloads.
--enable-prefix-caching
On by default in vLLM V1. Inflates apparent prompt throughput (see pitfalls).
--limit-mm-per-prompt '{"image":1}'
Multimodal models profile peak activation memory against worst-case images per prompt, stealing KV cache. Pin it.
--dtype
bf16 on Ada/Hopper; fp16 on Turing.
--enforce-eager
Skips CUDA graph capture: faster startup, slower decode. Fine for 1-token outputs, wrong for general serving. Don't leave it on in a benchmark you intend to generalize.
--api-key
The container 
webUrl
is publicly resolvable. An unauthenticated LLM endpoint is exposed. Pair with 
chalk container run --authenticated
.
应对你计划实际运行的配置进行基准测试，并明确说明设置了哪些参数。以下参数会显著影响结果：
参数为何重要
--max-num-seqs
限制并发序列数量。设置过高会浪费 scheduler 在短请求上的时间——随着该值增大，吞吐量反而可能下降。应对其进行参数扫描。
--max-num-batched-tokens
Prefill 分块预算。对于 prompt 较多或输出较短的工作负载，这是关键约束。
--enable-prefix-caching
在 vLLM V1 中默认开启。会抬高表面上的 prompt 吞吐量（见常见陷阱）。
--limit-mm-per-prompt '{\"image\":1}'
多模态模型会根据每个 prompt 的最坏情况图像数量评估峰值 activation memory，从而挤占 KV cache。请明确固定该值。
--dtype
Ada/Hopper 使用 bf16；Turing 使用 fp16。
--enforce-eager
跳过 CUDA graph capture：启动更快，但 decode 更慢。适用于只输出 1 个 token 的场景，不适合通用服务。在准备推广结果的基准测试中，不要一直启用它。
--api-key
container 的 
webUrl
可公开解析。未进行身份验证的 LLM endpoint 会暴露在外。请结合 
chalk container run --authenticated
使用。

3. Drive load

3. 施加负载

Use 
scripts/bench.py
(bundled with this skill). It sweeps prompt length and concurrency, and reads throughput from vLLM's own Prometheus counters (
vllm:prompt_tokens_total
, 
vllm:generation_tokens_total
, 
vllm:request_success_total
) across a steady-state window.
sh
./scripts/bench.py --base-url "$URL" --model "$MODEL" \
  --input-lens 128,512 --concurrency 8,32,64 --duration 30
Methodology rules, each of which fixes a real failure mode:
  • Sustained windows, not fixed request counts. Sub-10s windows are dominated by jitter. Use >=20s and a warm ramp before the measured window.
  • Server-side counters, not client timing. Removes client overhead and WAN latency from the measurement.
  • Sweep concurrency until throughput stops rising. One concurrency level tells you nothing about where saturation is.
  • Unique prompt prefixes. Identical prompts hit the prefix cache and inflate results (bundled script seeds each request).
  • Watch for client-side bottlenecks. If throughput drops at high concurrency, suspect the load generator's CPU before the server. Re-run from a bigger box, or from inside the cluster.
Where to run the driver. From a laptop is fine to start. To rule out the network path, run it as a container inside the cluster (
--cpu 6
, the vLLM image already has Python + httpx) and point it at the server's pod IP (
kubectl get pod <podName> -o jsonpath='{.status.podIP}'
), then repeat against the public 
webUrl
. If the two agree, the ingress is not your bottleneck — measure this rather than assuming it.
使用 
scripts/bench.py
（随本 skill 附带）。它会扫描 prompt 长度和 concurrency，并从 vLLM 自身的 Prometheus counters（
vllm:prompt_tokens_total
、
vllm:generation_tokens_total
、
vllm:request_success_total
）读取稳态窗口内的吞吐量。
sh
./scripts/bench.py --base-url "$URL" --model "$MODEL" \
  --input-lens 128,512 --concurrency 8,32,64 --duration 30
方法规则如下，每一条都用于修正一种实际失败模式：
  • 使用持续窗口，而不是固定请求数。 少于 10s 的窗口会被抖动主导。测量窗口前应先进行预热，并确保窗口时长 >=20s。
  • 使用服务端 counters，而不是客户端计时。 这样可以从测量中排除客户端开销和 WAN 延迟。
  • 持续扫描 concurrency，直到吞吐量不再上升。 单一 concurrency 水平无法告诉你系统在哪里达到饱和。
  • 使用唯一的 prompt 前缀。 相同 prompt 会命中 prefix cache 并夸大结果（附带脚本会为每个请求生成不同前缀）。
  • 留意客户端瓶颈。 如果 concurrency 较高时吞吐量反而下降，应先怀疑负载生成器的 CPU，而不是服务器。请在配置更高的机器上重新运行，或从集群内部运行。
压测驱动程序的运行位置。 可以先从 laptop 开始。为了排除网络路径的影响，请将其作为 container 在集群内运行（
--cpu 6
，vLLM 镜像已经包含 Python + httpx），并将其指向服务器的 pod IP（
kubectl get pod <podName> -o jsonpath='{.status.podIP}'
），然后再对公开的 
webUrl
重复测试。如果两者结果一致，说明 ingress 不是瓶颈——应当实测确认，而不是直接假设。

4. Confirm the GPU is the limiter

4. 确认 GPU 确实是瓶颈

A throughput number is only a hardware number if the hardware was saturated. Check DCGM on the node running the model:
sh
NODE=$(kubectl -n ns-<env> get pod <podName> -o jsonpath='{.spec.nodeName}')
DCGM=$(kubectl -n chalk-telemetry get pods -o wide | grep "dcgm.*$NODE" | awk '{print $1}')
kubectl -n chalk-telemetry port-forward "$DCGM" 19400:9400 &
curl -s localhost:19400/metrics | grep -E '^DCGM_FI_DEV_(GPU_UTIL|POWER_USAGE|FB_USED)'
Sample it during the run. 
GPU_UTIL
at 100 and 
POWER_USAGE
at the card's TDP (72W for L4, 300W for A100 40GB) means you are GPU-bound and the number is real. Utilization well under 100% means you measured your client, the network, or a scheduler cap — not the GPU. Note the exporter scrapes on an interval, so identical consecutive samples are expected.
只有在硬件达到饱和时，吞吐量数字才真正代表硬件性能。请检查运行模型的节点上的 DCGM：
sh
NODE=$(kubectl -n ns-<env> get pod <podName> -o jsonpath='{.spec.nodeName}')
DCGM=$(kubectl -n chalk-telemetry get pods -o wide | grep "dcgm.*$NODE" | awk '{print $1}')
kubectl -n chalk-telemetry port-forward "$DCGM" 19400:9400 &
curl -s localhost:19400/metrics | grep -E '^DCGM_FI_DEV_(GPU_UTIL|POWER_USAGE|FB_USED)'
应在测试运行期间采样。
GPU_UTIL
达到 100，且 
POWER_USAGE
达到 GPU 卡的 TDP（L4 为 72W，A100 40GB 为 300W），意味着系统受 GPU 限制，所得数字是真实的。如果利用率明显低于 100%，则你测到的是客户端、网络或 scheduler 限制，而不是 GPU。请注意，exporter 按固定间隔抓取 metrics，因此连续出现相同采样值属于正常现象。

5. Get the real $/hr

5. 获取真实的 $/hr

Do not trust third-party pricing pages or a web search summary. For GPU machine families the accelerator is often a separate SKU from vCPU and RAM, and aggregator sites frequently omit it — which understates the node by 3x.
Query GCP's Cloud Billing Catalog directly (Compute Engine service 
6F81-5844-456A
), summing the SKUs that make up the node:
py
undefined
不要信任第三方定价页面或网页搜索摘要。 对于 GPU 机器族，accelerator 往往是与 vCPU 和 RAM 分开的独立 SKU，而聚合定价网站经常将其遗漏——这会使节点成本被低估 3 倍。
直接查询 GCP 的 Cloud Billing Catalog（Compute Engine service 
6F81-5844-456A
），并汇总构成该节点的所有 SKU：
py
undefined

paginate skus?pageSize=5000&currencyCode=USD with an Authorization: Bearer token

paginate skus?pageSize=5000&currencyCode=USD with an Authorization: Bearer token

from 
gcloud auth print-access-token
, filter serviceRegions for your region,

from 
gcloud auth print-access-token
, filter serviceRegions for your region,

then match description against your machine family and accelerator.

then match description against your machine family and accelerator.


Worked example, `g2-standard-8` (8 vCPU, 32 GiB, 1x L4) in `us-central1`:

| SKU | Rate | Qty | Subtotal |
|---|---|---|---|
| G2 Instance Core running in Americas | $0.024988 /hr | 8 | $0.199904 |
| G2 Instance Ram running in Americas | $0.002927 /GiB-hr | 32 | $0.093664 |
| **Nvidia L4 GPU running in Americas** | **$0.560040 /hr** | 1 | **$0.560040** |
| | | | **$0.853608 /hr** |

Omitting the GPU SKU yields $0.2936/hr — a number that appears on public pricing
aggregators and is wrong. Sanity-check by computing spot the same way and
confirming it lands at the expected discount.

Also pull Spot and 1yr/3yr CUD rates; they are separate SKUs and are what a
production deployment would actually pay.

示例：`g2-standard-8`（8 vCPU、32 GiB、1x L4），位于 `us-central1`：

| SKU | 费率 | 数量 | 小计 |
|---|---|---|---|
| G2 Instance Core running in Americas | $0.024988 /hr | 8 | $0.199904 |
| G2 Instance Ram running in Americas | $0.002927 /GiB-hr | 32 | $0.093664 |
| **Nvidia L4 GPU running in Americas** | **$0.560040 /hr** | 1 | **$0.560040** |
| | | | **$0.853608 /hr** |

省略 GPU SKU 后得到 $0.2936/hr——这个数字会出现在公共定价聚合网站上，但它是错误的。请使用相同方法计算 spot 价格进行合理性检查，并确认其降幅符合预期。

还应获取 Spot 和 1yr/3yr CUD 费率；它们属于独立 SKU，也是生产环境实际会支付的价格。

6. Compute the cost

6. 计算成本

$/sec        = node_hourly / 3600
$/1M in-tok  = $/sec / prompt_tok_per_s * 1e6
$/1M req     = $/sec / rps * 1e6
Report $/1M input tokens and $/1M requests, plus the on-demand / spot / CUD variants.
Pick the right unit. For a classifier or guardrail that emits a single token, output tokens are identical to requests and "$/output token" is a degenerate metric — lead with $/classification. For a chat or coding model, decode dominates and $/output token is the meaningful figure; set 
--output-tokens
to a realistic completion length rather than 1.
$/sec        = node_hourly / 3600
$/1M in-tok  = $/sec / prompt_tok_per_s * 1e6
$/1M req     = $/sec / rps * 1e6
应报告 $/1M input tokens 和 $/1M requests，同时提供按需、spot 和 CUD 变体。
选择正确的单位。 对于只输出单个 token 的分类器或 guardrail，output tokens 与 requests 相同，“$/output token”是一个退化指标——应重点报告 $/classification。对于聊天或编程模型，decode 占主导，因此 $/output token 才是有意义的指标；应将 
--output-tokens
设置为符合实际的 completion 长度，而不是 1。

Pitfalls

常见陷阱

  • Prefix caching inflates $/input-token. A shared system prompt gets cached, so 
    prompt_tokens_total
    counts tokens that were never recomputed. The bundled script reports the hit rate. It is a fair comparison against a vendor who bills you for every input token, but it is not your real compute headroom — say which one you are quoting.
  • Saturation numbers are a floor, not a forecast. These are 100%-duty-cycle figures. At 20% real utilization, multiply by 5. Idle GPU time is the dominant cost in most production deployments.
  • Tool-calling models need the right parser and the right checkpoint. 
    --enable-auto-tool-choice --tool-call-parser hermes
    plus an instruct model. Some Coder-tuned checkpoints emit tool calls as fenced JSON text that the parser ignores, so 
    tool_calls
    returns null under 
    tool_choice: auto
    while 
    tool_choice: required
    works (guided decoding forces the shape).
  • Completed container pods are garbage-collected, taking their logs. End a one-shot driver script with 
    sleep
    so you can still 
    kubectl logs
    it.
  • chalk container exec
    is unimplemented for k8s containers     (
    only implemented for host containers
    ). Drive work via the entrypoint, or use 
    --compute-class host
    / the sandbox API.
  • Piping a script into 
    bash
    breaks silently     if it invokes anything that reads stdin — the child eats the rest of the script. Write it to a file and add 
    exec </dev/null
    .
  • Sandboxes have no network egress and no flag to grant it; even public IPs are blocked. Use 
    chalk container run --route <CIDR>
    when the workload must reach anything.
  • Prefix caching 会抬高 $/input-token。 共享 system prompt 会被缓存，因此 
    prompt_tokens_total
    会统计从未重新计算的 token。附带脚本会报告命中率。与按每个 input token 向用户收费的 vendor 比较时，这仍是公平的对比，但它不是你真实的计算余量——应明确说明所引用的是哪一种口径。
  • 饱和状态下的数字是下限，而不是预测值。 这些是 100% duty cycle 下的数据。如果实际利用率为 20%，则需要乘以 5。在大多数生产部署中，空闲 GPU 时间才是主要成本。
  • Tool-calling 模型需要正确的 parser 和 checkpoint。 使用 
    --enable-auto-tool-choice --tool-call-parser hermes
    ，并搭配 instruct 模型。部分经过 Coder 调优的 checkpoint 会将 tool call 输出为围栏形式的 JSON 文本，导致 parser 忽略它；因此在 
    tool_choice: auto
    下，
    tool_calls
    会返回 null，而 
    tool_choice: required
    可以正常工作（guided decoding 会强制输出结构）。
  • 已完成的 container pod 会被垃圾回收，并连同日志一起删除。 在一次性驱动脚本末尾加入 
    sleep
    ，以便之后仍可执行 
    kubectl logs
    。
  • chalk container exec
    尚未针对 k8s container 实现    （
    only implemented for host containers
    ）。请通过 entrypoint 驱动工作，或使用 
    --compute-class host
    / sandbox API。
  • 如果脚本通过管道传递给 
    bash
    后调用了任何读取 stdin 的内容，就会静默中断    ——子进程会吞掉脚本的剩余部分。请将脚本写入文件，并添加 
    exec </dev/null
    。
  • Sandbox 没有网络出口，也没有可用于授权网络出口的参数；即使公共 IP 也会被阻止。当工作负载必须访问外部资源时，请使用 
    chalk container run --route <CIDR>
    。

Reporting

报告要求

State the model, GPU, every non-default serving flag, the measured throughput, the DCGM saturation evidence, and the SKU breakdown behind the $/hr. A cost-per-token figure without those is not reproducible and should not be quoted.
应说明模型、GPU、每个非默认服务参数、实测吞吐量、DCGM 饱和度证据，以及 $/hr 背后的 SKU 明细。缺少这些信息的每 token 成本数据不可复现，不应被引用。