quantized-export

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Quantized Export

量化导出

The last stop after

checkpoint-promotion

hands off a

PROMOTE

verdict: a checkpoint that cleared the four-stage gate still isn't deployed until it's exported in the right format for its target runtime and proven to still work post-export. A

REJECT

verdict never reaches this skill — export starts only from a promoted checkpoint.

Input: a promoted checkpoint (or LoRA adapter) plus the target deployment surface — GPU class, serving stack, and whether long-context/code/math workloads are in scope. Output format: an exported artifact in the chosen format plus a smoke-test diff report comparing 3–5 golden outputs pre-export and post-export.

checkpoint-promotion

之后的最后一步是交付

PROMOTE

verdict：即使检查点通过了四阶段审核，也必须导出为适配目标运行时的格式，并验证导出后仍能正常工作，才算完成部署。获得

REJECT

verdict的检查点不会进入此环节——仅从已通过升级的检查点开始导出。

输入：已通过升级的检查点（或LoRA适配器），以及目标部署环境——GPU类别、服务栈，是否涉及长上下文/代码/数学工作负载。 输出格式：所选格式的导出产物，以及一份冒烟测试差异报告，对比导出前后3-5条黄金输出结果。

Format Map

格式映射

Pick format by hardware and deployment shape, not by habit — the wrong pick either wastes throughput headroom or breaks silently on specific workloads (see Workload Overrides).

FP8 is the default on Hopper-class GPUs and newer. It preserves near-bf16 quality at roughly half the memory, and it's the safe first choice whenever the target GPU supports it and no edge-device constraint applies.
AWQ INT4 targets older GPUs that predate FP8 hardware support. GPTQ is superseded for new deployments — don't reach for it on a fresh export; AWQ has better accuracy retention at the same bit width and wider current tooling support.
GGUF with Q4_K_M quantization, built from an imatrix, is the edge/llama.cpp format. Use it for local or CPU-adjacent deployment, not for GPU-serving throughput — it optimizes for footprint, not tokens/sec on a datacenter GPU.
NVFP4 is for Blackwell-at-scale deployments only — and explicitly NOT on GB10. NVFP4 on SM121 (GB10) runs ~32% slower than FP8 because the hardware lacks a native
```
cvt.e2m1x2
```
path unless the kernel is compiled
```
sm_121a
```
. Choosing NVFP4 on a GB10 target is a regression, not an upgrade — pick FP8 there instead.
Merged vs. LoRA-only is a separate axis from quant format. A merged export folds the adapter into the base weights: larger artifact, no base-model dependency at serve time. LoRA-only keeps the adapter separate: much smaller artifact, but the serving stack must load the exact same base model alongside it — a mismatched or wrong-revision base silently changes outputs. Pick merged when artifact portability matters more than storage; pick LoRA-only when disk footprint or multi-adapter serving matters more.

需根据硬件和部署场景选择格式，而非凭习惯——错误的选择要么浪费吞吐量余量，要么在特定工作负载下静默失效（详见工作负载覆盖规则）。

FP8是Hopper级及更新款GPU的默认格式。它在内存占用约为bf16一半的情况下，保留了接近bf16的质量，只要目标GPU支持且无边缘设备限制，它就是安全的首选。
AWQ INT4针对不支持FP8硬件的旧代GPU。GPTQ已被新部署场景淘汰——新导出时不要选用它；AWQ在相同位宽下保留了更高的精度，且当前工具支持更广泛。
带imatrix的GGUF Q4_K_M量化格式是边缘设备/llama.cpp专用格式。适用于本地或CPU邻近部署，不适用于GPU服务吞吐量优化——它针对内存占用优化，而非数据中心GPU的令牌/秒性能。
NVFP4仅适用于大规模Blackwell部署——明确禁止在GB10上使用。在SM121（GB10）上运行NVFP4比FP8慢约32%，因为除非内核编译为
```
sm_121a
```
，否则硬件缺乏原生
```
cvt.e2m1x2
```
路径。在GB10目标设备上选择NVFP4是性能倒退而非升级——应改用FP8。
合并格式与仅LoRA格式是独立于量化格式的选择维度。合并导出会将适配器折叠到基础权重中：产物体积更大，但服务时无需依赖基础模型。仅LoRA格式保持适配器独立：产物体积小得多，但服务栈必须加载完全相同的基础模型——不匹配或版本错误的基础模型会静默改变输出结果。当产物可移植性比存储更重要时选合并格式；当磁盘占用或多适配器服务更重要时选仅LoRA格式。

Worked Picks

示例选择

The core format-selection tradeoff, read as a lookup table for common scenarios:

Target	Workload	Format
Datacenter GPU	generic chat	FP8
Datacenter GPU	long-context/code/math	FP8 or W8A8 — never INT4
Older GPU generation	generic	AWQ INT4
Edge device / laptop	llama.cpp serving	GGUF Q4_K_M + imatrix
GB10	any workload	FP8 via vLLM nightly, or GGUF via llama.cpp locally — skip NVFP4

yaml

undefined

核心格式选择的权衡关系，可作为常见场景的查询表：

目标设备	工作负载	格式
数据中心GPU	通用对话	FP8
数据中心GPU	长上下文/代码/数学	FP8或W8A8——绝不要用INT4
旧代GPU	通用场景	AWQ INT4
边缘设备/笔记本电脑	llama.cpp服务	GGUF Q4_K_M + imatrix
GB10	任意工作负载	通过vLLM nightly使用FP8，或本地通过llama.cpp使用GGUF——跳过NVFP4

yaml

undefined

quick decision snippet — see the table above for the full map

快速决策代码片段——完整映射见上表

hopper_or_newer: fp8 older_gpu: awq-int4 edge_llama_cpp: gguf-q4_k_m+imatrix gb10_any_workload: fp8-vllm-nightly # never nvfp4 on GB10

undefined

hopper_or_newer: fp8 older_gpu: awq-int4 edge_llama_cpp: gguf-q4_k_m+imatrix gb10_any_workload: fp8-vllm-nightly # 绝不要在GB10上使用nvfp4

undefined

Workload Overrides

工作负载覆盖规则

The Format Map above is a default, not a rule that survives every workload. Long-context, code, and math workloads break at INT4 — quantization error compounds across long sequences and precise token-level reasoning in ways that don't show up on short, generic prompts. For any of these three workload classes, stay on FP8 or W8A8 even if the target hardware would otherwise justify INT4 on cost grounds.

Don't validate this override with MMLU or similar broad-knowledge benchmarks — they don't stress the failure mode. Measure with the actual task evals — the goldens and graders from
```
eval-harness-first
```
, run through the exported artifact — because INT4 degradation on long-context, code, or math shows up as task-specific failures (dropped context, broken syntax, arithmetic errors) well before it moves a knowledge benchmark.
If a task eval regresses after an INT4 export on one of these three workload classes, the fix is switching format, not re-tuning the quantization recipe — AWQ and GPTQ variants at the same bit width share the same compounding-error failure mode on these workloads.

上述格式映射是默认规则，并非适用于所有工作负载。长上下文、代码和数学工作负载在INT4格式下会失效——量化误差会在长序列和精确令牌级推理中累积，而这些问题在简短的通用提示中不会显现。对于这三类工作负载，即使目标硬件从成本角度适合INT4，也应坚持使用FP8或W8A8。

不要用MMLU或类似的广泛知识基准验证此规则——它们不会触发失效模式。需用实际任务评估来衡量——使用
```
eval-harness-first
```
维护的
```
eval/goldens.jsonl
```
中的黄金样本和评估器，运行导出产物进行测试，因为INT4在长上下文、代码或数学工作负载下的性能退化会表现为特定任务失败（丢失上下文、语法错误、算术错误），远早于知识基准分数下降。
如果在这三类工作负载下，INT4导出后的任务评估出现性能退化，解决方案是切换格式，而非重新调整量化方案——相同位宽的AWQ和GPTQ变体在这些工作负载下存在相同的误差累积失效模式。

The Smoke Test

冒烟测试

Export bugs are silent at the file level — a malformed export still produces a loadable artifact, so file-existence checks prove nothing. The smoke test is mandatory for every export, with no exception for a format that "should just work":

Load the exported artifact in its actual target runtime — vLLM for FP8/AWQ, llama.cpp for GGUF, not a quick sanity load in a different framework than the one that will serve it in production.
Run 3–5 golden prompts through it — pull these from the same
```
eval/goldens.jsonl
```
```
eval-harness-first
```
maintains, not a fresh ad hoc set.
Compare each output against the pre-export generation for the same prompt, same deterministic sampling settings — greedy decoding (temperature 0) and a fixed seed, persisted and reused between the pre- and post-export runs, not just nominally identical config. For a lossless export, byte match is the gate — any diff is a bug. For a lossy (quantized) export, byte match is expected to fail; the gate is task-grader verdict agreement instead — see
```
references/export-commands.md
```
's Smoke-Test Script Skeleton.

Run this as a gate, not a manual check:

bash

python smoke_test.py "$EXPORT_PATH" \
    eval/goldens.jsonl pre-export-outputs.jsonl

导出错误在文件层面是静默的——格式错误的导出仍会生成可加载的产物，因此文件存在性检查毫无意义。每次导出都必须执行冒烟测试，没有任何“理应正常工作”的格式例外：

在实际目标运行时中加载导出产物——FP8/AWQ用vLLM，GGUF用llama.cpp，不要用与生产服务框架不同的工具进行快速 sanity 加载。
运行3-5条黄金提示——从
```
eval-harness-first
```
维护的
```
eval/goldens.jsonl
```
中提取，而非临时生成新的提示集。
将每个输出与导出前的生成结果对比——使用相同的确定性采样设置：贪婪解码（temperature 0）和固定种子，该种子需在导出前后的运行中持久化复用，而非仅名义上相同的配置。对于无损导出，字节级匹配是通过标准——任何差异都是bug。对于有损（量化）导出，字节级匹配必然失败，此时通过标准为任务评估器 verdict 一致——详见
```
references/export-commands.md
```
中的冒烟测试脚本框架。

将此作为强制关卡，而非手动检查：

bash

python smoke_test.py "$EXPORT_PATH" \
    eval/goldens.jsonl pre-export-outputs.jsonl

non-zero exit on any pre/post mismatch

若导出前后结果不匹配，返回非零退出码

undefined

undefined

Failure Signatures

失效特征

What export bugs actually look like, not a clean pass/fail flag:

Template mismatch presents as garbled or run-on output — the chat template baked into the export doesn't match the one the checkpoint was trained and evaluated against, so turn boundaries or special tokens land in the wrong place.
Wrong quantization applied to
lm_head
presents as off-template or semantically nonsensical output that still looks fluent — the output head lost precision it needed even though the rest of the network quantized cleanly.

Never ship an export that skipped this step — a checkpoint's

PROMOTE

verdict says the un-exported checkpoint is good; it says nothing about the export pipeline. Re-run on any quant-method or runtime version bump, not only after the first export. Runnable command sequences for every format plus the smoke-test script skeleton:

references/export-commands.md

导出错误的实际表现，而非清晰的通过/失败标志：

模板不匹配表现为输出混乱或连续无意义——导出中嵌入的对话模板与检查点训练和评估时使用的模板不匹配，导致轮次边界或特殊令牌位置错误。
lm_head
应用了错误的量化表现为输出符合语法但语义无意义——输出头丢失了所需的精度，尽管网络其余部分量化正常。

绝不要跳过此步骤就发布导出产物——检查点的

PROMOTE

verdict仅表明未导出的检查点是合格的，对导出流水线不做任何保证。不仅首次导出后要运行，每次量化方法或运行时版本更新后都要重新运行。所有格式的可运行命令序列及冒烟测试脚本框架见：

references/export-commands.md

。