mlx-serving

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Apple Silicon LLM Serving (MLX / oMLX)

Apple Silicon LLM 服务（MLX / oMLX）

Operational guidance for running and tuning local LLM inference on Apple Silicon (M-series). Covers two common backends: Apple's

mlx-lm

and the oMLX server built on top of MLX.

Both expose OpenAI-compatible HTTP APIs, so client code is portable. The differences are in features, packaging, and the failure modes you hit when pushing models near the limits of unified memory.

在 Apple Silicon（M 系列）上运行和调优本地 LLM 推理的运维指南。涵盖两个常见后端：Apple 的

mlx-lm

，以及构建于 MLX 之上的 oMLX server。

两者都提供兼容 OpenAI 的 HTTP API，因此客户端代码可以移植使用。它们的差异在于功能、打包方式，以及模型逼近统一内存上限时遇到的故障模式。

Pre-Flight: Verify Environment

预检：验证环境

Before debugging any serving issue, confirm the basics:

bash

undefined

在调试任何 serving 问题之前，先确认基础环境：

bash

undefined

Hardware: confirm Apple Silicon and unified-memory size

system_profiler SPHardwareDataType | grep -E "Model|Chip|Memory"

Activate the venv where mlx-lm is installed before the next two probes,

otherwise they hit system Python and may report nothing useful.

python -c "import mlx.core as mx; print(mx.version, mx.default_device())" python -c "import mlx_lm; print(mlx_lm.version)"

oMLX (DMG install): inspect available subcommands and run health checks

omlx --help omlx diagnose menubar # checks Tahoe ControlCenter visibility (oMLX-specific)

Confirm a process is actually listening

lsof -nP -iTCP -sTCP:LISTEN | grep -E "omlx|mlx_lm"


Verify which backend is bound before debugging: both can serve OpenAI-
compatible endpoints on user-chosen ports, and a client request hitting the
wrong backend silently produces correct-looking but unrelated output.

---

lsof -nP -iTCP -sTCP:LISTEN | grep -E "omlx|mlx_lm"


在调试前，先确认实际绑定的是哪个后端：两者都可以在用户指定的端口提供兼容 OpenAI 的 endpoint，而客户端请求如果命中错误的后端，会静默返回看似合理却与目标无关的输出。

---

Backend Decision: mlx-lm vs oMLX

后端选择：mlx-lm vs oMLX

Concern	mlx-lm	oMLX
Install	`pip install mlx-lm`	DMG (macOS app bundle)
Process model	One model per `mlx_lm.server` process	Built-in registry; multi-model
Config surface	Minimal CLI flags	Rich per-model JSON ( `model_settings.json` )
Feature flags	`--temp` , `--top-p` , basic	turboquant_kv, dflash, MTP, specprefill, thinking budget
Tool calling	Format depends on chat template	Per-model parser (configurable)
Models > RAM	Not supported (OOM on load)	OptiQ proxy build (sensitivity-driven quant)
GUI	None	Native macOS app + menu bar
Default port	8080	8000
Best for	Library-first scripts, embedding in Python	Long-running daemon, multiple models, GUI ops

Skill bias: the depth here leans oMLX — feature flags, cache controls, upstream bug patterns. mlx-lm coverage is lighter and focuses on the shared substrate.

Pick mlx-lm when: you control the Python process, you want minimal surface area, you serve one model at a time, and the model fits comfortably in RAM.

Pick oMLX when: you need a daemon that survives terminal sessions, you serve multiple models behind one endpoint, you want feature flags (turboquant, dflash, MTP) without writing them yourself, or your model exceeds RAM and you need OptiQ proxy.

Both backends share the same fundamental constraints (unified memory, KV cache pressure, model architecture quirks), so most troubleshooting in this skill applies to either.

关注点	mlx-lm	oMLX
安装方式	`pip install mlx-lm`	DMG（macOS app bundle）
进程模型	每个 `mlx_lm.server` 进程运行一个模型	内置 registry；支持多模型
配置范围	精简的 CLI flags	丰富的 per-model JSON（ `model_settings.json` ）
功能开关	`--temp` 、 `--top-p` 等基础选项	turboquant_kv、dflash、MTP、specprefill、thinking budget
工具调用	格式取决于 chat template	per-model parser（可配置）
模型占用超过 RAM	不支持（加载时 OOM）	OptiQ proxy build（基于敏感度进行量化）
GUI	无	原生 macOS app + 菜单栏
默认端口	8080	8000
最适合	以库优先的脚本、嵌入 Python	长时间运行的 daemon、多模型、GUI 运维

Skill 偏好： 本文更深入地偏向 oMLX——功能开关、缓存控制、上游 bug 模式。对 mlx-lm 的覆盖较轻，主要聚焦于两者共享的基础层。

如果由你掌控 Python 进程、希望保持尽可能小的配置范围、一次只服务一个模型，并且模型能够轻松装入 RAM，请选择 mlx-lm。

如果你需要一个能够脱离终端会话持续运行的 daemon、通过一个 endpoint 提供多个模型、希望直接使用 turboquant、dflash、MTP 等功能开关而无需自行实现，或者模型超过了 RAM 并需要使用 OptiQ proxy，请选择 oMLX。

两种后端共享相同的根本限制（统一内存、KV cache 压力、模型架构特性），因此本 Skill 中的大多数故障排查方法都适用于二者。

Symptom → Cause Triage

症状 → 原因分诊

Symptom	Likely cause	First move
OOM during model load	Model + KV scratch > available RAM	Smaller quant; OptiQ proxy (oMLX); free other apps
OOM during long-context inference	KV cache growth	Enable `turboquant_kv` ; set `--paged-ssd-cache-dir` for spill-to-SSD (CLI flag is the operative control; `dflash_*` keys are version-dependent overrides)
Crash / hang at batch > 1	Architecture cache shape mismatch	Pin `--max-concurrent-requests 1` (oMLX); see upstream-bug-patterns
Tool calls return as plaintext content	Parser doesn't recognize the model's tool-call JSON shape	See upstream-bug-patterns; check `tool_calls` field in response
Throughput regression after config change	Feature flag interaction	Bisect via bench, one flag at a time
Server hangs on startup	Large checkpoint loading from cold disk	Wait; tail `server.log` ; check disk I/O via `iostat`
Wrong / garbled outputs	Quantization too aggressive, or wrong chat template	Reduce `turboquant_kv_bits` or disable; verify chat template matches model card
First request slow, rest fast	Cold prefix cache	Enable `specprefill` ; warm cache with a dummy request
Output cut off mid-sentence	`max_tokens` too low; thinking budget exhausted	Raise `max_tokens` ; check `thinking_budget_enabled`

For deeper triage on each symptom:

references/symptoms.md

症状	可能原因	首要操作
模型加载时 OOM	模型 + KV scratch 超过可用 RAM	使用更小的量化版本；使用 OptiQ proxy（oMLX）；退出其他应用以释放内存
长上下文推理期间 OOM	KV cache 增长	启用 `turboquant_kv` ；设置 `--paged-ssd-cache-dir` 以将数据溢出到 SSD（CLI flag 才是实际生效的控制项； `dflash_*` keys 是取决于版本的覆盖项）
batch > 1 时崩溃或挂起	架构的缓存形状不匹配	将 `--max-concurrent-requests` 固定为 1（oMLX）；参见 `upstream-bug-patterns`
工具调用以纯文本 content 返回	Parser 无法识别模型的工具调用 JSON 结构	参见 `upstream-bug-patterns` ；检查响应中的 `tool_calls` 字段
修改配置后吞吐量回退	功能开关之间相互作用	每次只修改一个开关，通过 benchmark 进行二分排查
服务器启动时挂起	正在从冷磁盘加载大型 checkpoint	等待；跟踪 `server.log` ；通过 `iostat` 检查磁盘 I/O
输出错误或乱码	量化过于激进，或 chat template 错误	降低 `turboquant_kv_bits` 或将其禁用；确认 chat template 与 model card 匹配
第一次请求较慢，后续请求较快	冷 prefix cache	启用 `specprefill` ；使用 dummy request 预热 cache
输出在句子中间被截断	`max_tokens` 太低；thinking budget 已耗尽	提高 `max_tokens` ；检查 `thinking_budget_enabled`

如需对每种症状进行更深入的分诊，请参阅

references/symptoms.md

。

Core Principles

核心原则

Lower
--max-concurrent-requests
to 1 for batch-fragile architectures. oMLX defaults to
```
--max-concurrent-requests 8
```
. Higher concurrency exposes architecture-specific batch handling bugs (notably in cache implementations like ChunkedKVCache used by Llama-4). For affected architectures, pin concurrency to 1 at launch and bench upward only after confirming clean output at batch>1.
One flag at a time. Feature flags interact (e.g.,
```
dflash
```
+
```
turboquant_kv
```
both touch the KV cache path). Changing two flags simultaneously and observing a regression makes attribution impossible. Bench between each change.
Bench, don't guess. "It feels faster" is not a tuning signal. Maintain a small bench suite (chat / coding / tool-calling correctness + throughput) and re-run it after every config change. See
```
references/bench-methodology.md
```
.
Match chat templates to model cards. Tool-calling and reasoning behavior depend on the chat template applied at request time. The model's HuggingFace card is authoritative.
Quantization is a curve, not a switch.
```
turboquant_kv_bits=4.0
```
keeps most quality; lower bits trade quality for memory. Always include a quality cell in the bench suite — a config that passes throughput but fails coding can be worse than slower-but-correct.
OptiQ proxy is for the "model > RAM" case only. It builds a sensitivity-driven proxy of the model so per-layer quant decisions can be made without holding the full model in memory. It's not a general speedup; for in-RAM models it adds startup cost without runtime benefit.
Server-side bugs masquerade as config problems. When a symptom reproduces across multiple config combinations, suspect upstream rather than chasing more flags. See
```
references/upstream-bug-patterns.md
```
.

对于对 batch 敏感的架构，将
--max-concurrent-requests
降至 1。 oMLX 默认使用
```
--max-concurrent-requests 8
```
。更高的并发数会暴露特定架构的批处理 bug，尤其是 Llama-4 使用的 ChunkedKVCache 等缓存实现。对于受影响的架构，启动时应将并发数固定为 1；只有在确认 batch>1 时能够稳定产生正确输出后，才通过 benchmark 逐步提高并发数。
每次只修改一个开关。 功能开关之间会相互影响，例如
```
dflash
```
和
```
turboquant_kv
```
都会访问 KV cache 路径。同时修改两个开关并观察到性能回退，将无法确定问题来源。每次修改之间都应进行 benchmark。
进行 benchmark，不要靠猜。 “感觉更快”并不是有效的调优信号。应维护一个小型 benchmark suite（对话、编程、工具调用正确性 + 吞吐量），并在每次修改配置后重新运行。参见
```
references/bench-methodology.md
```
。
让 chat template 与 model card 匹配。 工具调用和推理行为取决于请求时应用的 chat template。模型的 HuggingFace card 是权威依据。
量化是一条曲线，而不是一个开关。
```
turboquant_kv_bits=4.0
```
通常可以保留大部分质量；更低的 bits 则以质量换取内存节省。benchmark suite 中应始终包含质量测试项——一个吞吐量合格但编程任务失败的配置，可能比速度较慢但结果正确的配置更糟糕。
OptiQ proxy 仅用于“模型占用超过 RAM”的情况。 它会构建模型的 sensitivity-driven proxy，从而无需在内存中完整加载模型即可逐层作出量化决策。它不是通用加速方案；对于能够装入 RAM 的模型，它只会增加启动成本，而不会带来运行时收益。
服务端 bug 会伪装成配置问题。 如果某种症状在多种配置组合下都能复现，应怀疑上游问题，而不是继续尝试更多开关。参见
```
references/upstream-bug-patterns.md
```
。

References

参考资料

```
references/symptoms.md
```
— symptom triage with diagnostics and fixes (OOM, batch>1 crashes, tool-call-as-text, throughput regression, log triage)
```
references/omlx-feature-flags.md
```
— per-flag reference (turboquant_kv, dflash, MTP, specprefill, thinking budget, max-concurrent-requests, force_sampling) with interactions
```
references/bench-methodology.md
```
— bench-it-don't-guess: suite design, backend-agnostic harness shape, scoring, change attribution
```
references/upstream-bug-patterns.md
```
— when to suspect upstream vs config; two recurring bug patterns (cache-shape mismatch under batched scheduling; model tool-call format not parsed) with diagnostics and reporting guidance

```
references/symptoms.md
```
——包含诊断与修复方法的症状分诊指南，涵盖 OOM、batch>1 崩溃、工具调用作为文本返回、吞吐量回退和日志排查
```
references/omlx-feature-flags.md
```
——逐个功能开关的参考说明，包括 turboquant_kv、dflash、MTP、specprefill、thinking budget、max-concurrent-requests 和 force_sampling，以及它们之间的相互作用
```
references/bench-methodology.md
```
——“进行 benchmark，不要靠猜”：suite 设计、后端无关的 harness 结构、评分和变更归因
```
references/upstream-bug-patterns.md
```
——何时应怀疑上游问题而非配置问题；介绍两种反复出现的 bug 模式（批处理调度下的缓存形状不匹配，以及模型的工具调用格式未被解析），并提供诊断和报告指导

What This Skill Does Not Cover

本 Skill 不涵盖的内容

Cloud LLM hosting (Bedrock, OpenAI API, Anthropic API) — different surface entirely
Non-MLX local backends (llama.cpp, Ollama, vLLM) — overlapping problem space but different tooling
Model training, fine-tuning, or LoRA — see MLX training tutorials
General Python / HTTP debugging unrelated to MLX
Model card creation or HuggingFace upload workflows

云端 LLM 托管（Bedrock、OpenAI API、Anthropic API）——完全不同的服务层
非 MLX 本地后端（llama.cpp、Ollama、vLLM）——问题范围存在重叠，但工具链不同
模型训练、微调或 LoRA——请参阅 MLX 训练教程
与 MLX 无关的一般 Python / HTTP 调试
Model card 创建或 HuggingFace 上传工作流