input-token-overheads
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseInput Token Overheads
输入Token开销
Audit every source of per-turn input token cost on a Hermes Agent instance. Measure each, rank by cost, act on the top consumers.
审计Hermes Agent实例中每轮输入Token成本的所有来源。对每个来源进行计量,按成本排序,针对占比最高的开销项采取优化措施。
When to Use
使用场景
- User says "token overhead", "context too large", "why is input so expensive"
- Model output quality degrades from context dilution
- Cost optimization — fewer input tokens per turn means lower API spend
- After adding skills, plugins, or tools — verify the overhead delta
- 用户提及“token开销”“上下文过大”“输入成本为何如此高昂”时
- 因上下文稀释导致模型输出质量下降时
- 成本优化需求——每轮输入Token越少,API花费越低
- 添加技能、插件或工具后——验证开销增量
The Overhead Map
开销明细
Every turn, Hermes injects these blocks into the system prompt before the user's message:
| Block | When loaded | Cost model |
|---|---|---|
| Skill descriptions | Every turn (skill-retrieval top-K) | ~200 chars per description, K per turn |
| Memory (personal notes) | Every turn | Static, grows with usage |
| User profile | Every turn | Static, grows as preferences accumulate |
| Memory provider context | Every turn (if memory plugin active) | Dynamic, 5 memories recalled by default |
| Tool schemas (direct) | Every turn | Full JSON schema per enabled tool |
| Deferred tool catalog | Every turn (if configured) | Name + description only |
| Mandatory skills | Every turn (if configured) | Full SKILL.md body |
| Platform formatting rules | Every turn | Fixed, platform-specific |
| Behavioral rules | Every turn | Fixed system prompt text |
| Full skill body | On-demand (skill_view) | Only when a skill is loaded |
| Compression summary | After threshold | Replaces older messages with a summary |
On-demand (not per-turn): full SKILL.md via , deferred tool schemas via , reference files via .
skill_viewtool_describeskill_view(file_path=...)每一轮对话中,Hermes都会在用户消息之前将以下内容块注入系统提示词:
| 内容块 | 加载时机 | 成本模型 |
|---|---|---|
| 技能描述 | 每轮(技能检索Top-K) | 每条描述约200字符,每轮加载K条 |
| 记忆(个人笔记) | 每轮 | 静态内容,随使用时长增长 |
| 用户档案 | 每轮 | 静态内容,随偏好积累增长 |
| 记忆提供者上下文 | 每轮(若记忆插件激活) | 动态内容,默认召回5条记忆 |
| 工具模式(直接加载) | 每轮 | 每个启用工具的完整JSON模式 |
| 延迟加载工具目录 | 每轮(若已配置) | 仅包含名称+描述 |
| 强制加载技能 | 每轮(若已配置) | 完整SKILL.md内容 |
| 平台格式规则 | 每轮 | 固定内容,平台专属 |
| 行为规则 | 每轮 | 固定系统提示文本 |
| 完整技能内容 | 按需加载(skill_view) | 仅在技能加载时注入 |
| 压缩摘要 | 达到阈值后 | 用摘要替代旧消息 |
按需加载(非每轮):通过加载完整SKILL.md,通过加载延迟工具模式,通过加载参考文件。
skill_viewtool_describeskill_view(file_path=...)Health Ratio
健康比率
The health metric is overhead ratio: overhead tokens divided by the model's context window. The absolute number matters for cost; the ratio matters for quality.
| Ratio | Rating | Notes |
|---|---|---|
| < 5% | Excellent | Most of the window available for conversation |
| 5-15% | Healthy | Normal for a capable agent with tools, skills, memory |
| 15-25% | Acceptable | Approaching the limit. Consider trimming. |
| > 25% | Unhealthy | Eats conversation capacity. Cost and quality risk. |
Why the ratio matters: Three studies confirm that input length degrades model performance independent of content quality:
-
Lost in the Middle (Liu et al., TACL 2023) — Models follow a U-shaped curve: best recall at the start and end of context, severe degradation in the middle. Overhead sits at the top of every turn, but it pushes conversation history into the degradation zone. arxiv.org/abs/2307.03172
-
Same Task, More Tokens (Levy et al., ACL 2024) — Reasoning performance degrades at input lengths far shorter than the model's stated maximum. The degradation appears even when the extra tokens are padding with no distracting content. The model's technical context window is not its effective context window. aclanthology.org/2024.acl-long.818
-
Context Length Alone Hurts (Du et al., EMNLP 2025) — Performance degrades 14-85% as input length increases, even when retrieval is perfect, irrelevant tokens are replaced with whitespace, or all tokens except relevant ones are masked. The sheer length of the input is itself a limitation. aclanthology.org/2025.findings-emnlp.1264
Cost compounding: Overhead is paid every turn. At 10k tokens over 100 turns, that is 1M input tokens spent on overhead alone. Reducing overhead by 2k tokens saves 200k tokens per 100-turn session.
Mitigations from the research:
| Finding | Source | Action |
|---|---|---|
| Models recall start and end of context best; middle degrades | Liu et al. 2023 | Keep overhead at the top (Hermes already does this). Avoid pushing critical conversation history into the middle — lower compression threshold if history is being compressed too aggressively |
| Reasoning degrades well below the stated context window maximum | Levy et al. 2024 | Treat the effective context window as 50-70% of the advertised maximum. Target an overhead ratio under 10% of the advertised window, not the effective one |
| Sheer input length hurts even with perfect retrieval and no distraction | Du et al. 2025 | Reduce overhead aggressively. Every 1k tokens of overhead removed improves reasoning quality, not just cost. The study's mitigation: prompt the model to recite key evidence before solving — equivalent to Hermes compression summarizing relevant context |
| Tool calling degrades 7-85% as tool catalog grows from 8k to 120k tokens | LongFuncEval (arxiv 2505.10570) | Keep the enabled toolset count low. Prefer deferred tools (loaded on demand) over always-on schemas. Disable unused toolsets |
健康指标为开销比率:开销Token数除以模型上下文窗口大小。绝对值关乎成本,比率关乎输出质量。
| 比率 | 评级 | 说明 |
|---|---|---|
| < 5% | 优秀 | 窗口大部分空间可用于对话 |
| 5-15% | 健康 | 具备工具、技能、记忆的智能Agent的正常水平 |
| 15-25% | 可接受 | 接近上限,考虑精简 |
| > 25% | 不健康 | 占用对话容量,存在成本与质量风险 |
比率的重要性:三项研究证实,输入长度会独立于内容质量降低模型性能:
-
Lost in the Middle(Liu等人,TACL 2023)——模型遵循U型曲线:对上下文开头和结尾的召回效果最佳,中间部分性能严重下降。每轮开销位于上下文顶部,但会将对话历史挤入性能下降区域。arxiv.org/abs/2307.03172
-
Same Task, More Tokens(Levy等人,ACL 2024)——在远低于模型标称最大长度的输入下,推理性能就会下降。即使额外Token是无干扰的填充内容,性能仍会下降。模型的技术上下文窗口不等于有效上下文窗口。aclanthology.org/2024.acl-long.818
-
Context Length Alone Hurts(Du等人,EMNLP 2025)——随着输入长度增加,性能下降14-85%,即使检索完美、无关Token替换为空格或除相关Token外全部掩码。输入长度本身就是限制因素。aclanthology.org/2025.findings-emnlp.1264
成本复利:开销每轮都会产生。100轮对话中每轮1万Token开销,仅开销就会消耗100万输入Token。每轮减少2000Token开销,100轮会话可节省20万Token。
研究给出的缓解措施:
| 研究发现 | 来源 | 行动建议 |
|---|---|---|
| 模型对上下文开头和结尾的召回效果最佳,中间部分性能下降 | Liu等人2023 | 将开销保留在顶部(Hermes已实现)。避免将关键对话历史挤入中间区域——若历史被过度压缩,降低压缩阈值 |
| 推理性能在远低于标称最大上下文窗口时就会下降 | Levy等人2024 | 将有效上下文窗口视为标称值的50-70%。目标开销比率低于标称窗口的10%,而非有效窗口 |
| 即使检索完美且无干扰,输入长度本身也会影响性能 | Du等人2025 | 积极减少开销。每减少1000Token开销,不仅降低成本,还能提升推理质量。研究中的缓解措施:让模型在解决问题前复述关键证据——相当于Hermes压缩总结相关上下文 |
| 工具目录从8k增长到120kToken时,工具调用性能下降7-85% | LongFuncEval (arxiv 2505.10570) | 保持启用工具集数量较少。优先使用延迟加载工具(按需加载)而非始终启用的模式。禁用未使用的工具集 |
Procedure
操作流程
1. Measure each overhead source
1. 计量各开销来源
Run the audit script to get real numbers:
bash
python3 -c "
import yaml, pathlib, glob, os, re运行审计脚本获取真实数据:
bash
python3 -c "
import yaml, pathlib, glob, os, re--- Skill descriptions (skill-retrieval index) ---
--- Skill descriptions (skill-retrieval index) ---
files = glob.glob(os.path.expanduser('~/.hermes/skills/**/SKILL.md'), recursive=True)
total_desc = 0; count = 0; by_cat = {}
for f in files:
try:
text = pathlib.Path(f).read_text()
m = re.match(r'^---\n(.?)\n---\n', text, re.DOTALL)
if not m: continue
fm = yaml.safe_load(m.group(1))
if not fm: continue
desc = fm.get('description', '')
if not desc: continue
cat = f.split('/skills/')[1].split('/')[0]
by_cat.setdefault(cat, [0,0]); by_cat[cat][0] += len(desc); by_cat[cat][1] += 1
total_desc += len(desc); count += 1
except Exception: pass
avg = total_desc // max(count, 1)
K = int(os.environ.get('SKILL_RETRIEVAL_TOP_K', '6'))
print(f'Skills: {count} total, {total_desc} chars in descriptions')
print(f' Top-K per turn: {K*avg} chars ({Kavg//4} tokens) at K={K}')
print(f' By category (top 5):')
for cat, (sz, cnt) in sorted(by_cat.items(), key=lambda x: -x[1][0])[:5]:
print(f' {sz:>6} chars ({cnt:>2} skills) {cat}')
files = glob.glob(os.path.expanduser('~/.hermes/skills/**/SKILL.md'), recursive=True)
total_desc = 0; count = 0; by_cat = {}
for f in files:
try:
text = pathlib.Path(f).read_text()
m = re.match(r'^---\n(.?)\n---\n', text, re.DOTALL)
if not m: continue
fm = yaml.safe_load(m.group(1))
if not fm: continue
desc = fm.get('description', '')
if not desc: continue
cat = f.split('/skills/')[1].split('/')[0]
by_cat.setdefault(cat, [0,0]); by_cat[cat][0] += len(desc); by_cat[cat][1] += 1
total_desc += len(desc); count += 1
except Exception: pass
avg = total_desc // max(count, 1)
K = int(os.environ.get('SKILL_RETRIEVAL_TOP_K', '6'))
print(f'Skills: {count} total, {total_desc} chars in descriptions')
print(f' Top-K per turn: {K*avg} chars ({Kavg//4} tokens) at K={K}')
print(f' By category (top 5):')
for cat, (sz, cnt) in sorted(by_cat.items(), key=lambda x: -x[1][0])[:5]:
print(f' {sz:>6} chars ({cnt:>2} skills) {cat}')
--- Disabled skills (savings) ---
--- Disabled skills (savings) ---
config_path = os.path.expanduser('~/.hermes/config.yaml')
if not os.path.exists(config_path):
print(' Config: ~/.hermes/config.yaml not found — skipping disabled/compression stats')
else:
try:
with open(config_path) as fh:
cfg = yaml.safe_load(fh)
if cfg is None:
cfg = {}
disabled = cfg.get('skills',{}).get('disabled',[]) or []
print(f' Disabled: {len(disabled)} skills (saves ~{len(disabled)*avg} chars)')
comp = cfg.get('compression',{}) or {}
print(f' Compression: threshold={comp.get("threshold")}, target_ratio={comp.get("target_ratio")}, protect_last={comp.get("protect_last_n")}')
except Exception as e:
print(f' Config parse error: {e}')
"
For memory provider counts (if Mnemosyne is installed):
```bash
python3 -c "
import sqlite3, os, glob
for db in glob.glob(os.path.expanduser('~/.hermes/**/mnemosyne.db'), recursive=True):
conn = sqlite3.connect(db); c = conn.cursor()
for t in ['working_memory','episodic_memory','canonical_facts','memoria_facts']:
try:
c.execute(f'SELECT COUNT(*) FROM {t}'); print(f' {t}: {c.fetchone()[0]} rows')
except: pass
conn.close()
"Done: skill descriptions, disabled count, and compression config measured. Tool schemas (#1) and behavioral rules (#2) are fixed costs — estimate from the model's system prompt or check in-session for the total. The script measures the variable sources (#6, #7); the fixed sources (#1-#5) require in-session inspection.
/tokensconfig_path = os.path.expanduser('~/.hermes/config.yaml')
if not os.path.exists(config_path):
print(' Config: ~/.hermes/config.yaml not found — skipping disabled/compression stats')
else:
try:
with open(config_path) as fh:
cfg = yaml.safe_load(fh)
if cfg is None:
cfg = {}
disabled = cfg.get('skills',{}).get('disabled',[]) or []
print(f' Disabled: {len(disabled)} skills (saves ~{len(disabled)*avg} chars)')
comp = cfg.get('compression',{}) or {}
print(f' Compression: threshold={comp.get("threshold")}, target_ratio={comp.get("target_ratio")}, protect_last={comp.get("protect_last_n")}')
except Exception as e:
print(f' Config parse error: {e}')
"
若安装了Mnemosyne,获取记忆提供者数据:
```bash
python3 -c "
import sqlite3, os, glob
for db in glob.glob(os.path.expanduser('~/.hermes/**/mnemosyne.db'), recursive=True):
conn = sqlite3.connect(db); c = conn.cursor()
for t in ['working_memory','episodic_memory','canonical_facts','memoria_facts']:
try:
c.execute(f'SELECT COUNT(*) FROM {t}'); print(f' {t}: {c.fetchone()[0]} rows')
except: pass
conn.close()
"完成:已计量技能描述、禁用技能数量和压缩配置。工具模式(#1)和行为规则(#2)为固定成本——可从模型系统提示词估算,或在会话中查看获取总数。脚本计量了可变来源(#6、#7);固定来源(#1-#5)需在会话中查看。
/tokens2. Rank by cost
2. 按成本排序
Sort all sources by tokens per turn. The typical ranking:
- Tool schemas — largest fixed cost. Scales with enabled toolset count.
- Behavioral rules + system prompt — fixed text.
- Mandatory skills — full SKILL.md body per mandatory skill.
- Memory + user profile — static blocks.
- Deferred tool catalog — name + description per deferred tool.
- Skill descriptions — skill-retrieval top-K injection.
- Memory provider context — dynamic recall, 5 by default.
Done: sources ranked. Top 3 are the optimization targets.
按每轮Token数对所有来源排序,典型排序如下:
- 工具模式——最大固定成本,随启用工具集数量增加而增长
- 行为规则+系统提示词——固定文本
- 强制加载技能——每个强制技能的完整SKILL.md内容
- 记忆+用户档案——静态内容块
- 延迟加载工具目录——每个延迟工具的名称+描述
- 技能描述——技能检索Top-K注入内容
- 记忆提供者上下文——动态召回,默认5条
完成:已对来源排序,前3项为优化目标。
3. Act on top consumers
3. 针对高开销项采取行动
Tool schemas (largest fixed cost):
- Audit enabled toolsets: in the dashboard
hermes tools - Disable unused toolsets (each removes 1-3 tool schemas from every turn)
- Use in config.yaml to control per-profile toolset access
platform_toolsets.cli - Prefer deferred tools (loaded on demand) over always-on tools
Memory blocks:
- Load for compression tuning
skill_view(name='hermes-compression-tuning') - Prune memory entries that are stale or duplicated
- Keep the memory block under its budget — if full, batch-remove stale entries before adding new ones
Skill descriptions:
- Disable unused skills in under
config.yaml— each removed skill saves ~200 chars from the retrieval indexskills.disabled - Keep descriptions concise — the skill-retrieval plugin truncates at 200 chars. Descriptions over 200 chars waste tokens without improving routing
Memory provider (if installed):
- Run consolidation to move working to episodic, reducing the working set
- Invalidate stale facts
- Lower the recall parameter if context is tight
limit
Done: at least one optimization applied to each top-3 source.
工具模式(最大固定成本):
- 审计启用的工具集:在控制台运行
hermes tools - 禁用未使用的工具集(每个禁用工具集可减少每轮1-3个工具模式)
- 在config.yaml中使用控制每个配置文件的工具集访问权限
platform_toolsets.cli - 优先使用延迟加载工具(按需加载)而非始终启用的工具
记忆块:
- 加载进行压缩调优
skill_view(name='hermes-compression-tuning') - 删除过期或重复的记忆条目
- 保持记忆块在预算内——若已满,在添加新条目前批量删除过期内容
技能描述:
- 在config.yaml的中禁用未使用的技能——每个禁用技能可从检索索引中节省约200字符
skills.disabled - 保持描述简洁——技能检索插件会截断至200字符,超过200字符的描述会浪费Token且无法提升路由效果
记忆提供者(若已安装):
- 运行合并操作,将工作记忆转移至情景记忆,减少工作集大小
- 作废过期事实
- 若上下文紧张,降低召回参数
limit
完成:已对前3项高开销来源各应用至少一项优化措施。
4. Verify the delta
4. 验证优化效果
Re-run the audit script from step 1. Compare token estimates before and after.
Done: before/after delta reported. If no meaningful reduction, the remaining overhead is structural (system prompt + behavioral rules) and cannot be reduced without config changes.
重新运行步骤1的审计脚本,对比优化前后的Token估算值。
完成:已报告优化前后的差值。若未实现显著减少,剩余开销为结构性开销(系统提示词+行为规则),无法通过常规配置更改减少。
Pitfalls
常见陷阱
| Problem | Cause | Fix |
|---|---|---|
| Audit script returns 0 skills | Skills path is wrong or | Check |
| Disabling a toolset breaks a workflow | A skill depends on that toolset | Check |
| Memory pruning removes a needed fact | Aggressive removal without checking last-used | Check recall_count and last_recalled before removing |
| Compression triggers too early | | Raise it for longer context windows, but watch for quality degradation |
| Compression triggers too late | | Lower it — but compression summaries themselves cost tokens |
| Mandatory skill overhead seems unavoidable | It is configured in behavioral rules | Accept the cost, or remove the mandatory load requirement in config |
| 问题 | 原因 | 解决方法 |
|---|---|---|
| 审计脚本返回0个技能 | 技能路径错误或 | 检查 |
| 禁用工具集导致工作流中断 | 某技能依赖该工具集 | 禁用前检查技能前置元数据中的 |
| 记忆清理删除了必要事实 | 未检查最后使用时间就强制删除 | 删除前检查 |
| 压缩触发过早 | | 针对更大的上下文窗口提高阈值,但需注意质量下降风险 |
| 压缩触发过晚 | | 降低阈值——但压缩摘要本身也会消耗Token |
| 强制加载技能开销无法避免 | 在行为规则中配置了强制加载 | 接受该成本,或在配置中移除强制加载要求 |
Verification
验证步骤
- Re-run audit script — confirm token estimates dropped
- — confirm only needed toolsets enabled
hermes tools - Memory block — confirm under budget
- Memory provider counts — confirm working set reduced after consolidation
- Monitor next session: quality should not degrade from reduced context
- 重新运行审计脚本——确认Token估算值下降
- 运行——确认仅启用必要工具集
hermes tools - 检查记忆块——确认在预算内
- 检查记忆提供者数据——确认合并后工作集大小减少
- 监控下一次会话:质量不应因上下文减少而下降