• "Compare these models on..."
• "Which model is better for [task]?"
• "A/B test [model1] vs [model2]"
• "Blind comparison of..."
• "Test how different models handle this prompt"
• User wants to evaluate models before committing to one for a workflow
• Prompt engineering — seeing how different models interpret instructions

• Benchmarking (MMLU, GSM8K, etc.) → use

  evaluating-llms-harness

  skill
• Cost analysis → just check provider pricing pages
• Single model test → just switch model and ask directly
• Multi-source research synthesis → use

  deep-research

  skill (iterative research loop, not model comparison)

references/provider-tool-support.md

references/model-behavior.md

tests/

pytest

requirements-dev.txt

Tool-mode environment dependency:

--mode tools

runs real

web_search

. That needs either

SEARXNG_URL

set to a SearXNG instance or the

ddgs

CLI available on

PATH

(the fallback). Neither is a Python import dependency, but one of them must be present for live search; without both,

web_search

returns an error result. The other three modes (

simple

,

coding

,

review

) need neither.

undefined

--judge

provider:@effort:model_id

ollama-cloud:@medium:glm-5.3-flash

ollama-cloud@medium

max_tokens

reasoning_effort

provider@effort

~/.hermes/data/glm53flash_effort_ab/ab_results.jsonl

--max-turns

CLI override (added 2026-07-17): Pass

--max-turns N

to override the test bank's

max_turns

field at runtime without editing source. If not passed, the test bank default is used. This was added because killing a background run to patch

TEST_BANK["A"]["max_turns"]

in source code is fragile and disrupts parallel execution.

PROVIDERS

scripts/compare.py

OLLAMA_API_KEY

https://ollama.com/v1/chat/completions

NVIDIA_API_KEY

https://integrate.api.nvidia.com/v1/chat/completions

OPENROUTER_API_KEY

https://openrouter.ai/api/v1/chat/completions

Model inventories live elsewhere, on purpose. Exact model counts and IDs drift constantly, so they are deliberately kept out of this doc. For the live list run

python3 scripts/compare.py --list-models <provider>

(needs that provider's key);

references/providers.json

holds a curated, count-free snapshot with representative model IDs per provider.

1. Ollama Cloud — free (GLM, Qwen, Kimi, Gemini, Gemma)
2. NVIDIA — free (Yi, Llama, Nemotron)
3. OpenRouter — paid (per-token cost). Only use when:
   • The user explicitly requests an OpenRouter-only model (e.g. Claude, GPT-4o)
   • The user explicitly says to use OpenRouter
   • The free providers don't have a suitable model for the task

"This comparison will use OpenRouter which has per-token costs. Estimated cost: ~$0.01–0.05 per model per call (varies by model). Proceed?"

/v1/chat/completions

{
  "model": "<model_id>",
  "messages": [{"role": "user", "content": "<prompt>"}],
  "max_tokens": 8192,
  "temperature": 0.7
}

Authorization: Bearer <API_KEY>

max_tokens is set to 8192 in both

call_model_simple

and

call_model_with_tools

. The previous default of 4096 caused truncated code output on coding test J (LRU cache) — the model generated so much code it hit the 4096 limit before finishing the decorator section. If a model still truncates, increase the value in the payload dict.

• Explicit model names: "compare claude-sonnet-4.6 vs gpt-4o vs gemini-2.5-flash"
• Provider + model: "compare OpenRouter claude-sonnet-4.6 vs Ollama glm-5.2"
• Task-based: "which model is best for coding?" → suggest 2-4 candidates
• All from a provider: "test 3 OpenRouter models" → pick diverse ones

provider:model_id

• Free, leave the prefix off or name an Ollama/NVIDIA model → resolve on a free provider, e.g.

  ollama-cloud:glm-5.2

  ,

  ollama-cloud:qwen3-coder:480b

  ,

  nvidia:meta/llama-3.3-70b-instruct

  .
• A proprietary model (Claude, GPT, Gemini-Pro, DeepSeek-R1, …) → only OpenRouter carries it, e.g.

  openrouter:anthropic/claude-sonnet-4.6

  ,

  openrouter:openai/gpt-4o

  — paid, confirm first (see routing rule above).
• Unsure of the exact ID? Run

  python3 scripts/compare.py --list-models <provider>

  for the live list, or see

  references/providers.json

  for a curated set of representative IDs per provider. Don't hand-maintain a model table here.

1. Check if the task type maps to free models (e.g. coding → qwen3-coder, glm-5.2; general → glm-5.2, kimi-k2.5)
2. Suggest 2-4 free models from Ollama Cloud and NVIDIA
3. Only suggest OpenRouter models if the user asks for premium models (Claude, GPT-4o) or the free providers lack suitable options
4. If suggesting any paid models, flag the cost before running

curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data'] if 'KEYWORD' in m['id'].lower()]"

• 2-4 models per comparison (more = unwieldy in chat)
• If user doesn't specify, suggest a diverse set (different providers/sizes)
• Always confirm the model list with the user before running

Turn 1: Send prompt + tool definitions → Model returns tool_call(s)
Turn 2: Execute real tool → inject result → Model returns tool_call(s) or answer
Turn 3: ... until final answer or max turns (10 by default, configurable per-test via `max_turns` field in TEST_BANK)

web_search

web_extract

• web_search

  → SearXNG (self-hosted, via

  SEARXNG_URL

  ) with DDGS fallback

• web_extract

  → direct HTTP fetch with HTML-to-text conversion (3000 chars/page)

tools

• web_search(query, limit)

  — returns titles, URLs, snippets

• web_extract(urls)

  — returns page content as text

• Empty/error: if a model returns an error or empty response, note it and exclude from the comparison. Tell the user which model failed.
• Truncated: if response hit max_tokens, note it was truncated
• Refusal: if a model refused to answer, include it as-is (refusals are valid comparison data)

🧪 Blind Model Comparison
Prompt: "[truncated to 100 chars...]"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model A:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model B:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model C:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Vote: Reply with the letter of the best response (A, B, or C), or "tie".

• Full responses, not summaries — the user needs to judge quality
• Clear visual separation between responses
• Truncate at ~3000 chars per response for chat readability (note if truncated)
• If responses are very long (>3000 chars), save full versions to files and present truncated versions in chat with a note

• Letter: "A", "B", "C", "D"
• "tie" or "tie between A and B"
• "all bad" (valid — none won)
• Specific feedback: "A is better but B's code is cleaner"

• Section X: primarily from <model_name>
• Section Y: primarily from <model_name>

undefined

--mode tools

1. Script defines 5 tools as OpenAI function tools:

   web_search

   ,

   web_extract

   ,

   run_python

   ,

   read_file

   ,

   write_file

2. Sends prompt + tool defs to each model in parallel
3. When a model returns

   tool_calls

   , the script executes the real call:

   • web_search

     → SearXNG (self-hosted, via

     SEARXNG_URL

     ) with DDGS fallback

   • web_extract

     → direct HTTP fetch, HTML stripped to text, truncated to 3000 chars/page

   • run_python

     → executes Python code in a per-model sandbox (temp dir, 10s timeout, no network)

   • read_file

     → reads a file from the sandbox directory (path traversal blocked)

   • write_file

     → writes a file to the sandbox directory (path traversal blocked)
4. Feeds the real result back as a

   tool

   role message
5. Repeats until final answer or max turns (10 default, configurable per-test via

   max_turns

   in TEST_BANK)
6. Tracks all turns, tool calls, tokens per turn, convergence status

tempfile.mkdtemp

../

finally

web_search

web_extract

run_python

read_file

write_file

references/provider-tool-support.md

"I need a model for summarizing legal documents. Compare claude-sonnet-4.6, gpt-4o, and gemini-2.5-flash on this prompt: [legal text]"

"Compare glm-5.2 (free) vs claude-sonnet-4.6 (paid) on this coding prompt. Is the paid one worth it?"

"Test this prompt on 3 models — I want to see which ones follow my formatting instructions"

"Compare the same model (llama-4-maverick) on OpenRouter vs NVIDIA — is there a difference?"

• 2 consecutive failures → provider marked dead
• Cooldown: 15s (Ollama Cloud), 30s (NVIDIA), 20s (OpenRouter)
• Any success → resets failure counter immediately
• Cooldown expiry → provider gets another chance
• State persists to

  ~/.hermes/data/provider_health.json

  across runs

• Running comparisons across multiple providers — one flaky provider won't slow down the whole comparison
• Tool calling mode (10 turns per model) — a dead provider is skipped in seconds instead of waiting 60s × 10 turns = 10 minutes of timeouts
• Back-to-back comparisons — if a provider went down in a previous run, the next run knows to skip it

• Cost savings — user says "use codex cli for gpt5.6" to avoid OpenRouter per-token costs. Codex ($20/mo) and Cursor Pro+ ($60/mo) include model access without per-token charges.
• Model availability — some models (e.g.

  cursor-grok-4.5-high

  ) are only available through Cursor CLI, not on any API provider.

1. API models → run through

   scripts/compare.py

   as normal (background)
2. CLI models → run each separately via terminal in parallel:\n - Codex CLI (GPT-5.6):

   codex exec --skip-git-repo-check -m gpt-5.5 "<prompt>" > /tmp/codex_result.json 2>&1

   \n - Cursor CLI (Grok 4.5):

   export PATH="$HOME/.local/bin:$PATH" && export $(grep 'CURSOR_API_KEY=' ~/.hermes/.env | xargs) && agent -p --force --model "cursor-grok-4.5-high" --sandbox disabled "<prompt>" > /tmp/cursor_result.json 2>&1

3. Merge results — collect API script output + CLI outputs, present all responses anonymously (shuffle together), let user vote, then reveal.

codex

-m gpt-5.6-sol

agent

--model "cursor-grok-4.5-high"

CURSOR_API_KEY

Always run CLI models in background (

background=true

+

notify_on_complete=true

) — they can take 1-5 minutes. Run them in the same tool batch as the API script launch for true parallelism.

Cursor Grok 4.5 High is the heaviest internal tier. For lighter tasks use

cursor-grok-4.5-medium

. See the

cursor-cli

skill for full model selection guidance.

• Revealing identities too early — never let model names leak into the anonymous presentation. Double-check before sending.
• Always same order — shuffle every time. If model A is always first, the user may develop position bias.
• Not parallel — send all API calls in the same tool block. Sequential calls waste time.
• Ignoring failures — if a model errors, tell the user. Don't silently exclude it.
• Truncating too aggressively — the user needs enough text to judge quality. 3000 chars minimum, full if shorter.
• Forgetting to reveal — always reveal after the vote. The whole point is knowing which model won.
• Not confirming model list — always confirm which models before spending API calls. Models cost money.
• Spending paid API calls without confirmation — OpenRouter is per-token paid. Always confirm with the user before calling OpenRouter models. Default to free providers (Ollama Cloud, NVIDIA) when no provider is specified. Only escalate to OpenRouter when the user explicitly requests a premium model or confirms cost.
• Using max_tokens too low — 8192 is the current default in compare.py. The previous 4096 caused truncated code on test J (LRU cache decorator cut off mid-function). If a model still truncates, raise the value in the payload dict.
• Temperature mismatch — use the same temperature (0.7 default) for all models. Different temperatures = unfair comparison.
• API key leakage — never print API keys in output. Always use env var references in curl commands.
• Rate limits — OpenRouter and NVIDIA have rate limits. If comparing 4+ models, add a small delay between calls if needed. OpenRouter free-tier models (with

  :free

  suffix) are especially prone to HTTP 429 rate-limiting after just 1-2 calls.
• OpenRouter free-tier models (

  :free

  suffix) and the cost gate — the script conservatively treats ALL OpenRouter models as paid, even

  :free

  suffix models that cost $0. Set

  COMPARE_CONFIRM_PAID=1

  env var to bypass the cost confirmation gate. Example:

  COMPARE_CONFIRM_PAID=1 python3 scripts/compare.py --models "openrouter:tencent/hy3:free" ...

  . No additional cost confirmation needed since the user has already chosen free-tier models.
• NVIDIA NIM model IDs include the

  nvidia/

  prefix — when using NIM via compare.py, pass the full model ID including the org prefix:

  nvidia:nvidia/nemotron-3.5-lightning-30b-a3b

  (provider colon, then full model ID). Passing

  nvidia:nemotron-3.5-lightning-30b-a3b

  (without the

  nvidia/

  prefix) returns HTTP 404 "page not found" — NIM requires the full

  nvidia/...

  model ID in the API payload.
• Nemotron 3.5 Lightning free tier tool calling — only the OpenRouter

  :free

  suffix and NIM direct support

  tools

  . The OpenRouter paid tier (

  nvidia/nemotron-3.5-lightning

  without

  :free

  , CoreWeave-hosted) returns HTTP 404 on

  tools

  . Use

  nvidia:nvidia/nemotron-3.5-lightning-30b-a3b

  (NIM, free, stable) or

  openrouter:nvidia/nemotron-3.5-lightning:free

  (free, but prone to 502s).
• NVIDIA model list is stale — the

  /v1/models

  endpoint lists many models but most return 404/410 when called. Only

  meta/llama-3.1-70b-instruct

  ,

  meta/llama-3.3-70b-instruct

  , and

  mistralai/mixtral-8x7b-instruct-v0.1

  were alive as of June 2026. Always verify a model is alive with a simple chat completion before using it in a comparison. See

  references/provider-tool-support.md

  for the current alive list.
• Assuming tool calling support — not all models support the

  tools

  API parameter. Test with a simple tool-call request first. Models that mention "search" in text but don't emit

  tool_calls

  do NOT support tool calling. See

  references/provider-tool-support.md

  for the tested matrix.
• Not all models support tool calling —

  deepseek-v3.2

  (Ollama Cloud) mentions "search" in text but does NOT emit

  tool_calls

  .

  mistralai/mixtral-8x7b-instruct-v0.1

  (NVIDIA) returns HTTP 400 on the

  tools

  parameter. Test tool support per model before running

  --mode tools

  . See

  references/provider-tool-support.md

  .
• Efficiency without accuracy is waste — a model that uses few tokens but gets the wrong answer is not efficient. Always cross-reference efficiency stats with the judge score. gemma4:31b used 949 tokens and 2 turns but said Python 3.13 (wrong); kimi-k2.5 used 5434 tokens and 5 turns but got the correct answer with authoritative sources.
• Tool calling loop can hit max turns — some models keep searching without synthesizing. The 10-turn default cap (configurable per-test via

  max_turns

  in TEST_BANK) prevents infinite loops, but the result is marked "did not converge." Check the

  converged

  field in the trace. Increasing

  max_turns

  can help models that need more turns to synthesize, but watch for models that loop on the same URLs without converging — more turns won't fix a model that can't synthesize.
• No

  --max-turns

  CLI override (fixed 2026-07-17) — the script used to only support per-test

  max_turns

  in the TEST_BANK dict, requiring source edits to change the turn cap. This forced killing background runs to patch source. Now

  --max-turns N

  overrides at runtime. If the flag doesn't work, check that

  compare.py

  has the

  --max-turns

  argument parser entry.
• User prefers CLI models to cut costs — when the user says "use codex cli for gpt5.6" or "use cursor-grok-4.5-high", they want to avoid OpenRouter per-token charges. Route GPT-5.6 through Codex CLI and Grok 4.5 through Cursor CLI instead of OpenRouter. See the Hybrid Mode section above. This is a cost preference, not just a one-off request.\n- Codex CLI model version limits —

  codex exec -m gpt-5.6-sol

  fails with "requires a newer version of Codex" on v0.135. Test with a trivial prompt first (

  codex exec --skip-git-repo-check -m <model> "say hello"

  ). Fall back to

  gpt-5.5

  if the newer model is unsupported.\n- Codex CLI needs

  --skip-git-repo-check

  — without it, Codex refuses to run outside a git repo. Always include this flag.\n- Cursor CLI

  --force

  required for web access —

  --trust --sandbox disabled

  alone does NOT enable web search in headless mode. Web requests are still blocked. Use

  --force

  (which implies

  --trust

  ) to enable web access. Confirmed across 3 attempts:

  --trust

  → blocked,

  --trust --sandbox disabled

  → blocked,

  --force

  → works.\n- OpenRouter geofenced models — some models (e.g.

  meta/muse-spark-1.1

  ) return HTTP 403 "This model is only available in the United States" when called from outside the US. No workaround without a US proxy. Exclude from comparisons run from non-US locations.
• SearXNG default engines may return 0 results — Google, DuckDuckGo, Brave, Startpage, and Mojeek engines frequently fail on self-hosted SearXNG (rate limits, CAPTCHAs). Only Bing and Yandex reliably return results. The script's

  execute_web_search

  now passes

  &engines=bing,yandex,duckduckgo,google

  explicitly. If search returns 43-char

  {"results": [], "note": "No results found"}

  on every query, check SearXNG engine health with

  curl -s "http://$SEARXNG_URL/search?q=test&format=json&engines=bing"

  and update the engines list in

  scripts/compare.py

  .
• DDGS CLI syntax changed — the old

  ddgs --json -q ... -m ...

  syntax no longer works. The new CLI uses

  ddgs text -q ... -m ... -o <file>

  (outputs to file, not stdout). The script's DDGS fallback has been updated to use the new syntax. Verify with

  ddgs text --help

  if fallback fails.
• SEARXNG_URL not exported to background processes — the env var is loaded from

  ~/.hermes/.env

  by the script's

  load_env()

  function, but if you pass

  --mode tools

  to a background process (e.g. via

  terminal(background=true)

  ), make sure to

  export SEARXNG_URL=http://...

  explicitly in the command. The

  load_env()

  function only sets vars that are NOT already in

  os.environ

  , but background shells start with a minimal environment.
• compare.py requires 2+ models — the script exits with "Error: need at least 2 models to compare" if you pass only one. To retry a single model after rate limiting, use a direct curl API call instead of the script.
• Comparing reasoning-effort levels is now native (v1.2.0) — use the

  provider:@effort:model_id

  syntax on

  --models

  and

  --judge

  (see the "Reasoning-effort A/B" section above). The old wrapper-harness pattern (

  /root/.hermes/cache/ab_effort_run.py

  , virtual providers registered in a loader script) still works for multi-test batteries with per-test ground-truth judging; the native syntax covers single-run comparisons. For CLI judges (e.g. Codex gpt-5.6-sol): write the rubric + anonymized responses to a temp file, run

  codex exec --skip-git-repo-check -m gpt-5.6-sol "Read the file <path> and follow the instructions inside"

  , parse JSON from stdout+stderr, shuffle the Response 1/2 order per test. Ground truth for code_review tests comes from TEST_BANK

  evaluation

  +

  planted_issues

  keys (NOT

  evaluation_criteria

  ). Known findings (glm-5.3-flash, 2026-08-29): effort→reasoning-token count is NON-monotonic (high ≠ more thinking); high effort hit a 12-turn non-convergence on long tool chains while medium converged; medium won 4-2-2 on quality (8.6 vs 7.8 avg), high was ~3x faster median.
• Kimi K3 rate limiting on OpenRouter —

  moonshotai/kimi-k3

  hits HTTP 429 on the shared OpenRouter pool, especially when running multiple tests in parallel. Wait 15-30s and retry, or add a Moonshot API key to OpenRouter (BYOK) for dedicated rate limits.
• Sandbox tests (S1-S5) need models that support 5 tools — the sandbox tests pass all 5 tool definitions (web_search, web_extract, run_python, read_file, write_file) to the model. Some models may not support tool calling with 5 tools, or may only support 1-2 tools at a time. If a model fails on sandbox tests but passes research tests (A/B/C), it may be overwhelmed by the number of tool definitions. Test with a simpler sandbox prompt first.
• Sandbox Python execution is not network-isolated — the

  run_python

  tool runs

  subprocess.run([sys.executable, script])

  which can make network calls if the code imports

  urllib

  or

  socket

  . The 10-second timeout limits damage but is not a true sandbox. For production hardening, consider running in a container or seccomp filter. For model comparison purposes, the timeout is sufficient.
• Sandbox temp files are cleaned up after each model — the

  run_tool_loop

  function creates a temp dir per model and cleans it up in a

  finally

  block. If a model writes a file and then needs to read it back in a later turn, that works (same sandbox dir persists across turns). But if the script crashes mid-run, the

  finally

  block still cleans up.
• Sandbox output truncated to 2000 chars —

  run_python

  returns stdout and stderr truncated to 2000 chars each. If a model generates a lot of output (e.g. printing a large list), the truncated output may confuse it. Models that format output concisely will perform better on sandbox tests.