Back to Details

ag2-eval-comparison

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Evaluation — comparing builds (variants & pairwise)

评估 — 对比不同构建版本（变体与两两对比）

When to use

使用场景

  • Rank N models / prompts / configs on a leaderboard → 
    run_variants
  • Decide which of two is better, head-to-head → 
    run_pairwise
    with 
    pairwise_judge
    (LLM) or 
    human_pairwise
    (people)
For running and grading a single agent (scorers, CI, persistence), use 
ag2-evaluation
.
  • 在排行榜上对N个模型/提示词/配置进行排名 → 使用
    run_variants
  • 两两对比，判定两者中哪一个更优 → 搭配
    pairwise_judge
    （LLM自动判定）或
    human_pairwise
    （人工判定）使用
    run_pairwise
如需运行并评估单个Agent（评分器、置信区间、持久化），请使用
ag2-evaluation
。

Install

安装

bash
pip install "ag2[openai,tracing]"
Required. Run this install before delivering the code. If you cannot run commands, state the exact 
pip install
command.
bash
pip install "ag2[openai,tracing]"
必须执行。在运行代码前请先执行此安装命令。若无法运行命令，请明确说明该
pip install
命令。

Leaderboard — run_variants

排行榜 — run_variants

Variants
is a frozen dataclass holding a mapping of named 
Agent
instances plus an 
axis
label naming what you varied. Build each agent with the one thing that differs (config, prompt, tools, middleware, …), hold the rest fixed, score each, rank:
python
from ag2 import Agent
from ag2.config import OpenAIConfig, GeminiConfig
from ag2.eval import Variants, run_variants
from ag2.eval.scorers import agent_judge

board = await run_variants(
    suite,
    variants=Variants(
        {
            "gpt-4o": Agent("a", prompt="Answer helpfully.", config=OpenAIConfig("gpt-4o")),
            "flash":  Agent("a", prompt="Answer helpfully.", config=GeminiConfig("gemini-3-flash-preview")),
        },
        axis="config",                  # label for what was varied (used in summary)
    ),
    scorers=[agent_judge(OpenAIConfig("gpt-4o"), criterion="Helpful and accurate.", key="quality")],
    store_dir="runs",
    repeats=5,                          # optional: N runs per variant for stability
)
print(board.summary("quality"))         # ranked leaderboard
board.best("quality")                   # winning variant name (None if tied)
board.leaderboard("quality")            # list[LeaderboardRow] — variant, score, n, rank
board.results["gpt-4o"]                 # each variant's full RunResult
Vary whatever you like across the agents — set 
axis
to label it (e.g. 
"config"
, 
"prompt"
, 
"tools"
). Tied scores share a rank; a 3-way tie usually means the eval isn't discriminating — make it harder, or score quality with a judge.
Variants
是一个冻结数据类，包含命名**
Agent
实例**的映射，以及一个用于标记变量维度的
axis
标签。为每个Agent构建仅存在一处差异的版本（配置、提示词、工具、中间件……），保持其余部分一致，然后对每个版本评分、排名：
python
from ag2 import Agent
from ag2.config import OpenAIConfig, GeminiConfig
from ag2.eval import Variants, run_variants
from ag2.eval.scorers import agent_judge

board = await run_variants(
    suite,
    variants=Variants(
        {
            "gpt-4o": Agent("a", prompt="Answer helpfully.", config=OpenAIConfig("gpt-4o")),
            "flash":  Agent("a", prompt="Answer helpfully.", config=GeminiConfig("gemini-3-flash-preview")),
        },
        axis="config",                  # 标记变量维度（用于摘要展示）
    ),
    scorers=[agent_judge(OpenAIConfig("gpt-4o"), criterion="Helpful and accurate.", key="quality")],
    store_dir="runs",
    repeats=5,                          # 可选：每个变体运行N次以保证稳定性
)
print(board.summary("quality"))         # 排名后的排行榜
board.best("quality")                   # 获胜变体名称（平局则返回None）
board.leaderboard("quality")            # list[LeaderboardRow] — 变体、分数、次数、排名
board.results["gpt-4o"]                 # 每个变体的完整RunResult
可以在Agent之间任意设置变量差异 — 将
axis
设置为对应的标签（例如
"config"
、
"prompt"
、
"tools"
）。分数相同的变体共享排名；三方平局通常意味着评估缺乏区分度 — 可以提升评估难度，或使用判定器对质量进行评分。

Head-to-head (LLM) — run_pairwise + pairwise_judge

两两对比（LLM自动） — run_pairwise + pairwise_judge

A comparator picks a winner PER task. 
pairwise_judge
shows the pair in BOTH orders and counts a win only if it's consistent — else a tie (cancels position bias):
python
from ag2.eval import run_pairwise
from ag2.eval.scorers import pairwise_judge

result = await run_pairwise(
    suite, variant_a=agent_v1, variant_b=agent_v2,
    comparators=[pairwise_judge(OpenAIConfig("gpt-4o"), criterion="more helpful answer", key="quality")],
    store_dir="runs",
)
wr = result.win_rate("quality")         # B's win-rate
print(wr.rate, wr.ci, wr.wins, wr.losses, wr.ties)   # ties count 0.5; ci is a Wilson 95% interval
print(result.flips("quality"))          # int — count of cases where the two orders disagreed
variant_a
/ 
variant_b
are 
Agent
instances; 
comparators=
is a plural iterable. 
result.agreement("quality", "human")
returns an 
Agreement
(
.rate
, 
.cohen_kappa
, …) between two comparator keys. Use a judge model different from the variants.
比较器会针对每个任务选出获胜者。
pairwise_judge
会以两种顺序展示对比对，只有结果一致时才判定为获胜 — 否则判定为平局（消除位置偏差）：
python
from ag2.eval import run_pairwise
from ag2.eval.scorers import pairwise_judge

result = await run_pairwise(
    suite, variant_a=agent_v1, variant_b=agent_v2,
    comparators=[pairwise_judge(OpenAIConfig("gpt-4o"), criterion="more helpful answer", key="quality")],
    store_dir="runs",
)
wr = result.win_rate("quality")         # 变体B的胜率
print(wr.rate, wr.ci, wr.wins, wr.losses, wr.ties)   # 平局计0.5；ci为Wilson 95%置信区间
print(result.flips("quality"))          # 整数 — 两种顺序下结果不一致的案例数量
variant_a
/ 
variant_b
是**
Agent
实例**；
comparators=
是可迭代对象。
result.agreement("quality", "human")
返回两个比较器键之间的一致性（
.rate
、
.cohen_kappa
等）。请使用与变体不同的判定模型。

Head-to-head (human) — human_pairwise

两两对比（人工） — human_pairwise

Same unit, decided by a person. The pair is blinded and order-randomized; the default prints it and reads 
1
/ 
2
/ 
tie
. Pass your own async 
ask(task, response_1, response_2)
to collect a vote from a UI (returns 
"1"
, 
"2"
, or 
"tie"
):
python
from ag2.eval.scorers import human_pairwise

async def ask(task, response_1, response_2) -> str:
    return await my_ui.compare(task.inputs["input"], response_1, response_2)   # "1" / "2" / "tie"

result = await run_pairwise(suite, variant_a=agent_v1, variant_b=agent_v2,
                            comparators=[human_pairwise(key="quality", ask=ask)], store_dir="runs")
At scale, export a blinded manifest, label it in any tool, import it. 
evaluate_pairwise
is the grade-only twin of 
run_pairwise
(pairs two existing trace sources by 
task_id
):
python
from ag2.eval import evaluate_pairwise, DirectoryTraceSource
from ag2.eval.scorers import export_pairwise_cases, human_labels

a, b = DirectoryTraceSource("runs/champion"), DirectoryTraceSource("runs/challenger")
await export_pairwise_cases(a, b, criteria=["more helpful"], out="labels.jsonl", suite=suite)   # blinded JSONL
单元逻辑相同，但由人工判定结果。对比对会被盲化并随机排序；默认情况下会打印内容并读取
1
/
2
/
tie
输入。可以传入自定义异步函数
ask(task, response_1, response_2)
从UI收集投票结果（返回
"1"
、
"2"
或
"tie"
）：
python
from ag2.eval.scorers import human_pairwise

async def ask(task, response_1, response_2) -> str:
    return await my_ui.compare(task.inputs["input"], response_1, response_2)   # "1" / "2" / "tie"

result = await run_pairwise(suite, variant_a=agent_v1, variant_b=agent_v2,
                            comparators=[human_pairwise(key="quality", ask=ask)], store_dir="runs")
大规模场景下，可导出盲化清单，在任意工具中标记后再导入。
evaluate_pairwise
是
run_pairwise
的仅评分版本（通过
task_id
配对两个现有跟踪源）：
python
from ag2.eval import evaluate_pairwise, DirectoryTraceSource
from ag2.eval.scorers import export_pairwise_cases, human_labels

a, b = DirectoryTraceSource("runs/champion"), DirectoryTraceSource("runs/challenger")
await export_pairwise_cases(a, b, criteria=["more helpful"], out="labels.jsonl", suite=suite)   # 盲化JSONL文件

a person adds "preferred": "1" | "2" | "tie" per line, then:

人工为每一行添加 "preferred": "1" | "2" | "tie"，然后执行：

result = await evaluate_pairwise(a, b, suite=suite, store_dir="runs", comparators=[human_labels("labels.jsonl", criterion="more helpful", key="helpful")])

The manifest hides which model is which; its `first_variant` field de-blinds it for `human_labels`.
result = await evaluate_pairwise(a, b, suite=suite, store_dir="runs", comparators=[human_labels("labels.jsonl", criterion="more helpful", key="helpful")])

清单会隐藏模型信息；其`first_variant`字段会在`human_labels`中用于解除盲化。

Common pitfalls

常见误区

  • Judge == a variant's model — self-preference bias; use a different judge model.
  • Bare win-rate on few pairs — report 
    wr.ci
    (Wilson); a small n straddles 50%.
  • Passing factories, not agents — 
    run_variants
    (
    variants=Variants({name: Agent(...)})
    ) and 
    run_pairwise
    (
    variant_a=
    /
    variant_b=
    ) take 
    Agent
    instances    , not build callables. Vary the model per task with 
    model_config=
    (a 
    dict[task_id, ModelConfig]
    ) rather than rebuilding the agent. Keep 
    pairwise_judge
    's default swap (don't set 
    swap=False
    ) for unbiased verdicts.
  • 判定模型与变体模型相同 — 存在自我偏好偏差；请使用不同的判定模型。
  • 少量对比对仅使用原始胜率 — 请报告
    wr.ci
    （Wilson置信区间）；样本量小时结果可能接近50%。
  • 传入工厂函数而非Agent实例 — 
    run_variants
    （
    variants=Variants({name: Agent(...)})
    ）和
    run_pairwise
    （
    variant_a=
    /
    variant_b=
    ）接收的是**
    Agent
    实例**，而非构建可调用对象。如需为每个任务设置不同模型，请使用
    model_config=
    （一个
    dict[task_id, ModelConfig]
    ），而非重新构建Agent。请保留
    pairwise_judge
    的默认交换设置（不要设置
    swap=False
    ）以获得无偏差结果。

Going deeper

深入了解

  • website/docs/user-guide/evaluation/
    — 
    variants
    (the 
    Variants
    mapping + 
    axis
    ), 
    pairwise
    (comparators, win-rate, blinded labeling)
  • ag2-evaluation
    — single-agent run/grade, scorers, CI, persistence
  • website/docs/user-guide/evaluation/
    — 
    variants
    （
    Variants
    映射 + 
    axis
    ）、
    pairwise
    （比较器、胜率、盲化标记）
  • ag2-evaluation
    — 单Agent运行/评分、评分器、置信区间、持久化