logfire-evals

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Evaluate with pydantic_evals and Logfire

使用pydantic_evals和Logfire进行评估

How This Works

工作原理

pydantic_evals

runs your actual function or agent against a

Dataset

Case

s (input + expected output + metadata), scores each with one or more

Evaluator

s, and produces a report. It depends on

logfire

itself (the

datasets

extra pulls in the real SDK, not a mock), so whether

logfire.configure()

has run determines only whether results also upload to Logfire's Datasets & Experiments UI — omitting it keeps results entirely local and printed to the terminal, silently, not an error.

Agentic evaluators (tool-call correctness, trajectory matching) need more than that: they read the task's own execution span tree, so without a working

logfire.configure()

they don't just fail to upload — every case reports "No span tree available" and the check never ran at all.

pydantic_evals

会让你的实际函数或agent运行在包含

Case

（输入+预期输出+元数据）的

Dataset

上，通过一个或多个

Evaluator

为每个用例评分，并生成报告。它依赖

logfire

本身（

datasets

扩展会引入真实SDK而非模拟版本），因此

logfire.configure()

是否执行仅决定结果是否会上传到Logfire的Datasets & Experiments UI——如果省略该调用，结果将完全保存在本地并打印到终端，不会报错，仅静默处理。

智能体评估器（工具调用正确性、轨迹匹配）需要更多条件：它们会读取任务自身的执行跨度树，因此如果没有有效的

logfire.configure()

，它们不仅无法上传结果——每个用例都会显示“No span tree available”，且检查完全不会执行。

Step 1: Check for an Existing Braintrust Suite First

步骤1：先检查是否存在现有Braintrust套件

Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual

Eval(...)

calls or

from braintrust import Eval

in source, not just a

braintrust

dependency listed without any real usage? This path needs no CLI auth at all — don't run Step 2 for it.

Keep the existing

Eval()

code (Python

braintrust>=0.30.1

/ TypeScript

braintrust>=3.24.0

— verified versions) and redirect its next run to Logfire by changing environment variables only, no

pydantic_evals

involved:

bash

export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"  # EU: logfire-eu.pydantic.dev
export BRAINTRUST_API_KEY="<logfire-project-write-token>"                  # Project -> Settings -> Write tokens
unset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 "it still hit Braintrust" cause

This is a compatibility preview, not full parity: covers inline/callable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It does not cover Braintrust-hosted datasets/prompts/functions, BTQL, the model proxy, server-side scoring, or post-finalization feedback — and

summarize_scores=False

, a manual

flush()

without a comparison, or the Rust SDK never request the summary this endpoint needs, so nothing lands even though the run appears to succeed. Full detail and the concept-translation table (Braintrust "project" → Logfire dataset name, "scorer" → evaluator, etc.): https://pydantic.dev/docs/logfire/get-started/comparisons/migrate-from-braintrust/.

Skip straight to Step 5 (Verify) — the SDK's own printed result URL also opens directly in Logfire, and nothing else here (auth, dataset definition) applies to this path.

No existing Braintrust suite? Continue to Step 2 now, before the more detailed identification in Step 3 — nothing past this point requires knowing the function/agent or dataset shape yet.

先做个简单检查：当前仓库是否已有现成的Braintrust套件——源码中是否有实际的

Eval(...)

调用或

from braintrust import Eval

语句，而不只是列出

braintrust

依赖但未实际使用？这种情况完全不需要CLI认证——无需执行步骤2。

保留现有的

Eval()

代码（Python版本

braintrust>=0.30.1

/ TypeScript版本

braintrust>=3.24.0

——已验证版本），仅通过修改环境变量将下一次运行重定向到Logfire，无需涉及

pydantic_evals

：

bash

export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"  # 欧盟区域：logfire-eu.pydantic.dev
export BRAINTRUST_API_KEY="<logfire-project-write-token>"                  # 项目 → 设置 → 写入令牌
unset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # 如果设置了这些变量会覆盖上面的端点——这是“仍调用Braintrust”的头号原因

这是兼容性预览，并非完全对等：支持内联/可调用数据、本地任务和评分器、多评分项、每个名称对应一个标签，以及正常的摘要最终化。不支持Braintrust托管的数据集/提示词/函数、BTQL、模型代理、服务端评分或最终化后的反馈——且

summarize_scores=False

、无对比的手动

flush()

或Rust SDK不会请求此端点所需的摘要，因此即使运行看似成功，也不会有任何内容上传。详细信息和概念转换表（Braintrust“项目”→Logfire数据集名称，“scorer”→评估器等）：https://pydantic.dev/docs/logfire/get-started/comparisons/migrate-from-braintrust/.

直接跳至步骤5（验证）——SDK自身打印的结果URL可直接在Logfire中打开，此处的其他内容（认证、数据集定义）均不适用于此路径。

没有现有Braintrust套件？现在继续步骤2，无需等待步骤3中的详细识别——此步骤之后的内容无需提前了解函数/agent或数据集的结构。

Step 2: Authenticate When the Run Needs Logfire

步骤2：当运行需要Logfire时进行认证

Skip authentication and continue to Step 3 only when the user explicitly wants a local-only

pydantic_evals

run using evaluators that do not need span data; omit

logfire.configure()

in Step 4 so results stay in the terminal. Uploading results, using a hosted dataset, or running a span-based evaluator such as

ToolCorrectness

requires Logfire, so authenticate before opening or running evaluation files and target the exact project first.

For a Logfire-backed run, check first —

uvx logfire --non-interactive whoami

(JS:

npx logfire whoami

) — and skip to Step 3 if it already reports the right project and region. Otherwise, follow the full command sequence, flags, and gotchas (the

--non-interactive

requirement, why

auth

won't open a browser for you, the

LOGFIRE_TOKEN

-vs-credentials-file conflict, and token-file safety) in Authenticate and Select the Exact Project. This CLI flow is for

logfire.configure()

; Step 3's hosted-dataset operations use a separate API key with different scopes.

仅当用户明确希望使用不需要跨度数据的评估器进行本地

pydantic_evals

运行时，才跳过认证并继续步骤3；在步骤4中省略

logfire.configure()

，使结果仅保留在终端中。上传结果、使用托管数据集或运行基于跨度的评估器（如

ToolCorrectness

）需要Logfire，因此在打开或运行评估文件并定位到具体项目之前，先进行认证。

对于基于Logfire的运行，先检查——

uvx logfire --non-interactive whoami

（JS环境：

npx logfire whoami

）——如果已显示正确的项目和区域，则跳至步骤3。否则，请遵循认证并选择具体项目中的完整命令序列、标志和注意事项（

--non-interactive

要求、

auth

无法打开浏览器的原因、

LOGFIRE_TOKEN

与凭据文件的冲突、令牌文件的安全性）。此CLI流程适用于

logfire.configure()

；步骤3中的托管数据集操作使用具有不同权限范围的独立API密钥。

Step 3: Detect What to Evaluate

步骤3：确定评估对象

Identify the function or agent under test (a PydanticAI agent, an LLM-calling function, any callable that takes an input and returns an output) and whether a dataset already exists:

In-code dataset: a Python module defining
```
Case
```
/
```
Dataset
```
directly — the default for an agent-driven workflow.
Hosted/managed dataset: cases live in the Logfire UI, edited by non-engineers, pulled/pushed via a separate
```
LogfireAPIClient
```
(
```
from logfire.experimental.api_client import LogfireAPIClient
```
).
```
client.get_dataset(name)
```
with no type arguments returns a raw dict, not something
```
push_dataset
```
or
```
.evaluate_sync()
```
can take — pass the input/output (and metadata, if used) types to get back a real
```
pydantic_evals.Dataset
```
:
```
client.get_dataset(name, MyInputType, MyOutputType)
```
. If the stored dataset contains custom evaluators, also pass their classes with
```
custom_evaluator_types=[ExactMatch]
```
(and custom report evaluators with
```
custom_report_evaluator_types=[...]
```
) so they can be deserialized. Push with
```
client.push_dataset(dataset)
```
. This needs its own API key from Settings → API Keys (scoped
```
project:read_datasets
```
/
```
project:write_datasets
```
), not Step 2's CLI auth flow. Only relevant if the user specifically wants case editing outside code.

识别被测函数或agent（PydanticAI agent、调用LLM的函数、任何接收输入并返回输出的可调用对象），以及数据集是否已存在：

代码内数据集：Python模块中直接定义
```
Case
```
/
```
Dataset
```
——智能体驱动工作流的默认方式。
托管数据集：用例存储在Logfire UI中，可由非工程师编辑，通过独立的
```
LogfireAPIClient
```
（
```
from logfire.experimental.api_client import LogfireAPIClient
```
）拉取/推送。不带类型参数的
```
client.get_dataset(name)
```
返回原始字典，无法用于
```
push_dataset
```
或
```
.evaluate_sync()
```
——传入输入/输出（以及元数据，如果使用）类型，以获取真实的
```
pydantic_evals.Dataset
```
：
```
client.get_dataset(name, MyInputType, MyOutputType)
```
。如果存储的数据集包含自定义评估器，还需传入它们的类，如
```
custom_evaluator_types=[ExactMatch]
```
（自定义报告评估器则使用
```
custom_report_evaluator_types=[...]
```
），以便反序列化。使用
```
client.push_dataset(dataset)
```
推送。此操作需要设置→API密钥中的独立API密钥（权限范围为
```
project:read_datasets
```
/
```
project:write_datasets
```
），而非步骤2中的CLI认证流程。仅当用户明确希望在代码外编辑用例时才相关。

Step 4: Define the Dataset and Run It

步骤4：定义数据集并运行

bash

uv add 'logfire[datasets]'

python

from dataclasses import dataclass

import logfire
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance

logfire.configure()  # omit this and results stay local only, silently


@dataclass
class ExactMatch(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
        return ctx.output == ctx.expected_output


def classify_sentiment(text: str) -> str:
    ...  # the function under test


dataset = Dataset[str, str, None](
    name='sentiment-eval',
    cases=[
        Case(name='positive', inputs='I love this', expected_output='positive'),
        Case(name='negative', inputs='This is terrible', expected_output='negative'),
    ],
    evaluators=[ExactMatch(), IsInstance(type_name='str')],
)

report = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`
report.print(include_input=True, include_output=True)

Before running the full dataset, run a smoke test on 2-3 cases if the dataset is large or uses

LLMJudge

/any evaluator that makes a real, billed model call — a bug caught on 3 cases costs 3 model calls, the same bug caught on 300 costs 300:

python

smoke = Dataset(
    name=dataset.name,
    cases=dataset.cases[:3],
    evaluators=dataset.evaluators,
    report_evaluators=dataset.report_evaluators,
)
smoke_report = smoke.evaluate_sync(classify_sentiment)
smoke_report.print(include_input=True, include_output=True)

Confirm the smoke run has zero unexpected errors and the assertions that should pass do. Then, if the full dataset is large or uses paid model calls, tell the user the case count and which evaluators will make model calls, and get explicit confirmation before running the full dataset — don't run an expensive full pass on the strength of a clean smoke test alone without saying so.

Custom evaluators must be
@dataclass
subclasses — a plain class raises at run time. Case names must be unique within a dataset. The evaluators reached for most:

Evaluator	Checks
`Equals(value)` / `EqualsExpected()`	Exact match against a literal / `expected_output` (no-op if `expected_output` is unset — don't rely on it silently catching that)
`IsInstance(type_name)`	Output's type matches by name
`LLMJudge(rubric, model=None, score=False)`	LLM-as-judge scoring; costs a real model call per case per judge — prefer boolean/categorical rubrics over 1-10 scales (judges are unstable on continuous scores), and benchmark the judge against ~20-100 hand-labeled cases before trusting it
`ToolCorrectness(expected_tools, ...)`	Which tools an agent called — reads the span tree, so needs Step 2's `logfire.configure()` to work at all, not just to upload

Also available:

Contains

MaxDuration

TrajectoryMatch

ArgumentCorrectness

MaxToolCalls

MaxModelRequests

— same span-tree dependency as

ToolCorrectness

for the tool/trajectory ones; see

pydantic_evals.evaluators

for the full set. These five agentic (span-based) evaluators need

pydantic-evals>=2.4.0

— on an older pin, check

pyproject.toml

uv.lock

and upgrade before reaching for them, since the import itself is what fails, not a silent no-op.

The

Python

evaluator (arbitrary code execution) was removed for security reasons — don't reach for it even if an older example references it.

If editing a hosted dataset:

client.push_dataset(dataset)

overwrites server-side evaluators on every push, including removing ones you deleted locally — don't push a stale local copy over a dataset others have edited in the UI.

bash

uv add 'logfire[datasets]'

python

from dataclasses import dataclass

import logfire
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance

logfire.configure()  # 省略此调用，结果将仅保留在本地，静默处理


@dataclass
class ExactMatch(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
        return ctx.output == ctx.expected_output


def classify_sentiment(text: str) -> str:
    ...  # 被测函数


dataset = Dataset[str, str, None](
    name='sentiment-eval',
    cases=[
        Case(name='positive', inputs='I love this', expected_output='positive'),
        Case(name='negative', inputs='This is terrible', expected_output='negative'),
    ],
    evaluators=[ExactMatch(), IsInstance(type_name='str')],
)

report = dataset.evaluate_sync(classify_sentiment)  # 或 `await dataset.evaluate(...)`
report.print(include_input=True, include_output=True)

如果数据集较大或使用
LLMJudge
/任何会发起真实计费模型调用的评估器，请先在2-3个用例上运行冒烟测试——在3个用例上发现的bug仅需3次模型调用，而在300个用例上发现则需300次：

python

smoke = Dataset(
    name=dataset.name,
    cases=dataset.cases[:3],
    evaluators=dataset.evaluators,
    report_evaluators=dataset.report_evaluators,
)
smoke_report = smoke.evaluate_sync(classify_sentiment)
smoke_report.print(include_input=True, include_output=True)

确认冒烟测试没有意外错误，且应通过的断言均已通过。然后，如果完整数据集较大或使用付费模型调用，请告知用户用例数量以及哪些评估器会发起模型调用，并在运行完整数据集前获得明确确认——不要仅根据冒烟测试通过就直接运行昂贵的完整测试。

自定义评估器必须是
@dataclass
子类——普通类会在运行时抛出错误。用例名称在数据集内必须唯一。最常用的评估器：

Evaluator	检查内容
`Equals(value)` / `EqualsExpected()`	与字面量/ `expected_output` 完全匹配（如果未设置 `expected_output` 则无操作——不要依赖它静默处理这种情况）
`IsInstance(type_name)`	输出类型与名称匹配
`LLMJudge(rubric, model=None, score=False)`	以LLM作为评判器进行评分；每个用例每个评判器都会发起真实模型调用——优先选择布尔/分类规则而非1-10分制（评判器对连续分数的稳定性较差），并在信任评判器之前，针对约20-100个人工标记的用例进行基准测试
`ToolCorrectness(expected_tools, ...)`	智能体调用了哪些工具——读取跨度树，因此需要步骤2中的 `logfire.configure()` 才能正常工作，而不仅仅是上传结果

还可用：

Contains

MaxDuration

TrajectoryMatch

ArgumentCorrectness

MaxToolCalls

MaxModelRequests

——工具/轨迹相关的评估器与

ToolCorrectness

有相同的跨度树依赖；完整列表请查看

pydantic_evals.evaluators

。这五个智能体（基于跨度）评估器需要

pydantic-evals>=2.4.0

——如果使用旧版本，请检查

pyproject.toml

uv.lock

并升级后再使用，因为导入本身就会失败，而非静默无操作。

出于安全原因，

Python

评估器（任意代码执行）已被移除——即使旧示例提到它，也不要使用。

如果编辑托管数据集：

client.push_dataset(dataset)

每次推送都会覆盖服务端的评估器，包括删除你在本地删除的评估器——不要用过时的本地副本覆盖其他人在UI中编辑过的数据集。

Step 5: Verify

步骤5：验证

A report printing to the terminal isn't proof it reached Logfire — confirm the run actually landed. Never report a case as passed, a score, or a run as complete without having actually checked it in this session — if a run fails, cancels, or produces no scores, report that failure plainly; never substitute an invented score or a manual guess at what the result "should" be.

Came from the Step 1 Braintrust path (Step 2 skipped)? There's no

whoami

-resolved project to look up here — use the SDK's own printed result URL instead, which already opens directly in the right Logfire project. Confirm the same things below (completion, pass mix, case detail) from that page rather than searching by name.

Query for the run directly, if a Logfire MCP server or API is connected — the root span for a run is named
```
evaluate {name}
```
and carries
```
gen_ai.operation.name = 'experiment'
```
,
```
dataset_name
```
, and
```
task_name
```
attributes; find the most recent one matching your dataset's name and confirm
```
logfire.experiment.metadata
```
shows the case count and pass rate you expect. Otherwise, open AI Evaluations → Datasets & Experiments → Experiments in Logfire for the exact project from Step 2, and find the run by name/timestamp.
Read the Overview tab (or the queried metadata) first: completion count, assertion pass mix, task errors, average duration. If completion says "Not reported," the run sent case data but never signaled it finished — treat that as a broken run, not a passing one.
Open the Cases tab, starting from Needs Review / Failed / Errors, not the full list.
Drill into a failing case's trace in Live view for the actual evidence, rather than trusting the summary score alone.
Fix and re-run until the cases that should pass do, and any tool-call/trajectory checks show real span data, not "No span tree available."

Close with a final report built from what you just confirmed — the run name, exact case count and pass rate you queried, and which evaluators ran — not a template. Include the direct link to this experiment (the SDK's own printed result URL, or the Datasets & Experiments page you opened it from), so the user can see the run without having to ask where to look.

报告打印到终端并不代表已上传到Logfire——请确认运行确实已完成上传。在本次会话中未实际检查之前，切勿报告用例通过、评分或运行完成——如果运行失败、取消或未生成评分，请直接报告失败情况；切勿编造评分或猜测结果“应该是什么”。

**来自步骤1的Braintrust路径（跳过了步骤2）？**此处没有通过

whoami

解析的项目可供查找——请使用SDK自身打印的结果URL，它已直接指向正确的Logfire项目。从该页面确认以下内容（完成情况、通过情况、用例详情），而非按名称搜索。

如果已连接Logfire MCP服务器或API，直接查询运行记录——运行的根跨度名为
```
evaluate {name}
```
，带有
```
gen_ai.operation.name = 'experiment'
```
、
```
dataset_name
```
和
```
task_name
```
属性；找到与你的数据集名称匹配的最新记录，并确认
```
logfire.experiment.metadata
```
显示了你预期的用例数量和通过率。否则，打开Logfire中步骤2对应的具体项目的AI评估→数据集与实验→实验页面，按名称/时间戳查找运行记录。
先查看概览标签页（或查询到的元数据）：完成数量、断言通过情况、任务错误、平均时长。如果完成状态显示“未报告”，说明运行发送了用例数据但从未发出完成信号——将其视为失败运行，而非通过。
打开用例标签页，从需要审核/失败/错误的用例开始查看，而非完整列表。
在实时视图中深入查看失败用例的追踪记录以获取实际证据，而非仅依赖汇总评分。
修复并重新运行，直到应通过的用例全部通过，且所有工具调用/轨迹检查显示真实的跨度数据，而非“No span tree available”。

最后根据你刚刚确认的内容生成最终报告——运行名称、你查询到的准确用例数量和通过率，以及运行了哪些评估器——不要使用模板。包含此实验的直接链接（SDK自身打印的结果URL，或你打开的数据集与实验页面），以便用户无需询问即可查看运行记录。