Back to Details

databricks-mlflow-evaluation

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

MLflow 3 GenAI Evaluation

MLflow 3 GenAI 评估

Scope vs upstream 
mlflow/skills

与上游
mlflow/skills
的范围对比

The OSS 
mlflow/skills
repo ships 
agent-evaluation
 and related skills (
instrumenting-with-mlflow-tracing
, 
analyze-mlflow-trace
, 
retrieving-mlflow-traces
, 
querying-mlflow-metrics
) that cover the generic MLflow GenAI evaluation workflow — 
mlflow.genai.evaluate()
, scorers/judges, datasets, tracing setup, and the 5-step evaluation loop.
This skill layers Databricks-specific patterns on top of that workflow rather than restating it. Use this skill when you need any of:
  • Unity Catalog trace ingestion — production traces written into UC tables, log-based monitoring (
    patterns-trace-ingestion.md
    ).
  • MemAlign judge alignment via UC SME labeling sessions — aligning custom judges against domain-expert feedback collected in Databricks (
    patterns-judge-alignment.md
    ).
  • optimize_prompts()
    GEPA loop     — Databricks' automated prompt-optimization driver running on a UC dataset (
    patterns-prompt-optimization.md
    ).
  • Databricks-flavored scorer/dataset patterns — UC-table-backed datasets, tagging traces in the Databricks UI for inclusion (
    patterns-datasets.md
    , 
    patterns-scorers.md
    ).
For everything else — generic 
mlflow.genai.evaluate()
calls, scorer authoring patterns, dataset creation outside Databricks, MLflow tracing setup that isn't UC-table-bound — the upstream 
mlflow/skills/agent-evaluation
skill is the canonical source and is kept current by the MLflow team.
开源（OSS）
mlflow/skills
仓库提供了
agent-evaluation
及相关技能（
instrumenting-with-mlflow-tracing
、
analyze-mlflow-trace
、
retrieving-mlflow-traces
、
querying-mlflow-metrics
），涵盖通用的MLflow GenAI评估工作流——包括
mlflow.genai.evaluate()
、评分器/评审模型（scorers/judges）、数据集、追踪设置以及五步评估循环。
本技能在上述工作流基础上叠加了Databricks专属模式，而非重复通用内容。当你需要以下任意功能时，使用本技能：
  • Unity Catalog追踪数据摄入——将生产环境追踪数据写入UC表，实现基于日志的监控（
    patterns-trace-ingestion.md
    ）。
  • 通过UC领域专家（SME）标注会话实现MemAlign评审模型对齐——针对在Databricks收集的领域专家反馈，对齐自定义评审模型（
    patterns-judge-alignment.md
    ）。
  • optimize_prompts()
    GEPA循环    ——运行Databricks的自动提示词优化驱动程序，基于UC数据集执行优化（
    patterns-prompt-optimization.md
    ）。
  • Databricks风格的评分器/数据集模式——基于UC表的数据集、在Databricks UI中标记追踪数据以纳入数据集（
    patterns-datasets.md
    、
    patterns-scorers.md
    ）。
对于其他场景——通用
mlflow.genai.evaluate()
调用、评分器编写模式、Databricks外的数据集创建、非UC表绑定的MLflow追踪设置——上游
mlflow/skills/agent-evaluation
技能是权威来源，由MLflow团队持续维护更新。

Before Writing Any Code

编写代码前的准备

  1. Read GOTCHAS.md - 15+ common mistakes that cause failures
  2. Read CRITICAL-interfaces.md - Exact API signatures and data schemas
  1. 阅读GOTCHAS.md——包含15+种导致失败的常见错误
  2. 阅读CRITICAL-interfaces.md——明确API签名和数据 schema

End-to-End Workflows

端到端工作流

Follow these workflows based on your goal. Each step indicates which reference files to read.
根据你的目标选择以下工作流，每个步骤标注了需参考的文件。

Workflow 1: First-Time Evaluation Setup

工作流1：首次评估设置

For users new to MLflow GenAI evaluation or setting up evaluation for a new agent.
StepActionReference Files
1Understand what to evaluate
user-journeys.md
(Journey 0: Strategy)
2Learn API patterns
GOTCHAS.md
+ 
CRITICAL-interfaces.md
3Build initial dataset
patterns-datasets.md
(Patterns 1-4)
4Choose/create scorers
patterns-scorers.md
+ 
CRITICAL-interfaces.md
(built-in list)
5Run evaluation
patterns-evaluation.md
(Patterns 1-3)
适用于初次接触MLflow GenAI评估，或为新代理设置评估的用户。
步骤操作参考文件
1明确评估对象
user-journeys.md
（旅程0：策略）
2学习API模式
GOTCHAS.md
+ 
CRITICAL-interfaces.md
3构建初始数据集
patterns-datasets.md
（模式1-4）
4选择/创建评分器
patterns-scorers.md
+ 
CRITICAL-interfaces.md
（内置列表）
5运行评估
patterns-evaluation.md
（模式1-3）

Workflow 2: Production Trace -> Evaluation Dataset

工作流2：生产追踪数据 → 评估数据集

For building evaluation datasets from production traces.
StepActionReference Files
1Search and filter traces
patterns-trace-analysis.md
(MCP tools section)
2Analyze trace quality
patterns-trace-analysis.md
(Patterns 1-7)
3Tag traces for inclusion
patterns-datasets.md
(Patterns 16-17)
4Build dataset from traces
patterns-datasets.md
(Patterns 6-7)
5Add expectations/ground truth
patterns-datasets.md
(Pattern 2)
适用于从生产环境追踪数据构建评估数据集。
步骤操作参考文件
1搜索并过滤追踪数据
patterns-trace-analysis.md
（MCP工具章节）
2分析追踪数据质量
patterns-trace-analysis.md
（模式1-7）
3标记需纳入的追踪数据
patterns-datasets.md
（模式16-17）
4从追踪数据构建数据集
patterns-datasets.md
（模式6-7）
5添加预期结果/基准事实
patterns-datasets.md
（模式2）

Workflow 3: Performance Optimization

工作流3：性能优化

For debugging slow or expensive agent execution.
StepActionReference Files
1Profile latency by span
patterns-trace-analysis.md
(Patterns 4-6)
2Analyze token usage
patterns-trace-analysis.md
(Pattern 9)
3Detect context issues
patterns-context-optimization.md
(Section 5)
4Apply optimizations
patterns-context-optimization.md
(Sections 1-4, 6)
5Re-evaluate to measure impact
patterns-evaluation.md
(Pattern 6-7)
适用于调试执行缓慢或成本高昂的代理。
步骤操作参考文件
1按跨度分析延迟
patterns-trace-analysis.md
（模式4-6）
2分析令牌使用情况
patterns-trace-analysis.md
（模式9）
3检测上下文问题
patterns-context-optimization.md
（第5节）
4应用优化方案
patterns-context-optimization.md
（第1-4、6节）
5重新评估以衡量优化效果
patterns-evaluation.md
（模式6-7）

Workflow 4: Regression Detection

工作流4：回归检测

For comparing agent versions and finding regressions.
StepActionReference Files
1Establish baseline
patterns-evaluation.md
(Pattern 4: named runs)
2Run current version
patterns-evaluation.md
(Pattern 1)
3Compare metrics
patterns-evaluation.md
(Patterns 6-7)
4Analyze failing traces
patterns-trace-analysis.md
(Pattern 7)
5Debug specific failures
patterns-trace-analysis.md
(Patterns 8-9)
适用于对比代理版本并发现回归问题。
步骤操作参考文件
1建立基准线
patterns-evaluation.md
（模式4：命名运行）
2运行当前版本
patterns-evaluation.md
（模式1）
3对比指标
patterns-evaluation.md
（模式6-7）
4分析失败的追踪数据
patterns-trace-analysis.md
（模式7）
5调试特定失败案例
patterns-trace-analysis.md
（模式8-9）

Workflow 5: Custom Scorer Development

工作流5：自定义评分器开发

For creating project-specific evaluation metrics.
StepActionReference Files
1Understand scorer interface
CRITICAL-interfaces.md
(Scorer section)
2Choose scorer pattern
patterns-scorers.md
(Patterns 4-11)
3For multi-agent scorers
patterns-scorers.md
(Patterns 13-16)
4Test with evaluation
patterns-evaluation.md
(Pattern 1)
适用于创建项目专属的评估指标。
步骤操作参考文件
1理解评分器接口
CRITICAL-interfaces.md
（评分器章节）
2选择评分器模式
patterns-scorers.md
（模式4-11）
3多代理评分器相关
patterns-scorers.md
（模式13-16）
4通过评估测试评分器
patterns-evaluation.md
（模式1）

Workflow 6: Unity Catalog Trace Ingestion & Production Monitoring

工作流6：Unity Catalog追踪数据摄入与生产监控

For storing traces in Unity Catalog, instrumenting applications, and enabling continuous production monitoring.
StepActionReference Files
1Link UC schema to experiment
patterns-trace-ingestion.md
(Patterns 1-2)
2Set trace destination
patterns-trace-ingestion.md
(Patterns 3-4)
3Instrument your application
patterns-trace-ingestion.md
(Patterns 5-8)
4Configure trace sources (Apps/Serving/OTEL)
patterns-trace-ingestion.md
(Patterns 9-11)
5Enable production monitoring
patterns-trace-ingestion.md
(Patterns 12-13)
6Query and analyze UC traces
patterns-trace-ingestion.md
(Pattern 14)
适用于将追踪数据存储到Unity Catalog、为应用添加埋点、启用持续生产监控。
步骤操作参考文件
1将UC schema关联到实验
patterns-trace-ingestion.md
（模式1-2）
2设置追踪数据目标
patterns-trace-ingestion.md
（模式3-4）
3为应用添加埋点
patterns-trace-ingestion.md
（模式5-8）
4配置追踪数据源（应用/服务/OTEL）
patterns-trace-ingestion.md
（模式9-11）
5启用生产监控
patterns-trace-ingestion.md
（模式12-13）
6查询并分析UC追踪数据
patterns-trace-ingestion.md
（模式14）

Workflow 7: Judge Alignment with MemAlign

工作流7：借助MemAlign对齐评审模型

For aligning an LLM judge to match domain expert preferences. A well-aligned judge improves every downstream use: evaluation accuracy, production monitoring signal, and prompt optimization quality. This workflow is valuable on its own, independent of prompt optimization.
StepActionReference Files
1Design base judge with 
make_judge
(any feedback type)
patterns-judge-alignment.md
(Pattern 1)
2Run evaluate(), tag successful traces
patterns-judge-alignment.md
(Pattern 2)
3Build UC dataset + create SME labeling session
patterns-judge-alignment.md
(Pattern 3)
4Align judge with MemAlign after labeling completes
patterns-judge-alignment.md
(Pattern 4)
5Register aligned judge to experiment
patterns-judge-alignment.md
(Pattern 5)
6Re-evaluate with aligned judge (baseline)
patterns-judge-alignment.md
(Pattern 6)
适用于让LLM评审模型匹配领域专家偏好。对齐良好的评审模型可提升所有下游场景的效果：评估准确性、生产监控信号质量、提示词优化效果。本工作流可独立于提示词优化使用，自身具备很高价值。
步骤操作参考文件
1使用
make_judge
设计基础评审模型（支持任意反馈类型）
patterns-judge-alignment.md
（模式1）
2运行evaluate()，标记成功的追踪数据
patterns-judge-alignment.md
（模式2）
3构建UC数据集 + 创建领域专家标注会话
patterns-judge-alignment.md
（模式3）
4标注完成后通过MemAlign对齐评审模型
patterns-judge-alignment.md
（模式4）
5将对齐后的评审模型注册到实验
patterns-judge-alignment.md
（模式5）
6使用对齐后的评审模型重新评估（建立基准）
patterns-judge-alignment.md
（模式6）

Workflow 8: Automated Prompt Optimization with GEPA

工作流8：借助GEPA实现自动提示词优化

For automatically improving a registered system prompt using 
optimize_prompts()
. Works with any scorer, but paired with an aligned judge (Workflow 7) gives the most domain-accurate signal. For the full end-to-end loop combining alignment and optimization, see 
user-journeys.md
Journey 10.
StepActionReference Files
1Build optimization dataset (inputs + expectations)
patterns-prompt-optimization.md
(Pattern 1)
2Run optimize_prompts() with GEPA + scorer
patterns-prompt-optimization.md
(Pattern 2)
3Register new version, promote conditionally
patterns-prompt-optimization.md
(Pattern 3)
适用于使用
optimize_prompts()
自动优化已注册的系统提示词。可搭配任意评分器使用，但与对齐后的评审模型（工作流7）结合时，能获得最贴合领域的准确信号。如需结合对齐与优化的完整端到端循环，请查看
user-journeys.md
的旅程10。
步骤操作参考文件
1构建优化数据集（输入+预期结果）
patterns-prompt-optimization.md
（模式1）
2使用GEPA + 评分器运行optimize_prompts()
patterns-prompt-optimization.md
（模式2）
3注册新版本，按需推广
patterns-prompt-optimization.md
（模式3）

Reference Files Quick Lookup

参考文件速查

ReferencePurposeWhen to Read
GOTCHAS.md
Common mistakesAlways read first before writing code
CRITICAL-interfaces.md
API signatures, schemasWhen writing any evaluation code
patterns-evaluation.md
Running evals, comparingWhen executing evaluations
patterns-scorers.md
Custom scorer creationWhen built-in scorers aren't enough
patterns-datasets.md
Dataset buildingWhen preparing evaluation data
patterns-trace-analysis.md
Trace debuggingWhen analyzing agent behavior
patterns-context-optimization.md
Token/latency fixesWhen agent is slow or expensive
patterns-trace-ingestion.md
UC trace setup, monitoringWhen setting up trace storage or production monitoring
patterns-judge-alignment.md
MemAlign judge alignment, labeling sessions, SME feedbackWhen aligning judges to domain expert preferences
patterns-prompt-optimization.md
GEPA optimization: build dataset, optimize_prompts(), promoteWhen running automated prompt improvement
user-journeys.md
High-level workflows, full domain-expert optimization loopWhen starting a new evaluation project or running the full align + optimize cycle
参考文件用途阅读时机
GOTCHAS.md
常见错误汇总编写代码前务必阅读
CRITICAL-interfaces.md
API签名、schema编写任何评估代码时
patterns-evaluation.md
运行评估、对比结果执行评估任务时
patterns-scorers.md
创建自定义评分器内置评分器无法满足需求时
patterns-datasets.md
构建数据集准备评估数据时
patterns-trace-analysis.md
追踪数据调试分析代理行为时
patterns-context-optimization.md
令牌/延迟优化代理执行缓慢或成本过高时
patterns-trace-ingestion.md
UC追踪设置、监控设置追踪数据存储或生产监控时
patterns-judge-alignment.md
MemAlign评审模型对齐、标注会话、领域专家反馈让评审模型匹配领域专家偏好时
patterns-prompt-optimization.md
GEPA优化：构建数据集、optimize_prompts()、版本推广运行自动提示词优化时
user-journeys.md
高阶工作流、完整领域专家优化循环启动新评估项目或运行完整对齐+优化循环时

Critical API Facts

关键API要点

  • Use: 
    mlflow.genai.evaluate()
    (NOT 
    mlflow.evaluate()
    )
  • Data format: 
    {"inputs": {"query": "..."}}
    (nested structure required)
  • predict_fn: Receives 
    **unpacked kwargs
    (not a dict)
  • MemAlign: Scorer-agnostic (works with any 
    feedback_value_type
    -- float, bool, categorical); token-heavy on the embedding model so set 
    embedding_model
    explicitly
  • Label schema name matching: The label schema 
    name
    in the labeling session MUST match the judge 
    name
    used in 
    evaluate()
    for 
    align()
    to pair scores
  • Aligned judge scores: May be lower than unaligned judge scores -- this is expected and means the judge is now more accurate, not that the agent regressed
  • GEPA optimization dataset: Must have both 
    inputs
    AND 
    expectations
    per record (different from eval dataset)
  • Episodic memory: Lazily loaded -- 
    get_scorer()
    results won't show episodic memory on print until the judge is first used
  • optimize_prompts: Requires MLflow >= 3.5.0
See 
GOTCHAS.md
for complete list.
  • 使用： 
    mlflow.genai.evaluate()
    （而非
    mlflow.evaluate()
    ）
  • 数据格式： 
    {"inputs": {"query": "..."}}
    （必须使用嵌套结构）
  • predict_fn： 接收
    **unpacked kwargs
    （而非字典）
  • MemAlign： 与评分器无关（支持任意
    feedback_value_type
    ——浮点数、布尔值、分类值）；嵌入模型令牌消耗较大，请显式设置
    embedding_model
  • 标签schema名称匹配： 标注会话中的标签schema
    name
    必须与
    evaluate()
    中使用的评审模型
    name
    一致，才能让
    align()
    匹配评分结果
  • 对齐后评审模型的评分： 可能低于未对齐评审模型的评分——这是正常现象，说明评审模型现在更准确，而非代理性能退化
  • GEPA优化数据集： 每条记录必须同时包含
    inputs
    和
    expectations
    （与评估数据集不同）
  • 情景记忆： 延迟加载——
    get_scorer()
    的结果在打印时不会显示情景记忆，直到评审模型首次被使用
  • optimize_prompts： 需要MLflow >= 3.5.0
完整列表请查看
GOTCHAS.md
。

Related Skills

相关技能

  • databricks-docs - General Databricks documentation reference
  • databricks-model-serving - Deploying models and agents to serving endpoints
  • databricks-agent-bricks - Building agents that can be evaluated with this skill
  • databricks-python-sdk - SDK patterns used alongside MLflow APIs
  • databricks-unity-catalog - Unity Catalog tables for managed evaluation datasets
  • databricks-docs - Databricks通用文档参考
  • databricks-model-serving - 将模型和代理部署到服务端点
  • databricks-agent-bricks - 构建可通过本技能评估的代理
  • databricks-python-sdk - 与MLflow API搭配使用的SDK模式
  • databricks-unity-catalog - 用于托管评估数据集的Unity Catalog表",