Back to Details

build-review-interface

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Build a Custom Annotation Interface

构建自定义标注界面

Overview

概述

Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.
构建一个HTML页面，从数据源（JSON/CSV文件）加载轨迹，每次显示一条轨迹，带有通过/失败按钮、自由文本备注字段以及上一条/下一条导航按钮。将标签保存到本地文件（CSV/SQLite/JSON）。然后根据以下指南针对特定领域进行定制。

Data Display

数据展示

Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.
  • Collapse repetitive elements. If every trace shares the same system prompt, put it in a 
    <details>
    toggle.
  • Extract and surface key metadata. If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
  • Color-code by role or status. Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
  • Group related elements visually. Tool calls and their responses should be visually linked (indentation, shared border).
  • Collapse what doesn't help judgment. Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
  • Highlight what matters most. Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
  • Show the full trace. Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
  • Sanitize rendered content. Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.
以该领域最易读的格式呈现所有数据。电子邮件应显示为邮件样式，代码应有语法高亮，Markdown应渲染为对应格式，表格应显示为表格形式，JSON应进行格式化并支持折叠。
  • 折叠重复元素：如果每条轨迹都包含相同的系统提示，将其放入
    <details>
    折叠控件中。
  • 提取并突出显示关键元数据：如果轨迹中包含隐藏在数据中的属性名称、客户端类型或会话ID，提取这些信息并将其作为标题或徽章突出显示。
  • 按角色或状态进行颜色编码：使用左侧边框颜色，让用户一眼就能区分用户消息、助手消息、工具调用和系统提示。
  • 视觉上分组相关元素：工具调用及其响应应在视觉上关联（缩进、共享边框）。
  • 折叠对判断无帮助的内容：冗长的工具响应JSON、中间推理步骤和调试上下文应放在折叠控件后。
  • 突出最重要的内容：让审核人员需要判断的主要内容在视觉上占据主导地位。为关键实体（价格、日期、姓名）加粗。使用字体大小和间距创建层级结构。
  • 显示完整轨迹：包含所有中间步骤（工具调用、检索到的上下文、推理过程），而不仅仅是最终输出。默认折叠这些内容，但要保持可访问性。
  • 清理渲染内容：在渲染LLM输出之前剥离原始HTML。如果渲染的Markdown中包含可能是追踪像素的图片，禁用图片显示。

Feedback Collection

反馈收集

Annotate at the trace level. The reviewer judges the whole trace, not individual spans.
  • Binary Pass/Fail buttons as the primary action.
  • Free-text notes field for the reviewer to describe what went wrong (or right).
  • Defer button for uncertain cases.
  • Auto-save on every action.
Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes. But don't add these in the initial build.
在轨迹层面进行标注。审核人员判断的是整个轨迹，而非单个跨度。
  • 将二元的通过/失败按钮作为主要操作选项。
  • 提供自由文本备注字段，供审核人员描述问题（或优点）。
  • 提供暂缓按钮，用于处理不确定的情况。
  • 每次操作后自动保存。
一旦通过错误分析确定了失败类别，后续可以添加预定义的失败模式标签，如可点击的复选框、下拉菜单或选择列表，让审核人员除了撰写备注外，还能从已知类别中选择。但初始构建时不要添加这些功能。

Navigation and Status

导航与状态

  • Next/Previous buttons and keyboard arrow keys.
  • Trace counter showing position and progress ("12 of 87 remaining").
  • Jump to specific trace by ID.
  • Counts of labeled vs unlabeled traces.
  • 上一条/下一条按钮和键盘箭头键。
  • 轨迹计数器，显示当前位置和进度（如“剩余87条中的第12条”）。
  • 通过ID跳转到特定轨迹。
  • 已标注与未标注轨迹的数量统计。

Keyboard Shortcuts

键盘快捷键

Arrow keys = Navigate traces
1 = Pass              2 = Fail
D = Defer             U = Undo last action
Cmd+S = Save          Cmd+Enter = Save and next
Arrow keys = Navigate traces
1 = Pass              2 = Fail
D = Defer             U = Undo last action
Cmd+S = Save          Cmd+Enter = Save and next

Selecting Traces to Load

选择要加载的轨迹

Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.
构建应用以接受来自任何数据源（JSON/CSV文件）的轨迹。将采样逻辑放在应用外部的单独脚本中。从随机采样开始。

Additional Features

附加功能

Reference panel: Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.
Filtering: Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).
Clustering: Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.
参考面板：可切换的面板，在轨迹旁显示基准真值、预期答案或评分标准定义。
筛选功能：根据与产品相关的元数据维度（渠道、用户类型、管道版本）筛选轨迹。
聚类功能：根据元数据或语义相似度对轨迹进行分组。显示每个聚类的代表性轨迹，并支持深入查看。

Design Checklist

设计检查清单

  • Same layout, controls, and terminology on every trace
  • Pass and Fail buttons are visually distinct (color, size)
  • Keyboard shortcuts work for all primary actions
  • Full trace accessible even when sections are collapsed
  • Labels persist automatically without explicit save
  • Trace-level annotation (not span-level) as the default
  • All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)
  • 每条轨迹的布局、控件和术语保持一致
  • 通过和失败按钮在视觉上有明显区分（颜色、大小）
  • 所有主要操作都支持键盘快捷键
  • 即使部分内容折叠，完整轨迹仍可访问
  • 标签自动持久化，无需手动保存
  • 默认在轨迹层面进行标注（而非跨度层面）
  • 所有数据都以其原生格式渲染（Markdown渲染为HTML、代码带高亮、JSON格式化、表格渲染为HTML表格、URL显示为可点击链接）

Testing

测试

After building the interface, verify it with Playwright.
Visual review: Take screenshots of the interface with representative trace data loaded. Review each screenshot for:
  • Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
  • Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
  • Aesthetics: does the interface look professional and clean? Would a domain expert use this?
  • Responsiveness: does the layout hold at different window sizes?
Functional test: Write a Playwright script that performs a full annotation workflow:
  1. Load the app and verify traces are displayed
  2. Click Pass on a trace, verify the label is saved
  3. Click Fail on a trace, add a note, verify both are saved
  4. Click Defer, verify it is recorded
  5. Navigate forward and backward with buttons and keyboard shortcuts
  6. Verify the trace counter updates correctly
  7. Verify auto-save by reloading the page and checking labels persist
  8. Expand collapsed sections (system prompts, tool calls) and verify content is accessible
  9. Test that all keyboard shortcuts trigger the correct actions
构建界面后，使用Playwright进行验证。
视觉审查：加载代表性轨迹数据后，对界面进行截图。检查每张截图：
  • 布局和间距：视觉层级是否清晰？能否立即看到关键内容？
  • 可读性：所有数据是否都以原生格式渲染？是否存在原始JSON blob、未渲染的Markdown或未设置样式的内容？
  • 美观性：界面是否看起来专业整洁？领域专家是否愿意使用？
  • 响应性：在不同窗口尺寸下布局是否保持正常？
功能测试：编写一个Playwright脚本，执行完整的标注流程：
  1. 加载应用并验证轨迹已显示
  2. 点击某条轨迹的“通过”按钮，验证标签已保存
  3. 点击某条轨迹的“失败”按钮，添加备注，验证两者均已保存
  4. 点击“暂缓”按钮，验证操作已记录
  5. 使用按钮和键盘快捷键前后导航
  6. 验证轨迹计数器更新正确
  7. 通过重新加载页面并检查标签是否持久化，验证自动保存功能
  8. 展开折叠部分（系统提示、工具调用）并验证内容可访问
  9. 测试所有键盘快捷键是否触发正确操作