Back to Details

error-discovery

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Error Discovery Skill

Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.
This skill is meant for interactive sessions only.
This skill has two parts:
  • This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
  • review-loop.md covers phase 5. You run the interactive review session.
Read this file first to build everything. Once the app is running and the human starts reviewing, follow review-loop.md.
你正在运行一个交互式错误分析会话。用户拥有一个包含LLM输出或追踪数据的数据集（JSONL、CSV、JSON等），希望通过评审样本发现失败模式。
本Skill仅适用于交互式会话。
本Skill分为两部分：
  • 本文档涵盖第1至第4阶段。你将读取数据、设计UI、构建界面并选择样本。
  • review-loop.md涵盖第5阶段。你将运行交互式评审会话。
请先阅读本文档完成所有构建工作。应用运行后，当人类开始评审时，请遵循review-loop.md的指引。

Progress updates

进度更新

Each phase takes time, especially building the interface. Tell the user what you are doing at each step. Before starting a phase, say what you are about to do and why. When a step finishes, say what you did and what comes next. For example: "Reading 10 sample records to understand the data shape", "Building the HTML app with three views: article, map, and progress", "Clustering on 5 features to select diverse samples." Do not go silent for long stretches.
每个阶段都需要时间，尤其是构建界面的阶段。在每个步骤中都要告知用户你正在做什么。开始一个阶段前，说明你即将做什么以及原因。步骤完成后，说明你做了什么以及下一步计划。例如：“读取10条样本记录以了解数据结构”、“构建包含三种视图的HTML应用：文章视图、地图视图和进度视图”、“基于5个特征进行聚类以选择多样化样本”。请勿长时间保持沉默。

Phase 1: Understand the domain and data

阶段1：理解领域与数据

Before building anything, study the dataset thoroughly.
在构建任何内容之前，请彻底研究数据集。

1a: Read and inventory the data

1a：读取并梳理数据

Load the file. Examine 5 to 10 records across the distribution. For each record, identify:
  • All fields, their types, and what they represent
  • Which fields are the primary content the human needs to judge
  • Which fields are metadata (context for understanding, but not the thing being judged)
  • Which fields vary across records and which are constant
加载文件。检查分布中的5至10条记录。针对每条记录，确定：
  • 所有字段、它们的类型以及代表的含义
  • 哪些字段是人类需要判断的核心内容
  • 哪些字段是元数据（用于理解的上下文，但不是判断对象）
  • 哪些字段在记录间存在差异，哪些是固定不变的

1b: Identify the content structure

1b：识别内容结构

The data could take many forms. Determine which pattern fits:
  • Single text field, e.g., article, summary, email, essay, translation
  • Input/output pair, e.g., prompt + completion, question + answer, instruction + result
  • Multi-turn trace, e.g., a sequence of messages with different roles (system, human, assistant, tool calls, tool results, thinking/reasoning blocks). This is common for agent traces, chatbot logs, and agentic workflows.
  • Code, e.g., source files, patches, diffs, with or without surrounding context
  • Structured output, e.g., JSON, function calls, extracted entities, classifications
  • Composite, e.g., a task description + an agent trace + a final output
For multi-turn traces specifically, identify:
  • What roles or authors exist (system, user, assistant, tool, thinking, etc.)
  • Whether there are tool call / tool result pairs
  • Whether there are thinking/reasoning blocks
  • What the logical grouping of turns is, e.g., one "step" = thinking + tool call + tool result
数据可能有多种形式。确定符合哪种模式：
  • 单一文本字段，例如文章、摘要、邮件、随笔、翻译内容
  • 输入/输出对，例如提示词+补全内容、问题+答案、指令+结果
  • 多轮追踪数据，例如包含不同角色（系统、人类、助手、工具调用、工具结果、思考/推理模块）的消息序列。这在Agent追踪数据、聊天机器人日志和Agent工作流中很常见。
  • 代码，例如源文件、补丁、差异文件，可能带有或不带有上下文
  • 结构化输出，例如JSON、函数调用、提取的实体、分类结果
  • 复合结构，例如任务描述+Agent追踪数据+最终输出
针对多轮追踪数据，需特别识别：
  • 存在哪些角色或作者（系统、用户、助手、工具、思考等）
  • 是否存在工具调用/工具结果对
  • 是否存在思考/推理模块
  • 轮次的逻辑分组方式，例如一个“步骤”=思考+工具调用+工具结果

1c: Identify dimensions of variation

1c：识别变化维度

Think about what differs between data points and within each data point. You will use these dimensions to decide the visual design.
Between data points (vary across records):
  • Metadata, e.g., topic, model, difficulty, source, label, task type
  • Structural, e.g., length, number of turns, number of tool calls
  • Outcome, e.g., success/failure, score
Within each data point (vary across parts of one record):
  • Author/role of each segment (human vs agent vs system vs tool)
  • Content type of each segment (natural language vs code vs JSON vs thinking)
  • Importance/relevance of each segment (boilerplate system prompt vs the actual response)
思考数据点之间以及每个数据点内部的差异。你将利用这些维度来决定视觉设计。
数据点之间（记录间的差异）：
  • 元数据，例如主题、模型、难度、来源、标签、任务类型
  • 结构，例如长度、轮次数量、工具调用数量
  • 结果，例如成功/失败、评分
每个数据点内部（单条记录各部分的差异）：
  • 每个片段的作者/角色（人类 vs Agent vs 系统 vs 工具）
  • 每个片段的内容类型（自然语言 vs 代码 vs JSON vs 思考内容）
  • 每个片段的重要性/相关性（模板化系统提示词 vs 实际响应内容）

1d: Think about what "bad" means

1d：定义“错误”的含义

Think of a few plausible failure categories in this domain. Expect the human to discover most of them during review.
思考该领域中一些可能的失败类别。预计人类会在评审过程中发现大多数失败类别。

Phase 2: Design the visual encoding

阶段2：设计视觉编码

Before writing any code, design how every dimension of variation maps to a visual property. Use Gestalt principles and information visualization fundamentals.
在编写任何代码之前，设计如何将每个变化维度映射到视觉属性。运用格式塔原则和信息可视化基础理论。

Core Gestalt principles to apply

需应用的核心格式塔原则

  • Similarity (color, shape). Viewers perceive things that share a visual property as related. Use this for categorical dimensions. Give different message roles different colors, and different labels different badge colors.
  • Proximity (spacing). Viewers perceive things that are close together as grouped. Use this for logical units. Place turns in a conversation step close together, and separate steps by more space.
  • Common region (containers, backgrounds). Viewers perceive things inside a shared boundary as grouped. Use this for multi-part content, e.g., a tool call and its result share a container.
  • Figure/ground (opacity, contrast). Important content should be high-contrast (figure). Less important content should recede (ground). Use this for emphasis. Show boilerplate at low opacity.
  • 相似性（颜色、形状）。观看者会将具有相同视觉属性的事物视为相关联。将此用于分类维度。为不同的消息角色分配不同的颜色，为不同的标签分配不同的徽章颜色。
  • 接近性（间距）。观看者会将距离较近的事物视为一组。将此用于逻辑单元。将对话步骤中的连续消息紧密排列，步骤之间用更大的间距分隔。
  • 共同区域（容器、背景）。观看者会将共享边界内的事物视为一组。将此用于多部分内容，例如将工具调用及其结果放在同一个容器中。
  • 图形/背景（透明度、对比度）。重要内容应具有高对比度（图形）。次要内容应弱化（背景）。将此用于强调。将模板化内容以低透明度显示。

Visual encoding rules

视觉编码规则

Map each dimension of variation to exactly one visual channel. Do not use the same channel for two different things.
Color hue. Use for the categorical dimension with the most important distinction.
  • For multi-turn traces, use color for message role. Each role (human, assistant, system, tool, thinking) gets its own distinct hue at full saturation. Pick 3 to 5 easily distinguishable hues.
  • For single-content, use color for label or source category.
  • Do not use color hue for quantitative data.
Opacity / saturation. Use to show whether the human needs to read this content carefully.
  • Reduce opacity only for content the human genuinely does not need to read, e.g., repeated boilerplate that is identical across records, verbose tool schemas, auto-generated headers. Ask yourself: "if I removed this, would the human miss anything?"
  • Do not mute content by role. System messages, tool results, and thinking blocks can all contain the actual bug. Mute only specific content that is redundant or mechanical, regardless of which role produced it.
  • Normal content gets full opacity.
Spacing. Use for hierarchical structure.
  • Tight spacing (4 to 8px) between items within a logical group, e.g., consecutive messages in one "step" of an agent trace.
  • Medium spacing (16 to 24px) between logical groups, e.g., between steps.
  • Large spacing (32 to 48px) between major sections.
Typography. Use for content type.
  • Natural language prose: proportional font, normal size.
  • Code: monospace font, slightly smaller.
  • Metadata: small, muted, compact.
  • Thinking/reasoning: italic or a distinct font treatment to signal "internal."
Border / container. Use for grouping related parts.
  • Put a tool call and its tool result in a shared container with a subtle border.
  • Separate an input/output pair with a clear divider, or place them side by side.
Structural outlier flags. Show in the header only, not inline.
  • Pre-compute dataset-level averages for key structural features (length, heading count, paragraph count, etc.).
  • For each record, flag dimensions where it is a clear statistical outlier (e.g., top/bottom 10%).
  • Show these as small, compact badges in the header, e.g., "6 headings (more than 89%)" or "shorter than 95%."
  • Keep it minimal. Most records should have zero or one flag. Do not annotate inline content.
将每个变化维度精确映射到一个视觉通道。不要将同一个通道用于两种不同的用途。
颜色色相。用于最重要的分类维度区分。
  • 对于多轮追踪数据，用颜色区分消息角色。每个角色（人类、助手、系统、工具、思考）分配一个独特的全饱和度色相。选择3至5种易于区分的色相。
  • 对于单一内容，用颜色区分标签或来源类别。
  • 不要将颜色色相用于定量数据。
透明度/饱和度。用于显示人类是否需要仔细阅读该内容。
  • 仅对人类确实无需阅读的内容降低透明度，例如记录间重复的模板化内容、冗长的工具 schema、自动生成的标题。自问：“如果我删除这部分内容，人类会遗漏什么信息吗？”
  • 不要按角色弱化内容。系统消息、工具结果和思考模块都可能包含实际的错误。仅弱化冗余或机械性的特定内容，无论其由哪个角色生成。
  • 常规内容保持全透明度。
间距。用于层级结构。
  • 逻辑组内的项目使用紧密间距（4至8px），例如Agent追踪数据中一个“步骤”内的连续消息。
  • 逻辑组之间使用中等间距（16至24px），例如步骤之间。
  • 主要部分之间使用大间距（32至48px）。
排版。用于内容类型区分。
  • 自然语言散文：比例字体，常规大小。
  • 代码：等宽字体，略小尺寸。
  • 元数据：小字号、弱化、紧凑排版。
  • 思考/推理内容：斜体或独特的字体样式以表示“内部内容”。
边框/容器。用于分组相关部分。
  • 将工具调用及其结果放在带有细微边框的共享容器中。
  • 用清晰的分隔线分隔输入/输出对，或在两者都较短时并排显示。
结构异常标记。仅在标题中显示，不要内联显示。
  • 预先计算关键结构特征（长度、标题数量、段落数量等）的数据集级平均值。
  • 针对每条记录，标记明显统计异常的维度（例如前/后10%）。
  • 将这些标记为标题中的小型紧凑徽章，例如“6个标题（超过89%的记录）”或“比95%的记录更短”。
  • 保持简洁。大多数记录应具有0或1个标记。不要对内联内容添加注释。

Phase 3: Build the review interface

阶段3：构建评审界面

Architecture

架构

  • Python HTTP server (stdlib 
    http.server
    , no dependencies): 
    • GET /
      serves the HTML app
    • GET /api/samples
      returns the current sample set
    • POST /api/samples
      lets the agent push new samples
    • GET /api/annotations
      returns the current annotations
    • POST /api/annotations
      lets the app save annotations on every change
    • GET /api/graph
      returns the 2D projection of all records for the cluster map
    • GET /api/patterns
      returns the agent's current failure mode taxonomy
    • POST /api/patterns
      lets the agent push the updated taxonomy
    • GET /api/suggestions
      returns agent-suggested annotations
    • POST /api/suggestions
      lets the agent push suggestions
  • On-disk files in an 
    error_discovery_data/
    directory: 
    • samples.json
      , 
      annotations.json
      , 
      graph.json
      , 
      patterns.json
      , 
      suggestions.json
  • The HTML app auto-saves to the server on every annotation. It polls for new samples and suggestions.
  • Python HTTP服务器（标准库
    http.server
    ，无依赖）： 
    • GET /
      提供HTML应用
    • GET /api/samples
      返回当前样本集
    • POST /api/samples
      允许Agent推送新样本
    • GET /api/annotations
      返回当前标注内容
    • POST /api/annotations
      允许应用在每次更改时保存标注
    • GET /api/graph
      返回所有记录的2D投影用于聚类地图
    • GET /api/patterns
      返回Agent当前的失败模式分类体系
    • POST /api/patterns
      允许Agent推送更新后的分类体系
    • GET /api/suggestions
      返回Agent建议的标注内容
    • POST /api/suggestions
      允许Agent推送建议内容
  • 磁盘文件存储在
    error_discovery_data/
    目录中： 
    • samples.json
      、
      annotations.json
      、
      graph.json
      、
      patterns.json
      、
      suggestions.json
  • HTML应用在每次标注时自动保存到服务器。它会轮询新样本和建议内容。

HTML app structure

HTML应用结构

Three views, toggled from the top bar:
  1. Article/content view. The main review interface where the human reads and annotates.
  2. Map view. A 2D scatter plot (PCA or UMAP projection) of all records. It shows clusters, which items are in the sample, and which have been annotated. The reviewer can click a sample node to go to its content view.
  3. Progress view. Two sections:
    • Failure modes: a treemap of modes the agent has categorized so far. Each block is a failure mode, sized by annotation count. Inside each block, list the notes. The reviewer can click a note to go to that annotation in the content view.
    • Agent suggestions to review: a list of pending suggestions with checkboxes. Each row shows the failure mode, quoted text, and source record. The reviewer can click the text to go to that spot in the article. At the top, a "Select all" checkbox and "Accept selected" / "Dismiss selected" buttons. This lets the reviewer select all, uncheck the few they disagree with, and accept the rest in one click.
Content view design. Apply the visual encoding from Phase 2:
  • Header: title + label + topic. Keep it minimal. Add structural outlier flags (from Phase 2) only when the record is a genuine outlier.
  • Body: render the primary content using the appropriate treatment per content type.
    For multi-turn traces (agent logs, conversations, chat):
    • Each message/turn is a block. Left-align all blocks but use a colored left-border or background tint per role.
    • Role label (small, bold) at the top of each block, e.g., "System", "User", "Assistant", "Tool Call", "Tool Result", "Thinking".
    • Assign a distinct hue to each role. Be consistent across all records. All roles at full opacity by default.
    • Render thinking/reasoning blocks in a visually distinct way (e.g., lighter background, italic, or slightly indented) to show that they are internal monologue, while keeping full readability.
    • Show tool call function names prominently. Put parameters in collapsible formatted JSON.
    • Make tool results collapsible by default if they are long, with a summary line visible.
    • Only reduce opacity for content that is literally identical across records, e.g., the same system prompt repeated verbatim in every trace. If system messages vary, keep them fully visible.
    • Group related turns: a thinking block + the tool call it produces + the tool result, visually grouped with tight spacing and a shared container.
    For single text content (articles, summaries, etc.):
    • Render markdown as formatted HTML (use marked.js or similar).
    • Render plain text with paragraph breaks.
    For code / diffs:
    • Use syntax highlighting (highlight.js or Prism via CDN).
    • Show additions with green background and deletions with red background.
    • Show line numbers.
    For input/output pairs:
    • Stack them with a clear divider, or place them side by side if both are short.
    • Label each section.
  • Inline annotation: the reviewer selects text, a floating popover appears with a text input, they press Enter to save, and the span is highlighted. When the popover appears and the input is focused, the browser clears the native text selection. To prevent this, wrap the selected range in a temporary highlight span (e.g., class "pending-highlight" with a visible background) BEFORE focusing the input. Remove the temporary highlight when the annotation is saved, cancelled, or the popover is dismissed by clicking outside. This way the reviewer always sees what text they are annotating.
  • Margin notes: annotations and suggestions must appear as side notes in a right margin column, aligned vertically with their corresponding highlighted text. Use a two-column layout: the article body on the left (flex: 1, max-width ~720px) and a margin-notes column on the right (width ~240px). Each margin note is position: absolute inside the margin column, with its top offset calculated from the highlight element's position relative to the margin container (use getBoundingClientRect on both the highlight and the margin container, take the difference — do NOT add scrollTop, as that double-counts the scroll offset). Stack notes with a minimum gap so they do not overlap. Include hover linking: hovering a margin note outlines its highlight, and hovering a highlight outlines its margin note. Each margin note shows the quoted text, the reviewer's note, and edit/delete buttons (visible on hover). Do NOT use hover tooltips as the primary way to show annotation content — margin notes replace tooltips.
  • Agent suggestions: visually distinct from human annotations in both the inline highlight (dashed border, muted tint) and the margin note (different left-border color, an "agent suggestion" tag). The margin note shows accept/dismiss buttons (always visible, not hover-gated). The reviewer can accept (which promotes it to an annotation) or dismiss.
  • No quality labels, no dropdowns, no structured forms. Free-text notes only.
  • Auto-save to server on every change. Keep a localStorage backup.
  • Poll for new samples and suggestions periodically. Show a banner or toast when new ones arrive.
Map view:
  • 2D scatter of all records from PCA/UMAP projection.
  • Color by cluster (match hull colors), not by label.
  • Use shape to distinguish categories, e.g., circles for AI and squares for human.
  • Show sample items as larger nodes with a dark border.
  • Show annotated items in a distinct color, e.g., orange.
  • Draw cluster hulls or convex boundaries as subtle background shapes.
  • On hover, show a tooltip with title, metadata, and annotation count.
  • On click (sample nodes only), go to the content view for that item.
三种视图，可通过顶部栏切换：
  1. 文章/内容视图。人类阅读和标注的主要评审界面。
  2. 地图视图。所有记录的2D散点图（PCA或UMAP投影）。显示聚类、样本中的项目以及已标注的项目。评审者可以点击样本节点进入其内容视图。
  3. 进度视图。包含两个部分：
    • 失败模式：Agent已分类的模式树形图。每个块代表一个失败模式，大小由标注数量决定。每个块内列出注释。评审者可以点击注释进入该标注对应的内容视图。
    • Agent建议评审列表：带有复选框的待处理建议列表。每行显示失败模式、引用文本和来源记录。评审者可以点击文本进入文章中的对应位置。顶部有“全选”复选框和“接受选中项”/“驳回选中项”按钮。这让评审者可以全选，取消勾选少数不同意的项，然后一键接受其余项。
内容视图设计。应用阶段2的视觉编码：
  • 标题：标题+标签+主题。保持简洁。仅当记录确实是异常值时，添加阶段2中的结构异常标记。
  • 主体：根据内容类型使用相应的方式渲染核心内容。
    对于多轮追踪数据（Agent日志、对话、聊天）：
    • 每条消息/轮次是一个块。所有块左对齐，但每个角色使用不同颜色的左边框或背景色调。
    • 每个块顶部显示角色标签（小字号、加粗），例如“System”、“User”、“Assistant”、“Tool Call”、“Tool Result”、“Thinking”。
    • 为每个角色分配独特的色相。在所有记录中保持一致。所有角色默认全透明度。
    • 以视觉独特的方式渲染思考/推理块（例如浅色背景、斜体或略微缩进），以表明它们是内部独白，同时保持完全可读性。
    • 突出显示工具调用的函数名称。将参数放在可折叠的格式化JSON中。
    • 如果工具结果较长，默认设置为可折叠，显示摘要行。
    • 仅对记录间完全相同的内容降低透明度，例如每个追踪中重复的相同系统提示词。如果系统消息存在差异，则保持完全可见。
    • 分组相关轮次：将思考块+其生成的工具调用+工具结果进行视觉分组，使用紧密间距和共享容器。
    对于单一文本内容（文章、摘要等）：
    • 将markdown渲染为格式化HTML（使用marked.js或类似工具）。
    • 渲染纯文本并保留段落换行。
    对于代码/差异文件：
    • 使用语法高亮（通过CDN使用highlight.js或Prism）。
    • 新增内容显示绿色背景，删除内容显示红色背景。
    • 显示行号。
    对于输入/输出对：
    • 堆叠显示并使用清晰的分隔线，或在两者都较短时并排显示。
    • 为每个部分添加标签。
  • 内联标注：评审者选择文本后，会出现一个浮动弹窗，带有文本输入框，按下Enter即可保存，选中的文本会被高亮。当弹窗出现且输入框获得焦点时，浏览器会清除原生文本选择。为避免这种情况，在聚焦输入框之前，将选中范围包裹在临时高亮span中（例如类名为“pending-highlight”的可见背景）。当标注保存、取消或点击弹窗外部关闭时，移除临时高亮。这样评审者始终能看到正在标注的文本。
  • 边栏注释：标注和建议必须显示在右侧边栏的注释中，与对应的高亮文本垂直对齐。使用两栏布局：左侧为文章主体（flex: 1，最大宽度约720px），右侧为边栏注释列（宽度约240px）。每个边栏注释在边栏列中设置为position: absolute，其顶部偏移量根据高亮元素相对于边栏容器的位置计算（对高亮元素和边栏容器使用getBoundingClientRect，取差值——不要添加scrollTop，因为这会重复计算滚动偏移）。注释之间保留最小间距以避免重叠。包含悬停链接：悬停边栏注释时，其对应的高亮文本会显示轮廓；悬停高亮文本时，其对应的边栏注释会显示轮廓。每个边栏注释显示引用文本、评审者的注释以及编辑/删除按钮（悬停时可见）。请勿将悬停提示作为显示标注内容的主要方式——边栏注释替代提示框。
  • Agent建议：在高亮文本（虚线边框、弱化色调）和边栏注释（不同的左边框颜色、“Agent建议”标签）上都与人类标注视觉区分开。边栏注释显示接受/驳回按钮（始终可见，无需悬停）。评审者可以接受（将其升级为标注）或驳回。
  • 无质量标签、无下拉菜单、无结构化表单。仅支持自由文本注释。
  • 每次更改时自动保存到服务器。保留localStorage备份。
  • 定期轮询新样本和建议内容。当有新内容到达时显示横幅或提示框。
地图视图：
  • 所有记录的PCA/UMAP投影2D散点图。
  • 按聚类着色（匹配 hull 颜色），而非按标签着色。
  • 使用形状区分类别，例如圆形代表AI，方形代表人类。
  • 样本项显示为带有深色边框的大节点。
  • 已标注项显示为独特颜色，例如橙色。
  • 绘制聚类外壳或凸边界作为微妙的背景形状。
  • 悬停时显示包含标题、元数据和标注数量的提示框。
  • 点击（仅样本节点）进入该项目的内容视图。

Phase 4: Cluster and select initial samples

阶段4：聚类并选择初始样本

  1. Extract features appropriate to the content type (see Phase 1c for the dimensions).
  2. Cluster using KMeans or similar on normalized features. Target 6 to 10 clusters.
  3. Build the initial sample (15 to 25 items for datasets over 50):
    • Cluster representatives (about 60 to 70%): 1 to 2 items closest to each centroid, mixing categories/labels.
    • Random samples (about 30 to 40%): from the full dataset regardless of cluster. The clustering may not capture every important dimension, so random picks help cover what it misses.
    • Remove duplicates.
  4. Prioritize diversity. The goal is discovering failure modes, not estimating how common they are.
  1. 提取适合内容类型的特征（参见阶段1c的维度）。
  2. 使用KMeans或类似算法对标准化特征进行聚类。目标为6至10个聚类。
  3. 构建初始样本（数据集超过50条时选择15至25项）：
    • 聚类代表（约60%至70%）：每个聚类中心附近的1至2项，混合不同类别/标签。
    • 随机样本（约30%至40%）：来自整个数据集，不考虑聚类。聚类可能无法覆盖所有重要维度，因此随机选择有助于覆盖遗漏的部分。
    • 移除重复项。
  4. 优先考虑多样性。目标是发现失败模式，而非估计其出现频率。

Phase 5: Run the interactive review loop

阶段5：运行交互式评审循环

Once the app is running and the human starts reviewing, follow review-loop.md for the ongoing interactive session, including how to monitor annotations as they arrive.
If the session ends before a human can review (a non-interactive run, for example 
codex exec
or 
claude -p
), build and smoke-test the app, then stop the server and give the command to launch it later; never claim the review loop ran or that anything is still running.
应用运行后，当人类开始评审时，请遵循review-loop.md进行持续的交互式会话，包括如何监控到达的标注内容。
如果会话在人类能够评审之前结束（例如非交互式运行，如
codex exec
或
claude -p
），请构建并冒烟测试应用，然后停止服务器并提供稍后启动的命令；切勿声称评审循环已运行或任何内容仍在运行。