Back to Details

llm-auditor

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

LLM Auditor

LLM Auditor

Evaluate and improve the factual grounding of LLM-generated responses. Acts as an automated fact-checking layer that systematically analyzes claims against real-world information.
评估并提升LLM生成回复的事实可靠性。作为一个自动化事实核查层，系统地对照真实世界信息分析各项主张。

When to Use This Skill

何时使用该技能

Activate when the user says things like:
  • "audit that", "fact-check that", "double-check that"
  • "verify those claims", "is that accurate?"
  • "check your sources", "are you sure about that?"
  • "audit this: [text]"
  • Any request to validate factual accuracy of a previous response or provided text
当用户说出以下内容时激活：
  • “审计该内容”、“事实核查该内容”、“复核该内容”
  • “验证这些主张”、“那准确吗？”
  • “检查你的来源”、“你确定吗？”
  • “审计此内容：[文本]”
  • 任何要求验证之前回复或提供文本的事实准确性的请求

The Audit Process

审计流程

Execute these three phases sequentially. Do NOT skip phases.
按顺序执行以下三个阶段，不得跳过任何阶段。

Phase 1: Critic — Extract and Verify Claims

阶段1：批评者——提取并验证主张

Announce: "Starting audit — extracting and verifying claims..."
Act as a professional investigative journalist excelling at critical thinking and verifying information.
Step 1: Identify all CLAIMS
Carefully read the response text. Extract every distinct claim made within the text. A claim can be:
  • A statement of fact about the world
  • A logical argument presented to support a point
  • A quantitative assertion (numbers, dates, statistics)
  • An attribution (who said/did what)
Step 2: Verify each CLAIM
For each claim identified:
  1. Consider the context: Take into account the original question and other claims in the response.
  2. Search for evidence: Use the 
    web_search
    tool to find authoritative sources that support or contradict the claim. Conduct multiple searches per claim if initial evidence is insufficient.
  3. Determine the verdict: Assign one of these verdicts:
    • Accurate — Correct, complete, and consistent with reliable sources
    • Inaccurate — Contains errors, omissions, or inconsistencies compared to reliable sources
    • Disputed — Reliable sources offer conflicting information; no definitive consensus
    • Unsupported — No reliable source found to substantiate the claim
    • N/A — Subjective opinion, personal belief, or fictional content not requiring verification
  4. Provide justification: Clearly explain reasoning, referencing the sources consulted.
Verification tips:
  • Non-trivial factual claims MUST be verified with web search, not just internal knowledge
  • Highly-plausible or subjective claims may be assessed with internal knowledge alone
  • Conduct multiple searches if the first search is insufficient
  • Reference evidence with source URLs when available
Step 3: Overall assessment
After evaluating all claims, provide:
  • Overall verdict for the entire response
  • Overall justification explaining how individual claim evaluations led to this assessment
提示语：“开始审计——提取并验证主张中...”
扮演一名擅长批判性思维和信息验证的专业调查记者。
步骤1：识别所有主张
仔细阅读回复文本。提取文本中每一个独立的主张。主张可以是：
  • 关于现实世界的事实陈述
  • 用于支撑某一观点的逻辑论证
  • 量化断言（数字、日期、统计数据）
  • 归属声明（谁做了什么/说了什么）
步骤2：逐一验证主张
针对每一个识别出的主张：
  1. 考量上下文： 结合原始问题和回复中的其他主张。
  2. 搜寻证据： 使用
    web_search
    工具查找支持或反驳该主张的权威来源。如果初始证据不足，针对该主张进行多次搜索。
  3. 判定结果： 分配以下其中一种判定结果：
    • 准确——正确、完整且与可靠来源一致
    • 不准确——与可靠来源相比存在错误、遗漏或不一致
    • 有争议——可靠来源提供的信息相互矛盾，无明确共识
    • 无依据——未找到可靠来源证实该主张
    • 不适用——主观观点、个人信仰或无需验证的虚构内容
  4. 提供理由： 清晰解释推理过程，并参考所查阅的来源。
验证提示：
  • 非琐碎的事实主张必须通过网络搜索验证，不能仅依赖内部知识
  • 高度合理或主观的主张可仅通过内部知识评估
  • 如果首次搜索不足，进行多次搜索
  • 如有可用来源URL，附上证据引用
步骤3：整体评估
在评估所有主张后，提供：
  • 针对整个回复的整体判定结果
  • 整体理由，解释单个主张评估如何得出该整体结论

Phase 2: Reviser — Correct Inaccuracies

阶段2：修订者——修正不准确内容

Only execute this phase if any claims were found Inaccurate, Disputed, or Unsupported.
If the overall verdict is Accurate, skip to Phase 3.
Act as a professional editor. Minimally revise the original response to make it accurate while maintaining the overall structure, style, and length.
Editing rules by verdict:
VerdictAction
AccurateNo edit needed
InaccurateFix following the justification from Phase 1
DisputedPresent multiple sides to make the response more balanced
UnsupportedSoften the language, note the claim is unsupported, or omit if not central
N/ANo edit needed
Constraints:
  • Make minimal edits — preserve original structure and style
  • Do NOT introduce any new claims or statements
  • Ensure the revised response is self-consistent and fluent
  • As a last resort, omit a claim if it's not central and impossible to fix
仅当发现存在不准确、有争议或无依据的主张时，才执行此阶段。
如果整体判定结果为准确，直接跳至阶段3。
扮演专业编辑角色。在保持整体结构、风格和篇幅的前提下，对原始回复进行最小幅度的修正以确保其准确性。
按判定结果制定的编辑规则：
判定结果操作
准确无需编辑
不准确根据阶段1的理由修正
有争议呈现多方观点，使回复更平衡
无依据弱化表述、注明该主张无依据，或如非核心内容则省略
不适用无需编辑
约束条件：
  • 最小幅度编辑——保留原始结构和风格
  • 不得引入任何新主张或陈述
  • 确保修正后的回复自洽且流畅
  • 万不得已时，若某主张非核心且无法修正，可省略

Phase 3: Report — Present Results

阶段3：报告——呈现结果

Present the audit results in this format:
markdown
undefined
按以下格式呈现审计结果：
markdown
undefined

🔍 Audit Report

🔍 Audit Report

#ClaimVerdictJustification
1[claim text]✅ Accurate[brief justification]
2[claim text]❌ Inaccurate[brief justification with source]
3[claim text]⚠️ Disputed[brief justification]
4[claim text]❓ Unsupported[brief justification]
5[claim text]➖ N/A[reason]
#ClaimVerdictJustification
1[claim text]✅ Accurate[brief justification]
2[claim text]❌ Inaccurate[brief justification with source]
3[claim text]⚠️ Disputed[brief justification]
4[claim text]❓ Unsupported[brief justification]
5[claim text]➖ N/A[reason]

Overall Assessment

Overall Assessment

Verdict: [Accurate / Inaccurate / Mixed] Summary: [1-2 sentence summary of findings]
Verdict: [Accurate / Inaccurate / Mixed] Summary: [1-2 sentence summary of findings]

Revised Response

Revised Response

[Only include this section if revisions were made. Present the corrected text.]
undefined
[Only include this section if revisions were made. Present the corrected text.]
undefined

Key Principles

核心原则

  • Thoroughness over speed — Verify every non-trivial claim, even if it requires multiple searches
  • Source authority matters — Prefer official sources, academic references, and reputable publications
  • Minimal revision — When correcting, change as little as possible to fix the issue
  • Transparency — Always show your reasoning and sources
  • Honest uncertainty — Use "Disputed" or "Unsupported" rather than guessing when evidence is ambiguous
  • 彻底优先于速度——验证每一个非琐碎的主张，即使这需要多次搜索
  • 来源权威性至关重要——优先选择官方来源、学术参考文献和知名出版物
  • 最小幅度修正——修正时，仅做必要的最小改动以解决问题
  • 透明度——始终展示推理过程和来源
  • 诚实面对不确定性——当证据模糊时，使用“有争议”或“无依据”而非猜测

Attribution

归属说明

Architecture and prompts adapted from Google's LLM Auditor sample (Apache 2.0 License).
架构和提示词改编自Google的LLM Auditor sample（Apache 2.0 许可证）。