1. If the user passed

   --backend pup

   anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether

   mcp__datadog-llmo-mcp__get_llmobs_experiment_summary

   appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether

   pup

   is executable: run

   pup --version

   via Bash. A JSON response containing

   "version"

   confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:

   "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (

   claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs'

   ) or install pup."

--backend pup

• Invoke via Bash:

  pup llm-obs <subcommand> [flags]

• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in

  [{"type": "text", "text": "<json>"}]

  ).
• If pup returns an auth error, tell the user to run

  pup auth login

  and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).

3a9f1c2b

telemetry.intent

skill:llm-obs-experiment-analyzer[<inv_id>] — 

skill:llm-obs-experiment-analyzer:start[<inv_id>] — 

:start

skill:llm-obs-experiment-analyzer:start[3a9f1c2b] — Phase 1: get experiment summary to orient analysis

1. Extract one or two UUID-format strings as experiment IDs (first = baseline/primary, second = candidate).
2. Extract

   --output agent|file|notebook

   flag if present.
3. The remaining text (after IDs and flags) is the question, if any.

• 2 IDs + question → Comparative Q&A
• 2 IDs, no question → Comparative Exploratory
• 1 ID + question → Single Q&A
• 1 ID, no question → Single Exploratory

--output

AskUserQuestion

1. Analysis type: If no question text was provided in the arguments, ask whether the user wants exploratory analysis or has a specific question. Skip this call only if the user's intent is already clear from context (e.g. they typed a question alongside the IDs).
2. Output destination: If

   --output

   was not specified, ask where to deliver the report (chat, file, or Datadog notebook). Always ask this as its own standalone call.

1. Agent (default): Display the full report in the conversation.
2. File: Before starting, propose a path:

   evals/reports/YYYY-MM-DD-<experiment-slug>-analysis.md

   Present it to the user and let them confirm or adjust. Then proceed.
3. Notebook: Use

   mcp__datadog-mcp-core__create_datadog_notebook

   at the end. In pup mode, use

   pup notebooks create --title "TITLE" --file /tmp/nb_cells.json

   instead (see Tool Reference). If neither MCP nor pup is available, output these setup instructions instead of failing:

   To enable Datadog notebook export, add the MCP server:
     claude mcp add --transport http datadog-mcp https://mcp.datadoghq.com/api/unstable/mcp-server
   See: https://docs.datadoghq.com/bits_ai/mcp_server/setup/

   Then ask: "Would you like to fall back to file or agent output instead?" See Phase 5 for full notebook call details.

AskUserQuestion

get_llmobs_experiment_summary

• Scale: total samples and error count for each
• Metrics: which metrics exist in each; which are shared
• Dimensions: which dimensions exist in each; which are shared
• Immediate red flags (errors present, missing metrics, sparse data)
• Obvious improvements or regressions visible at the summary level

error_count > 0

get_llmobs_experiment_dimension_values

error_type

asyncio.exceptions.cancellederror

get_llmobs_experiment_summary

• Total samples, and error count (with

  error_type

  breakdown if non-zero)
• Available metrics grouped by

  metric_type

  as returned by the summary (

  score

  ,

  boolean

  ,

  categorical

  ). Do not infer semantic groupings or categories from label name patterns or prefixes — the label string is not a reliable signal for what a metric measures.
• Classify each metric using the statistics already returned by the summary (mean, min, max). Do not infer metric meaning from label names or prefixes. Use the classifications defined in Phase 1.5 when referencing metrics throughout the report.
• Available dimensions for segmentation
• Any immediate red flags

AskUserQuestion

always_zero

max == 0

perfect

min == 1

saturated

mean ≥ 0.99

min < 1

struggling

mean < 0.70

interesting

0.70 ≤ mean < 0.99

min < max

Found N metrics. Full breakdown:

| Metric | Mean | Class |
|--------|------|-------|
| <label> | <mean> | ⚠️ Struggling |
| <label> | <mean> | Interesting |
| <label> | <mean> | Saturated |
| <label> | 1.000 | Perfect (no signal) |
| <label> | 0.000 | Always zero (disabled?) |

always_zero

• "Struggling metrics (N) — Recommended": only shown if N ≥ 1. Description explicitly lists each metric label and its mean (e.g. "

  open_answer

  0.33,

  c_permanence

  0.68"). This is the grounded suggestion — based on observed pass rates, not label names. If there are no struggling metrics, replace this option with "Lowest-performing metrics (N)" covering the bottom N by mean.
• "Interesting + struggling (N)": shown only if there are interesting-class metrics in addition to struggling ones. Description lists them with means.
• "All metrics (N)": always shown. Note in the description that always-zero and perfect metrics add noise but are included.
• "A specific metric": always shown. Description says: "Choose one from the table printed above."

AskUserQuestion

mean: X.XX — class

always_zero

perfect

struggling

interesting

• If the user accepts "all", proceed with all metrics (including constant ones, but note their low signal value).
• If the user selects a grouping or a specific metric, restrict all analysis in Phases 2–4 strictly to that selection. Do not call

  get_llmobs_experiment_metric_values

  for any metric outside the selection.

• Segments where the candidate outperforms the baseline
• Segments where the candidate regresses
• Error types present in one but rare in the other
• Distribution shifts or coverage gaps
• Tradeoffs (e.g., higher recall, lower precision)

• Base URL:

  https://app.datadoghq.com/llm/experiment-comparison

• Required params:

  baselineExperimentId

  ,

  experimentIds

  (candidate%2Cbaseline),

  tableView=all

• Optional (include if discoverable):

  project

  ,

  compareDatasetId

  ,

  selectedEvaluation

• selectedEvaluation

  priority: overall/overall_score/rubric metric → primary metric → first shared metric
• Generate 2–4 links: primary comparison, regression view, calibration view (if applicable), worst-segment view (only if supported — never fabricate filters)

• Worst-performing segments (by metric × dimension)
• Any segments with surprising pass rates
• Overall pass rates and variance

• https://app.datadoghq.com/llm/experiments/{experiment_id}

get_llmobs_experiment_metric_values

• Per-segment and per-class delta analysis (comparative) or pass-rate analysis (single)
• Error overlap vs. unique failure mode analysis
• Sampling and qualitative inspection of representative failures (2–5 per issue)
• Clustered error theme analysis

• Prefer cheap, high-signal analyses first; do not stop early.
• Mask or redact PII in all outputs.
• Avoid destructive actions.

https://app.datadoghq.com/llm/experiments/{experiment_id}?selectedTab=overview&sp=[{"p":{"experimentId":"{experiment_id}","spanId":"{span_id}"},"i":"experiment-details"}]&spanId={span_id}

https://app.datadoghq.com/llm/experiments/{experiment_id}?selectedTab=overview&filter[{dimension}]={value}

• Clear wins where the candidate improves on the baseline
• Clear regressions or risks the candidate introduces
• Neutral or unchanged areas
• Root-cause hypotheses (1–4), tied to evidence
• Prioritized recommendations: ship as-is / block / gate by segment / combine behaviors

• Direct answer to the question with a clear verdict
• Supporting evidence (metrics, percentages, event examples)
• Relevant context (e.g., caveats, data limitations)

• Overall performance assessment
• Worst-performing segments and root causes
• Hypotheses for why failures occur
• Recommended next experiments

• Direct answer to the question with a clear verdict
• Supporting evidence from the experiment data

## Summary & Recommendations

## Synthesis

<path>

mcp__datadog-mcp-core__create_datadog_notebook

• name

  (by mode):

  Mode	Name
  Comparative Exploratory	Experiment Analysis: {baseline_short} (Baseline) vs {candidate_short} (Candidate) — YYYY-MM-DD
  Comparative Q&A	Experiment Q&A: {baseline_short} vs {candidate_short} — YYYY-MM-DD
  Single Exploratory	Experiment Analysis: {experiment_short} — YYYY-MM-DD
  Single Q&A	Experiment Q&A: {experiment_short} — YYYY-MM-DD
  where short = first 8 characters of the UUID.

• cells

  : one cell per report section — do NOT put the entire report in a single cell. Structure:
  • Cell 1 — Summary & Recommendations containing three

    ###

    subheaders: Experiment (link + executive summary), Key Findings (bullets), Recommendations (numbered list) — always present, always first, never skipped regardless of experiment complexity
  • Cell 2 — Orientation table
  • Cell 3 — What Changed (comparative modes only; omit for single)
  • Cell 4 — Signals / Answer to Question
  • Cells 5…N — one cell per Deep Dive Finding
  • Cell N+1 — Synthesis (issue tally, Overall Performance Assessment, Worst-Performing Segments, Root Cause Hypothesis, Recommended Next Experiments) — always present, always second-to-last
  • Cell N+2 — UI Links
  Omit the

  # Experiment Analysis Report

  top-level heading from all cells — it is already shown as the notebook title.

• time

  :

  { "live_span": "1h" }

"Report exported to notebook: <url>"

1. [Specific question derived from actual findings — e.g., "Want me to dig deeper into why the SQL scenarios regressed in the candidate?"]
2. [Another specific follow-up — e.g., "Should I compare error patterns between the two failing clusters?"]
3. [A third option if relevant]

Stay active after the report. Answer follow-up questions using the same MCP tools, referencing findings already gathered. Do not re-run analyses you've already performed unless new questions require it.

---

• Experiment IDs: Wherever a full experiment UUID appears, render it as a Markdown link to

  https://app.datadoghq.com/llm/experiments/{full_uuid}

  .
• Comparative table column headers: In the Orientation table and in every subsequent table that has Baseline/Candidate columns, wrap the entire column header as a link — not just the short ID. Format:

  [Baseline \

  {short_id}`]({baseline_url})

  and

  Candidate `{short_id}``. This makes the full header cell clickable, not just the ID portion.

undefined

Question: {original question text} (Q&A modes only — omit for Exploratory modes)

• {Finding 1}: one-line description with numbers (e.g. "+4.2pp on

  tool_accuracy

  across all segments")
• {Finding 2}: one-line description
• {Finding 3} (if present): one-line description [For Q&A modes: one-line verdict bullet + one-line rationale bullet]

1. {Recommendation 1}: specific, actionable next step tied to a finding
2. {Recommendation 2}: specific, actionable next step
3. {Recommendation 3} (if present): specific, actionable next step [Omit this subsection for Q&A modes unless a clear action follows from the answer.]

[dimension=value]

Agent

Evaluator/Dataset

Ambiguous

• [Span link]: [input → output → expected, what went wrong]

• Do not assume anything about the experiment (model, task, metrics, schema, dimensions). Infer everything by inspecting the data.
• Ground all conclusions in specific evidence: event IDs, counts, percentages.
• Show math: include counts and rates, not just qualitative claims.
• Avoid speculative explanations not supported by observed evidence.
• Mask or redact PII in all user-visible output.

create_datadog_notebook(name, cells, ...)

pup notebooks create --title "TITLE" --file /tmp/nb_cells.json

pup notebooks create --help

[{"attributes": {"definition": {"type": "markdown", "text": "## Section\n\nContent."}}, "type": "notebook_cells"}]

• Never show internal tool calls, schemas, or implementation details to the user.