1. If the user passed

   --backend pup

   anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether

   mcp__datadog-llmo-mcp__list_llmobs_evals

   appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether

   pup

   is executable: run

   pup --version

   via Bash. A JSON response containing

   "version"

   confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:

   "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (

   claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs'

   ) or install pup."

--backend pup

• Invoke via Bash:

  pup llm-obs <subcommand> [flags]

• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in

  [{"type": "text", "text": "<json>"}]

  ).
• If pup returns an auth error, tell the user to run

  pup auth login

  and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (

  1h

  ,

  7d

  ,

  30m

  ) and RFC3339 timestamps. Do not use

  now-

  -prefixed strings — strip the prefix when converting from a skill

  --timeframe

  argument:

  now-7d

  →

  7d

  ,

  now-24h

  →

  24h

  ,

  now-30d

  →

  30d

  .

• --summary

  on

  pup llm-obs spans search

  strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

3a9f1c2b

telemetry.intent

skill:llm-obs-trace-rca[<inv_id>] — 

skill:llm-obs-trace-rca:start[<inv_id>] — 

:start

skill:llm-obs-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode

1. get_llmobs_span_details

   : Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a single message.

2. get_llmobs_span_content

   : Each call is independent — always issue ALL in a single message.

3. get_llmobs_trace

   /

   find_llmobs_error_spans

   /

   get_llmobs_agent_loop

   : Parallelize across different traces in a single message.
4. Pipeline parallelism: Start

   get_llmobs_span_details

   for page 1 results immediately — don't wait to collect all pages.

# Session Classification Summary

trace_id

trace_id=

session_id=

verdict

failure_mode

none

detail

app_type

# Session Classification Summary

Root span kind: agent

LLM

no

partial

• < 5 entries → note low confidence, proceed anyway.
• Empty → report "No failures found in the classification output" and stop.

undefined

• get_llmobs_eval_aggregate_stats(eval_name, from, to)

• get_llmobs_evaluator(eval_name)

• total_count == 0

  → Note "no data." Skip this eval (or pivot to Generic if it's the only one).
• Boolean

  pass_rate == 1.0

  → Note "100% pass." Skip unless it's the only eval.
• Boolean with failures → Note counts and pass_rate. Continue.
• Score with assessment criteria → Note distribution and pass/fail counts. Continue.
• Score WITHOUT assessment criteria → Infer failures: bottom quartile, or below median if bimodal. Label as "inferred failures" in report.
• Categorical with assessment criteria → Note top_values and pass/fail. Continue.
• Categorical WITHOUT assessment criteria → Infer from context (e.g., "error", "incomplete", "off_topic" are likely failures). Ask user if genuinely ambiguous.

• Config returned (custom) → Store

  prompt_template

  ,

  assessment_criteria

  ,

  parsing_type

  ,

  output_schema

  .
• Config nil (OOTB) → Note prompt is not inspectable.

foo-boolean

foo-score

1. search_llmobs_spans(query="@evaluations.custom.<eval_name>:*", from, limit=50)

   . When multi-valued, issue one search per eval in parallel — merge result sets, dedup by

   (trace_id, span_id)

   .
2. Paginate until ≥15–20 failures OR no more pages. Cap at 200 spans total.

3. get_llmobs_span_details

   per trace_id batch (follow Parallelization Rules).
4. Extract per row: assessment, value, reasoning, span_id, trace_id, span_kind, content_info.
5. Separate into pass/fail buckets using thresholds from Step 1a.

@evaluations.custom.<eval_name>:*

get_llmobs_eval_aggregate_stats

total_count > 0

get_llmobs_evaluator

get_llmobs_span_content(field="output")

find_llmobs_error_spans(trace_id)

• Error message and stack trace
• Origin span kind and name
• Whether errors propagate from children to parents (cascade) or are isolated

• Top decile by

  duration

  (latency outliers)
• Agent spans with >N tool/LLM iterations (long-running loops)
• Retrieval spans returning 0 documents (RAG miss)
• Workflow spans whose child set is missing an expected step (compare against median child layout)
• Token efficiency: Check if

  non_cached_input_tokens ≈ input_tokens

  across LLM spans. If the app has stable system prompts (>1k tokens) and cache hit rate is 0%, flag as High severity — enabling

  cache_control: ephemeral

  on the system prompt would cut input token costs by 60–90%

content_info

span_kind

content_info

messages

messages[0]

content_info

documents

agent

get_llmobs_agent_loop

messages.count > 10

content_info

metadata

LLM Experiments traces: If root spans have

span_kind: experiment

and carry

input

,

output

, and

expected_output

structured fields, you are looking at a Datadog LLM Experiments trace. Each span represents one dataset record run. Read quality signal from the root span's

input

/

output

/

expected_output

fields via

get_llmobs_span_content

— not from LLM sub-span messages, which may contain stub or placeholder content. Evaluations attached to experiment spans are computed by the Experiments framework at run time and may not be registered as online Datadog evaluators (

get_llmobs_evaluator

will return 404 for them).

llm

agent

tool

workflow

retrieval

• Eval Signal: judge assessment + reasoning from

  get_llmobs_span_details

• Error Signal: error message + stack trace excerpt from

  find_llmobs_error_spans

• Generic: one-line description of the structural anomaly that flagged the row

• < 15 problematic rows: Combine Phases 2 and 3 into one pass. Still produce the checkpoint.
• > 80% share the same reasoning/error/symptom: Skip to Phase 4 with the dominant pattern. Still output checkpoint.
• > 50 problematic rows: Sample ~50, build taxonomy, then spot-check 10–15 more.

1. Use per-row signal from Phase 1 — do NOT re-fetch. Only call

   get_llmobs_span_content(field="input"/"output")

   for spans where the reasoning is insufficient (generic, empty, or just a stack trace with no app context).
2. If eval config is loaded (Eval Signal), distinguish early:
   • App failures: Output genuinely violates the eval's criteria
   • Eval failures: Output seems reasonable but eval criteria are too strict/ambiguous
3. Each pattern must be specific: "Agent called search instead of calculator for price computation" — NOT "tool issue."

1. Merge: Categories with < 3 occurrences → parent category or drop as noise.
2. Split: Categories with > 30% of failures → more specific sub-categories. Pull additional span content if needed.
3. Validate: 2–3 representative examples per category confirm the label fits.
4. Rank:

   priority = count × severity

   (severity: high / medium / low).

• get_llmobs_trace(trace_id)

  — span hierarchy; locate the symptom span and its parent/siblings/children

• find_llmobs_error_spans(trace_id)

  — check for runtime errors anywhere in the trace

get_llmobs_trace

get_llmobs_span_details

start_ms

llm

• get_llmobs_span_content(field="messages", path="$.messages[0]")

  on symptom span — system prompt

• get_llmobs_span_content(field="messages")

  on symptom span — full context received

• get_llmobs_span_content(field="documents")

  on sibling retrieval spans (if any)

• get_llmobs_span_content(field="input")

  on sibling tool spans (if any)

• get_llmobs_span_content(field="messages", path="$.messages[0]")

  on parent agent/workflow span

agent

• get_llmobs_agent_loop(trace_id, span_id)

  — full decision timeline (try first; if it returns 0 iterations, use the fallback below)

• get_llmobs_span_details

  on child spans — sort by

  start_ms

  to reconstruct the execution timeline

• get_llmobs_span_content(field="input"/"output")

  on child spans that look wrong

get_llmobs_agent_loop

get_llmobs_span_details

start_ms

span_kind

get_llmobs_agent_loop

tool

• get_llmobs_span_content(field="input")

  on symptom span — what parameters was it called with?

• get_llmobs_span_content(field="messages")

  on parent LLM span — did the LLM construct the call correctly?

workflow

• get_llmobs_span_details

  on all child spans — find which step first deviated

• get_llmobs_span_content(field="input"/"output")

  on the deviating child

• get_llmobs_span_content(field="metadata")

  on the symptom span — clustering signals (prompt version, feature flags)

1. Symptom — what the signal flagged (eval reasoning, error message, anomaly note). The signal only saw one span in isolation — its reasoning may be shallow.
2. Trace context — what surrounding spans reveal (parent instructions, sibling data, child decisions).
3. Root cause — the specific span and decision point where the failure originated. Often NOT the symptom span itself.

find_llmobs_error_spans

1. Save the report to

   llm-obs-rca-{ml_app}-{date}.md

2. Apply fixes (if codebase is available)
3. Deeper investigation of remaining categories
4. Export to a Datadog notebook — in pup mode, use

   pup notebooks create

   to create the notebook and

   pup notebooks edit NOTEBOOK_ID --file /tmp/nb_cells.json

   to append sections (see Tool Reference)
5. Re-run on an expanded time range (e.g.

   now-7d

   if current window was

   now-24h

   )

1. Call

   mcp__datadog-mcp-core__create_datadog_notebook

   with:

   • name

     :

     LLM Obs RCA: {ml_app} ({mode}) — YYYY-MM-DD

   • type

     :

     report

   • time_span

     :

     1w

   • cells

     : one cell per section (see Notebook Cell Structure below)
2. If the MCP call fails, inspect the error before giving up:
   • Auth / permission error (401, 403) → stop and tell the user.
   • Field validation error (error message names a specific field) → fix that field and retry the MCP call once.
   • Any other error (binding, serialization, unexpected response) → fall back to pup:
     • Write the notebook payload to

       /tmp/nb_rca_{ml_app}.json

       as a full API envelope:

       {"data": {"attributes": {"name": "...", "time": {...}, "cells": [...]}, "type": "notebooks"}}

     • Run

       pup notebooks create --file /tmp/nb_rca_{ml_app}.json

     • If pup is not available either, render the full notebook content as markdown in chat so the user has it.
3. After successful creation by either method, output the URL on its own line:

   RCA report exported to notebook: <url>

/eval-bootstrap

• No triple-backtick code blocks — use blockquotes (

  >

  ) for prompts/rubrics, inline code (

  `

  ) for short values
• Evidence as tables — not bullet lists
• Tool inputs as tables — Argument | Wrong value passed | Correct approach
• Action plan as a table — Priority | Action | Confidence | Impact

• {Finding 1} (~{pct}%): one-line root cause description
• {Finding 2} (~{pct}%): one-line root cause description
• {Finding 3} (if present): one-line root cause description

• {Recommendation 1}: specific next step tied to Finding 1
• {Recommendation 2}: specific next step tied to Finding 2

**Mode-appropriate metric rows:**

Eval Signal:

{eval_name}

Error Signal:

{type}

{span_name}

Generic:

---

query

"monitor_id:123 group_status:alert"

"monitor_id:123"

"{quoted judge reasoning}"

---

• Ground in evidence: Every claim references span IDs with clickable trace links:

  [Trace {first_8}...](https://app.datadoghq.com/llm/traces?query=trace_id:{full_32_char_id})

  .
• Root cause over symptom: "System prompt doesn't specify date format" not "model gave wrong answer."
• Show your math: "47 failures (34%)" not "many failures."
• Honest about uncertainty: < 5 examples = tentative. Flag it.
• Anonymize PII: No emails or names. User/org IDs are fine.
• MCP result parsing safety: Before writing any script that iterates over MCP tool results, inspect the raw structure first — check top-level keys and whether the payload is nested inside a content block (e.g.

  [{'type': 'text', 'text': '<json>'}]

  ). Extract and

  json.loads()

  the inner payload if needed. Never assume MCP results are bare dicts or lists.