Loading...
Loading...
Found 2 Skills
Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace <trace-id> [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace <id> with <change>"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing evaluators, root-causing failed traces, or RUM/HTTP session replay.
Bootstrap evaluators from production traces — emit SDK code, a framework-agnostic JSON spec, or publish online LLM-judge evaluators directly to Datadog. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.