Reuse working tracing for investigation or evaluation.
For OSS or SageMaker, keep the configured backend and proceed to Step 2.
-
Fetch a persisted application trace, not a local span or old trace.
-
Set a 60-second wall-clock process deadline for readback, including flush, SDK retries, requests, and backoff.
Terminate at expiry unless the user explicitly requests deeper troubleshooting.
-
Tool yields/thread-wait timeouts do not cancel calls. Without cancellation, report backend verification as unperformed.
-
Flush once (
mlflow.flush_trace_async_logging()
/
await mlflow.flushTraces()
); fetch the saved ID.
-
Without an ID, search once in the configured experiment with run-time bounds,
, and spans included.
-
Every UC search needs
; result limits do not bound scans. Python
has no
keyword.
For Python readback, adapt this recipe to the saved run values and enforce the deadline externally:
python
mlflow.flush_trace_async_logging()
if trace_id:
trace = mlflow.get_trace(trace_id)
else:
traces = mlflow.search_traces(
locations=[experiment_id],
filter_string=(f"trace.timestamp_ms >= {run_start_ms} "
f"AND trace.timestamp_ms <= {run_end_ms}"),
max_results=1, return_type="list", include_spans=True,
)
trace = traces[0] if traces else None
assert trace is not None, "No persisted trace read back"
assert run_start_ms <= trace.info.request_time <= run_end_ms
assert trace.data.spans, "No spans captured"
-
Inspect expected root/graph, model, retrieval, and tool spans and actual I/O. Confirm the application's request, not scorer or assistant telemetry.
-
Check experiment/location from returned metadata, or the bound UC location if no experiment ID exists.
-
Reuse returned spans; stop after success.
-
Retry at most once after a concrete quick fix within the original budget.
-
Stop on missing provider keys, auth/permission errors, warehouse timeouts, or
/429.
-
No parallel searches, repeated agent runs, direct UC SQL, raw REST, alternate credentials, or warehouse fallbacks.
-
Report operation/error, captured ID, successes, and unverified results. Request the missing prerequisite, not secrets.
-
Keep instrumentation changes if backend verification is blocked; pause evaluation until tracing is verified.
Return the verified trace ID/link and observed spans. For Databricks use the verified trace URL rules, not an experiment-only link. Do not label an unverified link as proof.
-
Deployment, async export, sampling: production.
-
Async/thread context, redaction, caught errors or links: advanced patterns.
-
Cross-service parent propagation: distributed tracing.
-
User ratings tied to traces: feedback collection.
-
Unreadable non-chat root previews: root span previews.
-
Explaining captured behavior:
; implementing a requested behavior fix:
, carrying the trace evidence. These are optional follow-on tasks.
-
Other integrations:
automatic tracing docs. Check installed signatures/types.
-
Missing/moved topic:
llms.txt. Docs unavailable: use local recipes and installed API introspection; stop if correctness remains uncertain.