ADK Observability Guide

Scaffolded project? Cloud Trace and prompt-response logging are pre-configured by Terraform. See
references/cloud-trace-and-logging.md
for infrastructure details, env vars, and verification commands.
No scaffold? Follow the ADK docs links below for manual setup. For production infrastructure, scaffold with
/adk-scaffold
.

Reference Files

File	Contents
`references/cloud-trace-and-logging.md`	Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally
`references/third-party.md`	Third-party integration setup patterns, trade-offs, and ADK docs links for each provider

Observability Tiers

Choose the right level of observability based on your needs:

Tier	What It Does	Scope	Default State	Best For
Cloud Trace	Distributed tracing — execution flow, latency, errors via OpenTelemetry spans	All templates, all environments	Always enabled	Debugging latency, understanding agent execution flow
Prompt-Response Logging	GenAI interactions exported to GCS, BigQuery, and Cloud Logging	ADK agents only	Disabled locally, enabled when deployed	Auditing LLM interactions, compliance
BigQuery Agent Analytics	Structured agent events (LLM calls, tool use, outcomes) to BigQuery	ADK agents with plugin enabled	Opt-in ( `--bq-analytics` at scaffold time)	Conversational analytics, custom dashboards, LLM-as-judge evals
Third-Party Integrations	External observability platforms (AgentOps, Phoenix, MLflow, etc.)	Any ADK agent	Opt-in, per-provider setup	Team collaboration, specialized visualization, prompt management

Ask the user which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.

Cloud Trace

ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.

Span Hierarchy

invocation
  └── agent_run (one per agent in the chain)
        ├── call_llm (model request/response)
        └── execute_tool (tool execution)

Setup by Deployment Type

Deployment	Setup
Agent Engine	Automatic — traces are exported to Cloud Trace by default
Cloud Run (scaffolded)	Automatic — `otel_to_cloud=True` in the FastAPI app
Cloud Run (manual)	Configure OpenTelemetry exporter in your app
Local dev	Works with `make playground` ; traces visible in Cloud Console

View traces: Cloud Console → Trace → Trace explorer

For detailed setup instructions (Agent Engine CLI/SDK, Cloud Run, custom deployments), fetch the ADK docs:

WebFetch: https://google.github.io/adk-docs/integrations/cloud-trace/index.md

Prompt-Response Logging

Captures GenAI interactions (model name, tokens, timing) and exports to GCS (JSONL), BigQuery (external tables), and Cloud Logging (dedicated bucket).

Privacy Modes

Prompt-response logging is privacy-preserving by default — only metadata is logged. Controlled by

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT

Value	Behavior
`false`	Logging disabled
`NO_CONTENT`	Enabled, metadata only — tokens, model name, timing (default in deployed environments)
`true`	Enabled with full prompt/response content (not recommended for production)

For Agent Engine: the platform requires

true

during deployment, but the app overrides to

NO_CONTENT

at runtime.

Behavior by Environment

Environment	Prompt-Response Logging	Why
Local dev ( `make playground` )	Disabled	No `LOGS_BUCKET_NAME` set
Dev (Terraform deployed)	Enabled ( `NO_CONTENT` )	Terraform sets env vars
Staging / Production	Enabled ( `NO_CONTENT` )	Terraform sets env vars

To enable locally, set

LOGS_BUCKET_NAME

and

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT

before running

make playground

To disable in a deployed environment, set

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false

deployment/terraform/service.tf

and re-apply.

For scaffolded project infrastructure details (Terraform resources, env vars, verification), see

references/cloud-trace-and-logging.md

For ADK logging docs (log levels, configuration, debugging):

WebFetch: https://google.github.io/adk-docs/observability/logging/index.md

BigQuery Agent Analytics Plugin

An optional plugin that logs structured agent events directly to BigQuery via the Storage Write API. Enables:

Conversational analytics — session flows, user interaction patterns
LLM-as-judge evals — structured data for evaluation pipelines
Custom dashboards — Looker Studio integration
Tool provenance tracking — LOCAL, MCP, SUB_AGENT, A2A, TRANSFER_AGENT

Enabling

Method	How
At scaffold time	`uvx agent-starter-pack create . --bq-analytics`
Post-scaffold	Add the plugin manually to `app/agent.py` (see ADK docs)

Infrastructure (BigQuery dataset, GCS offloading) is provisioned automatically by Terraform when enabled at scaffold time.

Key Features

Auto-schema upgrade (new fields added without migration)
GCS offloading for multimodal content (images, audio)
Distributed tracing via OpenTelemetry span context
SQL-queryable event log for all agent interactions

For full schema, SQL query examples, and Looker Studio setup:

WebFetch: https://google.github.io/adk-docs/integrations/bigquery-agent-analytics/index.md

Third-Party Integrations

ADK supports six third-party observability platforms. Each uses OpenTelemetry or custom instrumentation to capture agent behavior.

Platform	Key Differentiator	Setup Complexity	Self-Hosted Option
AgentOps	Session replays, 2-line setup, replaces native telemetry	Minimal	No (SaaS)
Phoenix	Open-source, custom evaluators, experiment testing	Low	Yes
MLflow	OTel traces to MLflow Tracking Server, span tree visualization	Medium (needs SQL backend)	Yes
Monocle	1-call setup, VS Code Gantt chart visualizer	Minimal	Yes (local files)
Weave	W&B platform, team collaboration, timeline views	Low	No (SaaS)
Freeplay	Prompt management + evals + observability in one platform	Low	No (SaaS)

Ask the user which platform they prefer — present the trade-offs and let them choose. For setup details on each, see

references/third-party.md

Troubleshooting

Issue	Solution
No traces in Cloud Trace	Verify `otel_to_cloud=True` in FastAPI app; check service account has `cloudtrace.agent` role
Prompt-response data not appearing	Check `LOGS_BUCKET_NAME` is set; verify SA has `storage.objectCreator` on the bucket; check app logs for telemetry setup warnings
Privacy mode misconfigured	Check `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` value — use `NO_CONTENT` for metadata-only, `false` to disable
BigQuery Analytics not logging	Verify plugin is configured in `app/agent.py` ; check `BQ_ANALYTICS_DATASET_ID` env var is set
Third-party integration not capturing spans	Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry
Traces missing tool spans	Tool execution spans appear under `execute_tool` — check trace explorer filters
High telemetry costs	Switch to `NO_CONTENT` mode; reduce BigQuery retention; disable unused tiers

Deep Dive: ADK Docs (WebFetch URLs)

For detailed documentation beyond what this skill covers, fetch these pages:

Topic	URL
Observability overview	`https://google.github.io/adk-docs/observability/index.md`
Agent activity logging	`https://google.github.io/adk-docs/observability/logging/index.md`
Cloud Trace integration	`https://google.github.io/adk-docs/integrations/cloud-trace/index.md`
BigQuery Agent Analytics	`https://google.github.io/adk-docs/integrations/bigquery-agent-analytics/index.md`
AgentOps	`https://google.github.io/adk-docs/integrations/agentops/index.md`
Phoenix (Arize)	`https://google.github.io/adk-docs/integrations/phoenix/index.md`
MLflow tracing	`https://google.github.io/adk-docs/integrations/mlflow/index.md`
Monocle	`https://google.github.io/adk-docs/integrations/monocle/index.md`
W&B Weave	`https://google.github.io/adk-docs/integrations/weave/index.md`
Freeplay	`https://google.github.io/adk-docs/integrations/freeplay/index.md`

adk-observability-guide

NPX Install

SKILL.md Content