ADK Evaluation Guide

Requires:

agents-cli

(

uv tool install google-agents-cli

) — install uv first if needed.

Scaffolded project? If you used

/google-agents-cli-scaffold

, you already have

agents-cli eval run

,

tests/eval/evalsets/

, and

tests/eval/eval_config.json

. Start with

agents-cli eval run

and iterate from there.

Reference Files

references/criteria-guide.md

references/user-simulation.md

references/builtin-tools-eval.md

references/multimodal-eval.md

The Eval-Fix Loop

How to iterate

1. Start small: Begin with 1-2 eval cases, not the full suite
2. Run eval:

   agents-cli eval run

3. Read the scores — identify what failed and why
4. Fix the code — adjust prompts, tool logic, instructions, or the evalset
5. Rerun eval — verify the fix worked
6. Repeat steps 3-5 until the case passes
7. Only then add more eval cases and expand coverage

Shortcuts That Waste Time

temperature=0

What to fix when scores fail

tool_trajectory_avg_score

tool_uses

IN_ORDER

ANY_ORDER

response_match_score

final_response_match_v2

rubric_based

hallucinations_v1

IN_ORDER

ANY_ORDER

rubric_based_tool_use_quality_v1

Choosing the Right Criteria

tool_trajectory_avg_score

response_match_score

final_response_match_v2

rubric_based_final_response_quality_v1

rubric_based_tool_use_quality_v1

hallucinations_v1

safety_v1

hallucinations_v1

safety_v1

references/user-simulation.md

tool_trajectory_avg_score

references/multimodal-eval.md

references/criteria-guide.md

Running Evaluations

# Scaffolded projects — agents-cli:
agents-cli eval run --evalset tests/eval/evalsets/my_evalset.json

# With explicit config file:
agents-cli eval run --evalset tests/eval/evalsets/my_evalset.json --config tests/eval/eval_config.json

# Run all evalsets in tests/eval/evalsets/:
agents-cli eval run --all

agents-cli eval run

--evalset PATH

--config PATH

--all

agents-cli eval compare baseline.json candidate.json

Configuration Schema (

eval_config.json

)

Full example

{
  "criteria": {
    "tool_trajectory_avg_score": {
      "threshold": 1.0,
      "match_type": "IN_ORDER"
    },
    "final_response_match_v2": {
      "threshold": 0.8,
      "judge_model_options": {
        "judge_model": "gemini-flash-latest",
        "num_samples": 5
      }
    },
    "rubric_based_final_response_quality_v1": {
      "threshold": 0.8,
      "rubrics": [
        {
          "rubric_id": "professionalism",
          "rubric_content": { "text_property": "The response must be professional and helpful." }
        },
        {
          "rubric_id": "safety",
          "rubric_content": { "text_property": "The agent must NEVER book without asking for confirmation." }
        }
      ]
    }
  }
}

"response_match_score": 0.8

judge_model_options

user_simulator_config

references/criteria-guide.md

EvalSet Schema (

evalset.json

)

{
  "eval_set_id": "my_eval_set",
  "name": "My Eval Set",
  "description": "Tests core capabilities",
  "eval_cases": [
    {
      "eval_id": "search_test",
      "conversation": [
        {
          "invocation_id": "inv_1",
          "user_content": { "parts": [{ "text": "Find a flight to NYC" }] },
          "final_response": {
            "role": "model",
            "parts": [{ "text": "I found a flight for $500. Want to book?" }]
          },
          "intermediate_data": {
            "tool_uses": [
              { "name": "search_flights", "args": { "destination": "NYC" } }
            ],
            "intermediate_responses": [
              ["sub_agent_name", [{ "text": "Found 3 flights to NYC." }]]
            ]
          }
        }
      ],
      "session_input": { "app_name": "my_app", "user_id": "user_1", "state": {} }
    }
  ]
}

• intermediate_data.tool_uses

  — expected tool call trajectory (chronological order)

• intermediate_data.intermediate_responses

  — expected sub-agent responses (for multi-agent systems)

• session_input.state

  — initial session state (overrides Python-level initialization)

• conversation_scenario

  — alternative to

  conversation

  for user simulation (see

  references/user-simulation.md

  )

Common Gotchas

The Proactivity Trajectory Gap

google_search

save_preferences

tool_trajectory_avg_score

EXACT

1. Use

   IN_ORDER

   or

   ANY_ORDER

   match type — tolerates extra tool calls between expected ones
2. Include ALL tools the agent might call in your expected trajectory
3. Use

   rubric_based_tool_use_quality_v1

   instead of trajectory matching
4. Add strict stop instructions: "Stop after calling save_preferences. Do NOT search."

Multi-turn conversations require tool_uses for ALL turns

tool_trajectory_avg_score

{
  "conversation": [
    {
      "invocation_id": "inv_1",
      "user_content": { "parts": [{"text": "Find me a flight from NYC to London"}] },
      "intermediate_data": {
        "tool_uses": [
          { "name": "search_flights", "args": {"origin": "NYC", "destination": "LON"} }
        ]
      }
    },
    {
      "invocation_id": "inv_2",
      "user_content": { "parts": [{"text": "Book the first option"}] },
      "final_response": { "role": "model", "parts": [{"text": "Booking confirmed!"}] },
      "intermediate_data": {
        "tool_uses": [
          { "name": "book_flight", "args": {"flight_id": "1"} }
        ]
      }
    }
  ]
}

App name must match directory name

App

name

# CORRECT - matches the "app" directory
app = App(root_agent=root_agent, name="app")

# WRONG - causes "Session not found" errors
app = App(root_agent=root_agent, name="flight_booking_assistant")

The

before_agent_callback

Pattern (State Initialization)

KeyError

async def initialize_state(callback_context: CallbackContext) -> None:
    state = callback_context.state
    if "user_preferences" not in state:
        state["user_preferences"] = {}

root_agent = Agent(
    name="my_agent",
    before_agent_callback=initialize_state,
    instruction="Based on preferences: {user_preferences}...",
)

Eval-State Overrides (Type Mismatch Danger)

session_input.state

.append()

"state": { "feedback_history": "" }

"state": { "feedback_history": [] }

Model thinking mode may bypass tools

tool_config

mode="ANY"

Common Eval Failure Causes

tool_uses

hallucinations_v1

name

temperature=0

tool_trajectory_avg_score

google_search

references/builtin-tools-eval.md

IN_ORDER

ANY_ORDER

get_text_from_content()

references/multimodal-eval.md

Deep Dive: ADK Docs

• Evaluation overview:

  https://adk.dev/evaluate/index.md

• Criteria reference:

  https://adk.dev/evaluate/criteria/index.md

• User simulation:

  https://adk.dev/evaluate/user-sim/index.md

Debugging Example

1. Check if agent uses

   google_search

   — if so, see

   references/builtin-tools-eval.md

2. Check if using

   EXACT

   match and agent calls extra tools — try

   IN_ORDER

3. Compare expected

   tool_uses

   in evalset with actual agent behavior
4. Fix mismatch (update evalset or agent instructions)

Proving Your Work

• After running eval: Paste the scores table output so the user can see exactly what passed and failed.
• After fixing a failure: Show before/after scores for the specific case you fixed, and confirm no other cases regressed.
• Before declaring "eval passes": Confirm ALL cases pass, not just the one you were working on. Run

  agents-cli eval run

  (or

  agents-cli eval run --all

  ) one final time.
• Before moving to deploy: Show the final

  agents-cli eval run

  output with all cases above threshold. This is the gate — no exceptions.

Related Skills

• /google-agents-cli-workflow

  — Development workflow and the spec-driven build-evaluate-deploy lifecycle

• /google-agents-cli-adk-code

  — ADK Python API quick reference for writing agent code

• /google-agents-cli-scaffold

  — Project creation and enhancement with

  agents-cli scaffold create

  /

  scaffold enhance

• /google-agents-cli-deploy

  — Deployment targets, CI/CD pipelines, and production workflows

• /google-agents-cli-observability

  — Cloud Trace, logging, and monitoring for debugging agent behavior