ADK Evals Skill

What are Evals?

adk dev

When to Use This Skill

• Writing evals — file format, assertions, turn types, setup
• Running evals — CLI commands, filtering, output interpretation
• Testing specific primitives — how to test actions, tools, workflows, conversations, tables, state
• The testing loop — write → run → inspect traces → iterate
• CI integration — exit codes,

  --format json

  flag, tagging strategies
• Eval configuration — idleTimeout, judgePassThreshold, judgeModel

• "How do I write an eval?"
• "How do I test my workflow?"
• "How do I assert that a tool was called with specific params?"
• "My eval is failing, how do I debug it?"
• "How do I test that the bot stays silent?"
• "How do I run evals in CI?"
• "How do I seed state before an eval?"
• "How do I trigger a workflow in an eval?"

Available Documentation

references/eval-format.md

references/testing-workflow.md

references/test-patterns.md

How to Answer

1. Writing an eval → Read

   eval-format.md

   for structure and assertions
2. Running evals → Read

   testing-workflow.md

   for CLI commands and output
3. Testing a specific primitive → Read

   test-patterns.md

   for the relevant section
4. Debugging a failure → Combine

   testing-workflow.md

   (inspect traces) +

   eval-format.md

   (check assertion syntax)

Quick Reference

Eval file structure

import { Eval } from '@botpress/adk'

export default new Eval({
  name: 'greeting',
  type: 'regression',
  tags: ['basic'],

  setup: {
    state: { bot: { welcomeSent: false } },
    workflow: { trigger: 'onboarding', input: { userId: 'test-1' } },
  },

  conversation: [
    {
      user: 'Hi!',
      assert: {
        response: [
          { not_contains: 'error' },
          { llm_judge: 'Response is friendly and offers to help' },
        ],
        tools: [{ not_called: 'createTicket' }],
        state: [{ path: 'conversation.greeted', equals: true }],
      },
    },
  ],

  outcome: {
    state: [{ path: 'conversation.greeted', equals: true }],
  },

  options: {
    idleTimeout: 20000,
    judgePassThreshold: 4,
  },
})

Turn types

user: 'message'

event: { type, payload }

expectSilence: true

Assertion categories

response

tools

state

tables

workflow

timing

CLI commands

adk evals                        # run all evals
adk evals <name>                 # run one eval
adk evals --tag <tag>            # filter by tag
adk evals --type regression      # filter by type
adk evals --verbose              # show all assertions
adk evals --format json          # JSON output for CI

adk evals runs                   # list recent runs
adk evals runs --latest          # most recent run
adk evals runs --latest -v       # with full details

Critical Patterns

user

event

// CORRECT
{ user: 'hello', expectSilence: true }
{ event: { type: 'payment.failed' }, expectSilence: true }

expectSilence

// WRONG — missing user or event
{ expectSilence: true }

// CORRECT — verifies the LLM extracted the right values
{ called: 'createTicket', params: { priority: { equals: 'high' } } }

// INCOMPLETE — doesn't verify params were correct
{ called: 'createTicket' }

outcome

// CORRECT — final state checked once after all turns
outcome: {
  state: [{ path: 'conversation.resolved', equals: true }],
  tables: [{ table: 'ticketsTable', row_exists: { status: { equals: 'open' } } }],
}

// WRONG — table may not be written until after all turns
conversation: [
  {
    user: 'Create a ticket',
    assert: { tables: [{ table: 'ticketsTable', row_exists: { status: { equals: 'open' } } }] },
  },
]

// CORRECT — start in a known state
setup: {
  state: {
    user: { plan: 'pro' },
    conversation: { phase: 'support' },
  },
}

// WRONG — depends on the bot correctly processing setup turns
conversation: [
  { user: 'I am on the pro plan' },      // hoping bot sets user.plan
  { user: 'I need help with billing' },   // actual test turn
]

Example Questions

• "Write an eval that tests my createTicket tool is called with the right priority"
• "How do I assert that the bot stays silent after an internal event?"
• "How do I test a multi-turn conversation where context is retained?"

• "How do I run only regression evals?"
• "How do I see which assertions failed and why?"
• "How do I integrate evals into GitHub Actions?"

• "My eval says the tool wasn't called but I think it was — how do I check?"
• "How do I inspect what the bot actually did during an eval?"

• "How do I test a workflow that uses step.sleep()?"
• "How do I verify a row was written to a table after a conversation?"
• "How do I test that state changed from the seeded value?"

Response Format

1. Show the complete

   new Eval({})

   call with realistic field values
2. Include imports (

   import { Eval } from '@botpress/adk'

   )
3. Explain each assertion and why it's the right choice for that scenario
4. Point out any mutual exclusivity rules if relevant (

   expectSilence

   vs

   assert.response

   ,

   user

   vs

   event

   )
5. Suggest the CLI command to run it:

   adk evals <name>

1. Ask for or show the failing assertion (

   expected

   /

   actual

   diff)
2. Suggest opening traces in the Control Panel to see what the bot did
3. Identify whether the issue is in the eval assertion or the bot's behavior