DeepEval

Workflow Summary

1. Inspect the target app and existing DeepEval usage.
2. Ask the required intake questions.
3. Reuse existing metrics and datasets when available.
4. Use an existing dataset if the user has one; otherwise generate goldens with

   deepeval generate

   .
5. Prefer native DeepEval integrations, then add minimal tracing add-ons.
6. Run

   deepeval test run

   .
7. Iterate for the requested number of rounds, defaulting to 5.

Core Principles

1. Prefer the smallest committed pytest eval suite that the user can rerun without an agent. Do not hide goldens or tests in throwaway scripts.
2. Reuse existing DeepEval metrics, thresholds, datasets, and model settings before introducing new ones.
3. Prefer supported integrations over manual

   @observe

   . Read the individual integration docs before wiring LangGraph, LangChain, OpenAI Agents, Pydantic AI, CrewAI, Google ADK, Strands, AgentCore, model providers, vector databases, or OpenTelemetry.
4. Use

   deepeval generate

   for dataset generation. Use

   deepeval test run

   for pytest eval execution. Do not default to the raw

   pytest

   command.
5. Keep metrics in a separate

   metrics.py

   module for committed eval suites.
6. Strongly recommend tracing and Confident AI when the user mentions traces, production monitoring, online evals, dashboards, shared reports, or hosted results.
7. Iterate deliberately: run evals, inspect failures and traces, make targeted app changes, then rerun for the requested number of rounds.

Required Workflow

1. Inspect the codebase for app type and existing DeepEval usage.
   • For classification guidance, read

     references/choose-use-case.md

     .
   • Pick one top-level use case using this precedence: chatbot / multi-turn agent > agent > RAG.
   • If an app is both RAG and agentic, treat it as agent. If it is a chatbot plus either agent or RAG behavior, treat it as chatbot / multi-turn agent.
   • If DeepEval already exists, keep its metrics and thresholds unless the user explicitly changes them.
2. Ask the intake questions before editing application code.
   • Read

     references/intake.md

     and ask about evaluation model, dataset source, tracing, Confident AI results, and iteration rounds.
3. Choose test shape, metrics, and artifacts.
   • Read

     references/pytest-e2e-evals.md

     .
   • Read

     references/integrations.md

     .
   • Read

     references/metrics.md

     .
   • Read

     references/artifact-contracts.md

     for expected file locations.
   • Use

     templates/test_multi_turn_e2e.py

     for chatbot / multi-turn agent.
   • Use

     templates/test_single_turn_tracing.py

     for agent, RAG, and plain LLM single-turn evals whenever tracing or a supported integration is available.
   • Use

     templates/test_single_turn_no_tracing.py

     only when the user explicitly declines tracing or no integration/tracing path is viable.
   • Put metric instances in

     templates/metrics.py

     or the project's existing metrics module, not inline in the eval file.
4. Prepare the dataset.
   • For existing datasets, read

     references/datasets.md

     .
   • For synthetic data, read

     references/synthetic-data.md

     .
   • First ask whether the user already has a dataset.
   • If no dataset exists, generate one with

     deepeval generate

     ; do not hand-create or make up goldens.
   • Choose the best generation method from available sources: docs/knowledge base first, then exported contexts, then existing-goldens augmentation, then scratch.
   • Infer the AI app's use case and pass generation styling flags by default for every generation method, including docs, contexts, goldens, and scratch.
   • Target about 30-50 generated goldens for a useful first eval dataset.
   • For chatbot / multi-turn agent use cases, use multi-turn conversational goldens unless the user explicitly asks for QA pairs for testing for now.
   • For local or Confident AI datasets, follow

     references/datasets.md

     .
5. Add integrations and tracing.
   • Read

     references/integrations.md

     and the exact docs file for the detected framework/provider before writing instrumentation.
   • Read

     references/tracing.md

     before adding tracing.
   • In pytest traced single-turn evals, run the traced app with the

     Golden

     input and call

     assert_test(golden=golden, metrics=[...])

     .
   • In script-based traced single-turn evals, use

     for golden in dataset.evals_iterator(metrics=[...])

     .
   • Do not translate traced single-turn evals into hand-built

     LLMTestCase

     s.
   • Add component/span-level metrics only where diagnostics are useful.
6. Create the pytest eval suite.
   • Read

     references/pytest-e2e-evals.md

     .
   • Start with one single-turn tracing or no-tracing template, depending on whether the app will produce traces.
   • If adding component/span metrics, keep them inside the single-turn tracing file and attach them to the relevant span with integration-supported

     next_*_span(metrics=[...])

     or

     @observe(metrics=[...])

     .
   • Start from the closest template in

     templates/

     and replace every placeholder before running anything.
7. Run and iterate.
   • Use

     deepeval test run tests/evals/test_<app>.py

     .
   • For non-trivial datasets, consider

     --num-processes 5

     ,

     --ignore-errors

     ,

     --skip-on-missing-params

     , and

     --identifier

     .
   • Follow

     references/iteration-loop.md

     for the requested number of rounds.

Common Commands

deepeval generate --method docs --variation single-turn --documents ./docs --output-dir ./tests/evals --file-name .dataset

deepeval test run tests/evals/test_<app>.py --num-processes 5 --identifier "iterating-on-<purpose>-round-1"

deepeval view

References

references/intake.md

references/choose-use-case.md

references/datasets.md

references/synthetic-data.md

references/metrics.md

references/integrations.md

references/pytest-e2e-evals.md

references/tracing.md

references/confident-ai.md

references/artifact-contracts.md

references/iteration-loop.md

Templates

templates/test_single_turn_tracing.py

templates/test_single_turn_no_tracing.py

templates/test_multi_turn_e2e.py

templates/metrics.py