eval-generator

/eval-suite-planner

• Plan found: Use it as the blueprint. Say: "Generating test cases from your eval suite plan (X scenarios)." Generate one test case per row.
• No plan, but user provides an agent description: Generate from scratch. Say: "Generating eval scenarios for: [agent task in your own words]." If the description is fewer than two sentences or doesn’t mention success criteria, ask exactly one clarifying question, then wait.
• No plan and no description: Say: "I need either an agent description or a plan from

  /eval-suite-planner

  . Run

  /eval-suite-planner <your agent description>

  first for the best results, or give me a description and I’ll generate directly."

• Handles multi-step tasks that require context across turns (e.g., booking a trip with departure, return, and seat selection)
• Needs to ask clarifying questions before completing a request
• Must maintain state (e.g., remembering a customer’s account after initial identification)
• Has handoff or escalation flows that depend on prior turns

• Answers standalone questions (FAQ, policy lookup, factual retrieval)
• Routes to a single tool per request
• Produces a self-contained output per input (e.g., a summary, a classification)

• 6-8 total scenarios
• At least 2 happy-path / core business cases
• At least 2 edge cases (empty input, long input, ambiguous input, malformed input)
• At least 1 adversarial case (prompt injection, out-of-scope request, policy violation attempt)
• Fill remaining with whatever gives the most signal for this agent

• Up to 20 test cases per test set
• Each test case supports up to 12 total messages (6 user-agent Q&A pairs)
• Supported test methods: General Quality, Keyword Match, Capability Use (called "Capabilities match" in conversation UI), Custom (Classification)
• NOT supported in conversation mode: Compare Meaning, Text Similarity, Exact Match

• 4-6 conversation test cases
• At least 2 happy-path multi-step task completions
• At least 1 conversation with a clarification loop (user gives vague input, agent asks, user clarifies)
• At least 1 conversation with a mid-conversation topic switch or escalation
• At least 1 adversarial turn mid-conversation (e.g., user attempts prompt injection after establishing context)

Conversation Test Case #N: [Scenario Name]

Turn 1 - User: [realistic user message] Turn 1 - Agent (expected): [expected agent response or behavior description]

Turn 2 - User: [follow-up that depends on Turn 1 context] Turn 2 - Agent (expected): [expected response maintaining context]

Turn 3 - User: [further follow-up or new request within same session] Turn 3 - Agent (expected): [expected response]

Test method: General Quality / Keyword Match / Capability Use / Custom (Classification) Keywords (if Keyword Match): [comma-separated keywords expected in any agent response] What this tests: [1 sentence: what multi-turn capability is being evaluated] Critical turn: [which turn is most likely to fail and why]

• Each turn must build naturally on the previous turn — do not write turns that could stand alone
• Agent expected responses should describe the behavior, not the exact wording (these are reference responses for the LLM judge)
• Include at least one conversation where the user’s intent shifts or expands across turns
• Flag which turn is the "critical" turn — the one most likely to fail (e.g., Turn 3 where context from Turn 1 must be retained)

1. Quick conversation set — auto-generates 10 short conversations from agent description and instructions
2. Full conversation set — generates conversations from agent knowledge sources or topics (short or long)
3. Use test chat — converts the latest test chat session into a conversation test case
4. Manual entry — add user questions and agent reference responses turn by turn in the UI

• Guide what they enter manually in the Copilot Studio conversation editor
• Compare against AI-generated conversation sets to check coverage
• Document the multi-turn scenarios that matter before creating test sets in the UI

• Evaluation instructions: "Evaluate the agent's response for HR policy compliance. Check: protects privacy, avoids discrimination or bias, provides safe HR-aligned guidance, does not give legal advice."
• Labels:

  Compliant

  (Pass) — "Response follows all HR policies and guidelines." /

  Non-Compliant

  (Fail) — "Response violates one or more HR policy requirements."

• Domain-specific compliance or regulatory requirements the built-in methods cannot express
• Tone or style evaluation (e.g., "Is the response empathetic?" or "Does it match our brand voice?")
• Safety or guardrail checks beyond what General Quality's abstention criterion covers
• Classification accuracy where labels are complex or context-dependent (not a simple exact match)

1. Start a refinement run — configure a test run with the rubric attached; the AI grades each Generative Answer test case on 1–5 and writes a detailed rationale.
2. Review in Standard view first — Standard refinement view hides AI grades so humans grade without bias. This is critical: research shows seeing AI grades anchors human judgment.
3. Human grading — assign a grade (1–5) and write reasoning for every test case. Reasoning must reference specific rubric criteria ("Grade 4: accurate technical info, professional tone, but lacks timeline estimates for Grade 5"). Vague reasoning ("Pretty good") undermines refinement.
4. Mark examples — flag select test cases as Good or Bad examples. Focus on misaligned cases and edge cases. Quality over quantity — a few well-chosen examples beat many mediocre ones.
5. Check alignment — switch to Full refinement view to compare AI vs. human grades. Alignment per test case:

   alignment = 100% × (1 − |AI − Human| / 4)

   . Average across all graded cases for the run score.

   Average alignment	Assessment	Action
   90–100%	Excellent	Rubric is reliable; move to testing mode
   75–89%	Good	Mostly aligned; refine edge cases
   60–74%	Fair	Needs improvement; focus on misalignment patterns
   < 60%	Poor	Significant refinement or rubric redesign needed

6. Refine the rubric — click "Refine Rubric" to let AI update the rubric based on all human grades, reasoning, examples, and misalignment patterns. Use "Save As" for early iterations (preserve history), "Save" once stable.
7. Re-run — duplicate the test run with the refined rubric. Compare alignment to the prior iteration.
8. Iterate — repeat until alignment reaches 75–90%+. Don’t chase 100% — some subjectivity is inherent, and diminishing returns kick in around 85–90%.

• Grade 5 (default): Only exemplary responses pass — use for critical communications (IR reports, executive summaries)
• Grade 4: Strong or better responses pass — appropriate for most business communications
• Grade 3: Acceptable or better — minimum functional quality for internal tools

• Do NOT use General Quality when there IS a specific correct answer — use Compare Meaning or Keyword Match instead. General Quality has no expected response to compare against.
• Do NOT use Compare Meaning for factual claims with precise numbers/dates/names — use Keyword Match (All) to ensure the exact facts appear.
• Do NOT use Compare Meaning for adversarial/negative tests — there is often no "expected" answer. Use General Quality for adversarial handling and edge cases.
• Do NOT use General Quality for tool routing tests — use Capability Use to verify the correct connector/topic fired.
• Do NOT use Text Similarity when meaning is what matters — use Compare Meaning. Text Similarity penalizes paraphrasing.
• Do NOT use Exact Match for natural language responses — Exact Match fails on any variation in wording. Only use for labels, codes, or structured data.

• Every

  Question

  must be a realistic input the agent would receive in production — specific, not a placeholder
• Every

  Expected Response

  must be concrete and testable — never vague
• For General Quality, leave Expected Response blank (the LLM judge evaluates without one)
• For Keyword Match, put the required keywords in Expected Response
• For adversarial cases, the Expected Response should describe what the agent should NOT do

/xlsx

"Question","Expected response","Testing method"
"How do I return an item?","The agent should explain the return policy...","Compare meaning"
"What are your hours?","","General quality"
"Can I get a refund after 90 days?","90 days, refund policy, exception","Keyword match"
"What is the order status for #12345?","Order #12345 is in transit","Exact match"

• Three columns, in this exact order:

  Question

  ,

  Expected response

  ,

  Testing method

• Every value must be enclosed in double quotes
• Any double quotes inside a value must be escaped as

  ""

• Question

  maps to the Question column from the test case table

• Expected response

  maps to the Expected Response column (leave empty string

  ""

  for General Quality rows)

• Testing method

  must use one of these exact values (case-sensitive as shown):

  • General quality

  • Compare meaning

  • Similarity

    (this is Text Similarity — note: the CSV value is "Similarity", not "Text Similarity")

  • Exact match

  • Keyword match

1. Import all other test cases via the CSV
2. In the Copilot Studio UI, manually add test cases that use Capability Use or Custom
3. Note this in the docx report so the customer knows which cases to add manually

• Download the CSV template: In Copilot Studio, after selecting New evaluation, you can download a ready-to-use CSV template under Data source. Use this template to verify your column format before importing.
• Copilot Studio's built-in test generation — alternatives to CSV import: Besides importing the CSV this skill generates, customers can also create test sets directly in Copilot Studio using these methods (all under New evaluation → Single response):
  • Quick question set — auto-generates 10 questions from the agent's description, instructions, and capabilities. Good for a fast baseline or to seed a larger test set.
  • Full question set — generates questions from a specific knowledge source (text, Word, Excel — up to 5 MB) or from topics. Best for agents using generative orchestration (knowledge) or classic orchestration (topics). You choose how many questions to generate.
  • Use your test chat — converts the latest test chat session into test cases. Useful after exploratory testing — just replay the questions you already asked.
  • Create from analytics themes — uses themes (preview) from production conversations to generate test cases focused on one area (e.g., "billing and payments" questions). Requires the themes analytics feature to be enabled.
  • Manual entry — write questions yourself directly in the UI. Recommend combining approaches: use this skill's CSV for structured, scenario-based test cases, then supplement with Copilot Studio's auto-generation to catch gaps the plan didn't anticipate. Note these options in the docx report.
• Test set limits: A single test set supports up to 100 test cases. Each question can be up to 1,000 characters (including spaces). If your eval plan exceeds 100 scenarios, split into multiple test sets by category or quality signal.
• 89-day result retention: Test results are only available in Copilot Studio for 89 days. After that, they are deleted. Always export results to CSV immediately after running evals — especially for baseline runs you will compare against later. Use the export option under test results to save a permanent copy.
• Version your test sets like code: Test sets are living artifacts — they change as the agent evolves, new failure modes appear, and business requirements shift. Treat them with the same rigor as source code:
  • Keep a changelog: Every time you add, remove, or modify a test case, record what changed and why. A shared spreadsheet or version-controlled CSV works. Without this, you will not know whether a score change came from the agent improving or the test set shifting underneath it.
  • Tag baselines: Before a major agent change (new knowledge source, updated system prompt, new tool), snapshot the current test set as a named baseline (e.g., "v2.1 — pre-knowledge-update"). Run the new agent version against the old baseline first, then against an updated test set. This separates "agent changed" from "test changed."
  • Retire, don’t delete: When a test case becomes irrelevant (deprecated feature, changed policy), move it to a "retired" section instead of deleting it. You may need it for regression testing if the feature returns or the policy reverts.
  • Track provenance: For each test case, note where it came from — was it generated from the eval plan, added from a production incident, suggested by a subject matter expert, or auto-generated by Copilot Studio? Provenance helps you prioritize: real-failure-sourced cases are higher signal than synthetic ones.
  • Review cadence: Revisit the full test set quarterly (or after any major agent change). Stale expected responses are the #1 cause of false failures over time — the agent got better but the expected response still reflects the old behavior.
• User profiles — simulate different user experiences: Copilot Studio lets you assign a user profile to a test set so evals run under that user's authentication. This is critical when knowledge sources or SharePoint sites are role-gated — a director and an intern may get different answers from the same agent. To test this:
  • Create separate test sets per profile (e.g., "Core Scenarios — Director" and "Core Scenarios — Intern")
  • In each test set, select Manage profile and choose the appropriate account
  • Compare results across profiles to find role-based gaps
  • Limitation: Multi-profile evaluation is only supported for agents without connector dependencies. If the agent uses tools/connectors that require authentication, the eval must run under the logged-in account that owns those connections — selecting a different profile will fail with "This account cannot connect to tools". In that case, test knowledge-source differences with user profiles, but test tool-dependent scenarios under the tool owner's account.
  • Note which profile was used in the docx report — test results in Copilot Studio show the profile, but exported CSVs may not.
• GCC (Government Community Cloud) limitations: If the customer is in a GCC environment, two features are unavailable:
  • No user profiles — the test-set profile feature described above is not available in GCC. All evals run under the maker’s own account.
  • No Text Similarity method — the “Similarity” test method cannot be used. Replace any Text Similarity test cases with Compare Meaning (for semantic matching) or Keyword Match (for specific phrasing). All other test methods work normally.
  • Ask the customer early: “Are you in a GCC environment?” If yes, omit Text Similarity rows from the CSV and note the restriction in the docx report. Source: About agent evaluation.

/docx

• Title: "Eval Test Set: [Agent Name]" (derive agent name from the plan or description)
• Evaluation mode: State whether this is a Single Response or Conversation test set, and why that mode was chosen
• Plan summary: Brief summary of the eval suite plan this was generated from (or the agent description if generated from scratch)
• Test cases: Each test case formatted clearly with:
  • Scenario name
  • Input / question (for single-response) or full conversation turns (for conversation mode)
  • Expected response or reference responses
  • Test method
  • Pass score (if applicable)
• Method rationale: Include the Why These Methods? section from Step 3 so the customer understands the reasoning behind each test method choice
• Import guide:
  • For single-response: Document the CSV column format (Question, Expected response, Testing method), list which test methods are CSV-importable vs. UI-only, and note pass scores to configure in the UI after import
  • For conversation: Explain that CSV import is not available, list the four creation methods (Quick set, Full set, Test chat, Manual entry), and recommend using the conversation test cases in this report as a blueprint for manual entry
• Human review checkpoints: Include the full checkpoint table from Step 4 so the customer has a printed checklist to work through before running evals
• Reviewer notes: The scenario suggestion and mandatory reminder from Step 4
• Operational reminders: Include the 89-day result retention warning (export results immediately after running), the 100-test-case limit per test set, and the CSV template download tip

/eval-suite-planner

/eval-result-interpreter

1. What property of the response this method actually measures
2. Why that property matters for the specific scenarios it was assigned to
3. What would go wrong if a different method were used instead

Compare Meaning (used for scenarios #1, #3, #5): These scenarios have a specific correct answer, but the exact wording does not matter — only the meaning. Compare Meaning uses an LLM to judge semantic equivalence, so the agent can phrase the answer differently and still pass. If you used Keyword Match instead, valid paraphrases would fail. If you used General Quality, there would be no expected answer to compare against, so factual errors would go undetected.

General Quality (used for scenarios #2, #7): These are adversarial and edge-case scenarios where there is no single correct answer — the agent just needs to respond helpfully and safely. General Quality uses an LLM judge to evaluate relevance, groundedness, completeness, and appropriate abstention without needing an expected response.

Keyword Match (All) (used for scenario #4): This scenario requires specific facts (policy numbers, dates, dollar amounts) that must appear verbatim. Compare Meaning might accept a response that captures the gist but drops a critical number. Keyword Match ensures every required fact is present.

• Why conversation mode was chosen over single-response (what multi-turn behavior is being tested)
• Why the available conversation methods (General Quality, Keyword Match, Capability Use, Custom) are sufficient — or flag if the customer should also create a complementary single-response test set for scenarios that need Compare Meaning, Exact Match, or Text Similarity

• One more scenario to consider: Describe an additional scenario worth adding manually — something that did not fit but is realistic.
• Mandatory reminder: "This test set was AI-generated and must be reviewed by someone who knows the agent’s domain before use. No eval should run without human validation of inputs, expected responses, and method choices."

• Each case must be independently understandable — no references to "the previous case"
• When using a plan, generate exactly the scenarios listed — do not add or remove scenarios without saying why
• The Copilot Studio table (single-response) or conversation blocks (conversation mode) is the primary output displayed in conversation; the CSV (single-response only) and docx report are generated afterward
• Make inputs realistic and specific: use names, dates, product references, and context that a real user would provide
• The CSV must be valid and importable into Copilot Studio without manual editing
• For conversation mode, explicitly recommend whether the customer should also create a complementary single-response test set