1. NEVER update the metric prompt without explicit user approval. Always present the proposed changes and wait for confirmation before running

   coval metrics update

   .
2. Only operate on one metric at a time. If the user provides a project with multiple metrics, ask which one to focus on.
3. Only support text LLM judge metrics. Valid types:

   METRIC_LLM_BINARY

   ,

   METRIC_CATEGORICAL

   ,

   METRIC_NUMERICAL_LLM_JUDGE

   . Reject audio metrics, toolcall, metadata, regex, pause, and composite types.
4. NEVER fabricate data. If a CLI command or API call fails, report the error — do not invent scores or transcripts.
5. Use the CLI first, fall back to curl only when the CLI does not return required fields (e.g., the LLM explanation under

   result.llm.answer_explanation

   ).

coval metrics get <metric_id> --format json

• metric_name

  — the full display name

• metric_type

  — must be one of:

  METRIC_LLM_BINARY

  ,

  METRIC_CATEGORICAL

  ,

  METRIC_NUMERICAL_LLM_JUDGE

• prompt

  — the current evaluation prompt

• min_value

  /

  max_value

  — for numerical metrics only

• categories

  — for categorical metrics only

"This skill only supports text LLM judge metrics (binary, categorical, numerical). The metric

{metric_name}

is type

{metric_type}

, which is not supported."

coval review-projects get <project_id> --format json

• The

  linked_metric_ids

  includes the target metric ID
• Note the assignees and linked simulation IDs

Metric: {metric_name} (

{metric_id}

) Type: {metric_type} Current prompt: "{prompt}" Project: {project_display_name} ({N} simulations, {M} assignees)

Proceeding to gather annotations and calculate agreement.

• ground_truth_float_value

  or

  ground_truth_string_value

  — the human label

• reviewer_notes

  — the reviewer's reasoning (may be null)

Agreement Report

Metric: {metric_name} ({metric_type}) Total completed annotations: {N} Agreement rate: {agree_count}/{N} ({percentage}%)

Breakdown

	Count	%
Agree	{agree_count}	{agree_pct}%
Disagree	{disagree_count}	{disagree_pct}%

{For numerical metrics only:} Mean absolute error: {mae} Score range: {min_value}–{max_value}

agrees == false

1. Read the transcript carefully — understand the full conversation
2. Read the reviewer notes — understand WHY the human disagreed with the machine
3. Read the LLM explanation (if available) — understand the machine's reasoning
4. Determine who was more likely correct and why the machine erred

• Systematic bias: Does the machine consistently score too high or too low?
• Edge cases: Are there conversation patterns the prompt doesn't account for?
• Ambiguity: Is the prompt vague about certain scenarios?
• Missing context: Does the prompt fail to instruct the LLM about important nuances?
• Misinterpretation: Does the LLM misunderstand what the prompt is asking?

agrees == true

1. Read a sample of reviewer notes (if any) — understand what the metric handles well
2. Identify what the current prompt does correctly — these strengths must be preserved

Analysis

What the metric gets right

• {bullet points from agreement analysis}

Disagreement patterns

• Pattern 1: {description} (seen in {N} annotations)
  • Example: Simulation {sim_id} — Human: {value}, Machine: {value}
  • Reviewer note: "{note}"
  • Root cause: {why the machine got this wrong}
• Pattern 2: ...

Key insights from reviewer notes

• {synthesized themes from reviewer feedback}

Disagreements

#	Simulation	Human Label	Machine Label	Reviewer Note
1	{sim_id}	{human}	{machine}	{note or "—"}
2	{sim_id}	{human}	{machine}	{note or "—"}
...

How would you like to resolve these?

1. Update the metric prompt — fix the prompt so the machine aligns with the human labels
2. Update human labels — correct the human labels to align with the machine scores
3. Both — update some labels and fix the prompt

If choosing option 2 or 3, specify which disagreements to relabel (e.g., "update labels for #1, #3, #5" or "all").

1. For each annotation the user selects, confirm the new ground-truth value:

   "Annotation #{n} (Simulation {sim_id}): Current human label is

   {human_value}

   . Update to

   {machine_value}

   (the machine score)? Or enter a different value."

2. After confirmation, update each annotation:
   bash

   coval review-annotations update <annotation_id> \
     --ground-truth-float <new_value> \
     --notes "Label corrected during metric improvement review. Previous value: {old_value}"

   For string-based metrics, use

   --ground-truth-string

   instead of

   --ground-truth-float

   .
3. After all label updates, recalculate the agreement rate with the corrected labels and present the updated report.
4. If the user chose option 2 (labels only), skip to the end — no prompt changes needed.
5. If the user chose option 3 (both), continue to Phase 4 with the remaining disagreements that were NOT resolved by label updates.

1. Preserves what works — keep language that leads to correct evaluations
2. Addresses disagreement patterns — add explicit instructions for edge cases
3. Incorporates reviewer feedback — use the language and reasoning from reviewer notes
4. Remains clear and concise — avoid bloating the prompt with excessive detail
5. Maintains the same evaluation format — the metric type and output format must not change

Proposed Prompt Update

Current prompt

{current_prompt}

Proposed prompt

{new_prompt}

Changes explained

• {bullet point explaining each significant change and which disagreement pattern it addresses}

Would you like to apply this prompt update? (yes/no)

• If yes → proceed to Phase 5
• If no → ask what they'd like to change. Iterate on the prompt until they approve or decide to stop.
• If they want to edit manually — that's fine too. The analysis is the primary value.

coval metrics update <metric_id> --prompt "<approved_prompt>"

coval metrics get <metric_id> --format json

Metric prompt updated successfully. You can re-run simulations and create a new review project to validate the improvement.