Arize Experiment Skill

Concepts

• Experiment = a named evaluation run against a specific dataset version, containing one run per example
• Experiment Run = the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata
• Dataset = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version
• Evaluation = a named metric attached to a run (e.g.,

  correctness

  ,

  relevance

  ), with optional label, score, and explanation

Prerequisites

ax

ax

• command not found

  or version error → see references/ax-setup.md

• 401 Unauthorized

  / missing API key → run

  ax profiles show

  to inspect the current profile. If the profile is missing or the API key is wrong: check

  .env

  for

  ARIZE_API_KEY

  and use it to create/update the profile via references/ax-profiles.md. If

  .env

  has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
• Space ID unknown → check

  .env

  for

  ARIZE_SPACE_ID

  , or run

  ax spaces list -o json

  , or ask the user
• Project unclear → check

  .env

  for

  ARIZE_DEFAULT_PROJECT

  , or ask, or run

  ax projects list -o json --limit 100

  and present as selectable options

List Experiments:

ax experiments list

ax experiments list
ax experiments list --dataset-id DATASET_ID --limit 20
ax experiments list --cursor CURSOR_TOKEN
ax experiments list -o json

Flags

--dataset-id

--limit, -l

--cursor

-o, --output

-p, --profile

Get Experiment:

ax experiments get

ax experiments get EXPERIMENT_ID
ax experiments get EXPERIMENT_ID -o json

Flags

EXPERIMENT_ID

-o, --output

-p, --profile

Response fields

id

name

dataset_id

dataset_version_id

experiment_traces_project_id

created_at

updated_at

Export Experiment:

ax experiments export

--all

ax experiments export EXPERIMENT_ID
# -> experiment_abc123_20260305_141500/runs.json

ax experiments export EXPERIMENT_ID --all
ax experiments export EXPERIMENT_ID --output-dir ./results
ax experiments export EXPERIMENT_ID --stdout
ax experiments export EXPERIMENT_ID --stdout | jq '.[0]'

Flags

EXPERIMENT_ID

--all

--output-dir

.

--stdout

-p, --profile

REST vs Flight (

--all

)

• REST (default): Lower friction -- no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page.
• Flight (

  --all

  ): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port (

  flight.arize.com:443

  ) which some corporate networks may block.

--all

[
  {
    "id": "run_001",
    "example_id": "ex_001",
    "output": "The answer is 4.",
    "evaluations": {
      "correctness": { "label": "correct", "score": 1.0 },
      "relevance": { "score": 0.95, "explanation": "Directly answers the question" }
    },
    "metadata": { "model": "gpt-4o", "latency_ms": 1234 }
  }
]

Create Experiment:

ax experiments create

ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json
ax experiments create --name "claude-test" --dataset-id DATASET_ID --file runs.csv

Flags

--name, -n

--dataset-id

--file, -f

-o, --output

-p, --profile

Passing data via stdin

--file -

echo '[{"example_id": "ex_001", "output": "Paris"}]' | ax experiments create --name "my-experiment" --dataset-id DATASET_ID --file -

# Or with a heredoc
ax experiments create --name "my-experiment" --dataset-id DATASET_ID --file - << 'EOF'
[{"example_id": "ex_001", "output": "Paris"}]
EOF

Required columns in the runs file

example_id

output

additionalProperties

Delete Experiment:

ax experiments delete

ax experiments delete EXPERIMENT_ID
ax experiments delete EXPERIMENT_ID --force   # skip confirmation prompt

Flags

EXPERIMENT_ID

--force, -f

-p, --profile

Experiment Run Schema

{
  "example_id": "required -- links to dataset example",
  "output": "required -- the model/system output for this example",
  "evaluations": {
    "metric_name": {
      "label": "optional string label (e.g., 'correct', 'incorrect')",
      "score": "optional numeric score (e.g., 0.95)",
      "explanation": "optional freeform text"
    }
  },
  "metadata": {
    "model": "gpt-4o",
    "temperature": 0.7,
    "latency_ms": 1234
  }
}

Evaluation fields

label

correct

incorrect

partial

score

explanation

label

score

explanation

Workflows

Run an experiment against a dataset

1. Find or create a dataset:
   bash

   ax datasets list
   ax datasets export DATASET_ID --stdout | jq 'length'

2. Export the dataset examples:
   bash

   ax datasets export DATASET_ID

3. Process each example through your system, collecting outputs and evaluations
4. Build a runs file (JSON array) with

   example_id

   ,

   output

   , and optional

   evaluations

   :
   json

   [
     {"example_id": "ex_001", "output": "4", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}},
     {"example_id": "ex_002", "output": "Paris", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}}
   ]

5. Create the experiment:
   bash

   ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json

6. Verify:

   ax experiments get EXPERIMENT_ID

Compare two experiments

1. Export both experiments:
   bash

   ax experiments export EXPERIMENT_ID_A --stdout > a.json
   ax experiments export EXPERIMENT_ID_B --stdout > b.json

2. Compare evaluation scores by

   example_id

   :
   bash

   # Average correctness score for experiment A
   jq '[.[] | .evaluations.correctness.score] | add / length' a.json
   
   # Same for experiment B
   jq '[.[] | .evaluations.correctness.score] | add / length' b.json

3. Find examples where results differ:
   bash

   jq -s '.[0] as $a | .[1][] | . as $run |
     {
       example_id: $run.example_id,
       b_score: $run.evaluations.correctness.score,
       a_score: ($a[] | select(.example_id == $run.example_id) | .evaluations.correctness.score)
     }' a.json b.json

4. Score distribution per evaluator (pass/fail/partial counts):
   bash

   # Count by label for experiment A
   jq '[.[] | .evaluations.correctness.label] | group_by(.) | map({label: .[0], count: length})' a.json

5. Find regressions (examples that passed in A but fail in B):
   bash

   jq -s '
     [.[0][] | select(.evaluations.correctness.label == "correct")] as $passed_a |
     [.[1][] | select(.evaluations.correctness.label != "correct") |
       select(.example_id as $id | $passed_a | any(.example_id == $id))
     ]
   ' a.json b.json

jq 'length' a.json

Download experiment results for analysis

1. ax experiments list --dataset-id DATASET_ID

   -- find experiments

2. ax experiments export EXPERIMENT_ID

   -- download to file
3. Parse:

   jq '.[] | {example_id, score: .evaluations.correctness.score}' experiment_*/runs.json

Pipe export to other tools

# Count runs
ax experiments export EXPERIMENT_ID --stdout | jq 'length'

# Extract all outputs
ax experiments export EXPERIMENT_ID --stdout | jq '.[].output'

# Get runs with low scores
ax experiments export EXPERIMENT_ID --stdout | jq '[.[] | select(.evaluations.correctness.score < 0.5)]'

# Convert to CSV
ax experiments export EXPERIMENT_ID --stdout | jq -r '.[] | [.example_id, .output, .evaluations.correctness.score] | @csv'

Related Skills

• arize-dataset: Create or export the dataset this experiment runs against → use

  arize-dataset

  first
• arize-prompt-optimization: Use experiment results to improve prompts → next step is

  arize-prompt-optimization

• arize-trace: Inspect individual span traces for failing experiment runs → use

  arize-trace

• arize-link: Generate clickable UI links to traces from experiment runs → use

  arize-link

Troubleshooting

ax: command not found

401 Unauthorized

No profile found

Experiment not found

ax experiments list

Invalid runs file

example_id

output

example_id mismatch

example_id

No runs found

ax experiments get

Dataset not found

ax datasets list

Save Credentials for Future Use