/ar:setup — Create New Experiment

Usage

/ar:setup                                    # Interactive mode
/ar:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower
/ar:setup --list                             # Show existing experiments
/ar:setup --list-evaluators                  # Show available evaluators

What It Does

If arguments provided

python {skill_path}/scripts/setup_experiment.py \
  --domain {domain} --name {name} \
  --target {target} --eval "{eval_cmd}" \
  --metric {metric} --direction {direction} \
  [--evaluator {evaluator}] [--scope {scope}]

If no arguments (interactive mode)

1. Domain — Ask: "What domain? (engineering, marketing, content, prompts, custom)"
2. Name — Ask: "Experiment name? (e.g., api-speed, blog-titles)"
3. Target file — Ask: "Which file to optimize?" Verify it exists.
4. Eval command — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)"
5. Metric — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)"
6. Direction — Ask: "Is lower or higher better?"
7. Evaluator (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?"
8. Scope — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?"

setup_experiment.py

Listing

# Show existing experiments
python {skill_path}/scripts/setup_experiment.py --list

# Show available evaluators
python {skill_path}/scripts/setup_experiment.py --list-evaluators

Built-in Evaluators

benchmark_speed

p50_ms

benchmark_size

size_bytes

test_pass_rate

pass_rate

build_speed

build_seconds

memory_usage

peak_mb

llm_judge_content

ctr_score

llm_judge_prompt

quality_score

llm_judge_copy

engagement_score

After Setup

• Experiment path and branch name
• Whether the eval command worked and the baseline metric
• Suggest: "Run

  /ar:run {domain}/{name}

  to start iterating, or

  /ar:loop {domain}/{name}

  for autonomous mode."