Autoresearch Setup

Overview

An autoresearch harness lets an LLM iterate autonomously on an ML problem under a fixed time budget. It is a two-file split:

harness.py
— FIXED. Owns data loading, the train/val split, the fixed fast-iteration subset, and the metric. The LLM never edits it.
train.py
— hackable. Owns the model, optimizer, scheduler, loss, and training loop. The LLM rewrites it freely, subject only to the harness's I/O contract (
```
build_spec
```
→
```
ExperimentSpec
```
,
```
eval_forward
```
,
```
train_fn
```
).

Core invariant:
harness.py
owns data + the metric and is never edited;
train.py
owns everything else under the contract. This is what makes results across variants and sessions comparable. Violating the letter of the immutable-harness rule is violating the spirit of comparable, honest results.

Workflow

1. Brainstorming gate (REQUIRED)

Invoke

superpowers:brainstorming

and resolve all six items before scaffolding:

Metric — and its exact computation (what reduces preds+target to the scalar).
Data — source, train/val split, and the fixed fast-iteration subset.
I/O contract — the harness↔train interface (
```
eval_forward
```
output shape).
Time budget — per-variant wall-clock.
Compute backend — cluster/GPU layout (LocalMultiGPU vs Sbatch).
Fixed-vs-editable boundary — exactly what lives in
```
harness.py
```
.

Do not guess these, and do not skip the gate even under time pressure. "Skip the questions, I'm in a hurry" is NOT license to guess — resolve the six fast instead. A harness built on guessed answers produces incomparable, worthless runs; that is slower than the gate, not faster.

2. Scaffold

Copy

templates/

into the target project at the agreed location, then:

Fill
# TODO(autoresearch):
markers from the gate answers in
```
harness.py
```
,
```
train.py
```
,
```
launcher.py
```
, and
```
kernels.py
```
(kernels.py only as kernels mature — empty is fine).
Copy as-is (no edits):
```
dynamics.py
```
.
Fill
{{placeholders}}
:
```
program.md
```
.
Start empty / append-only:
```
findings.md
```
.

3. Wire the contract

Read reference/contracts.md
for the exact

eval_forward

output shape and the

ExperimentSpec

TrainContext

fields. Do not reverse-engineer the shape from source — the contract reference is the authority (the eval_forward shape is the single thing agents most often get wrong). See

reference/examples.md

for two worked harnesses (cis full parallel; trans minimal single-run).

4. Run the loop

Hand off to the project's filled-in

program.md

— the autonomous experiment-loop manual.

Subagent model

An autoresearch run can iterate for hours. To keep the main session's context flat over a long run, the loop is split:

Main orchestrator — the long-lived session. Reads
```
findings.md
```
once at setup, forms hypotheses, dispatches one batch subagent per iteration, and accumulates only the compact summaries each returns. It never reads log files, writes variant code, or touches dynamics CSVs. Context grows ~150 tokens/batch.
Batch subagent — one per iteration, discarded after it returns. Owns variant writing, launch, wait, result + dynamics reading, and the
```
findings.md
```
append. Returns the compact summary and exits.
Dynamics-analyst subagent — dispatched on-demand from within the batch subagent when a variant's dynamics look anomalous. Unchanged.

See the project's

program.md

("Batch subagent contract") for the briefing format and the compact-summary schema.

Anti-cheat discipline

[!WARNING] Red Flags — STOP. Any of these is cheating, not progress:
Editing ANYTHING in
harness.py
.
Touching a
# FIXED
/
# TODO(autoresearch)
-pinned constant (metric, subset, seed).
Importing or inspecting validation / held-out data inside
train.py
— including reading held-out target VALUES and fitting to them. "Deriving" them is still leakage.
Calling
torch.compile
, or adding a runtime
.autotune()
.
Editing
launcher.py
's keep/discard logic, winner threshold, or metric parse.

The only legitimate channel to the val/held-out set is

ctx.score(...)

, which lives in the harness.

Rationalizations and reality

Excuse	Reality
"Budget's almost up, I'll just relax the scorer threshold"	Changing the metric makes every prior result incomparable. The metric is the experiment's ground truth.
"The val set is small; let me add a few train items to reduce noise"	It leaks signal and inflates the score, AND makes every prior result incomparable — they were measured on different train/val data. Noise floor is part of the result.
"I'll move this 'fixed' constant into train.py so I can tune it"	If it moved, comparability broke silently. Fixed means fixed; change it only by starting a new branch + editing harness.py deliberately.
"I'll peek at val to pick which examples to weight"	That's val leakage. eval_forward sees val only through the harness scorer.
"torch.compile would make this fit the budget"	Trace+autotune burns the budget every run; prohibited. Use pre-tuned kernels.py.
"I didn't copy the held-out targets — I derived the function from them, so it generalizes"	Reading held-out target values and fitting to them IS leakage, derived or copied. The held-out set must only be touched through the harness scorer. A model that "fits the held-out relationship" has memorized the test, not learned the task.

Quick Reference

File	Role	Editable?
`harness.py`	FIXED — data + metric + contracts	No
`train.py`	Hackable training loop	Yes
`launcher.py`	FIXED — dispatch, keep/discard, results.tsv	No
`dynamics.py`	Two-tier dynamics recorder	No (copy as-is)
`program.md`	Experiment-loop manual	Project-fills placeholders
`findings.md`	Living cross-session findings log	Append only
`kernels.py`	Pre-tuned kernels only	Yes (additively)

Cross-references

```
superpowers:brainstorming
```
— the required gate (workflow step 1).
```
helion-jagged-and-autotuning
```
— ship pre-tuned kernels; never runtime-autotune.
```
pixi
```
— environment setup for running the harness.

Common Mistakes

Mistake	Fix
Guessing the metric/split/budget instead of gating	Run `superpowers:brainstorming` first; resolve all six items
`eval_forward` shape mismatch (silently uncomparable metric)	Read `reference/contracts.md` ; return what the harness scorer's metric consumes
Logging only a final number	Use the two-tier `dynamics.py` recorder (snapshot header + wide body)
Editing `launcher.py` to manufacture a win	Launcher is FIXED; relaxing keep/discard or the parse is cheating
"Deriving" held-out targets in train.py	Leakage. Touch the held-out set only via `ctx.score(...)`
Reading log files or dynamics CSVs in the main orchestrator loop	Delegate to the batch subagent; the main context should only ever see compact summaries
Batch subagent returning raw log excerpts in its summary	The summary must match the compact schema — one line per variant plus one dynamics line per variant, nothing else

autoresearch-setup

NPX Install

Tags

SKILL.md Content