Harbor CLI Reference
Harbor CLI manages the full evaluation lifecycle: creating tasks, running agents, viewing results. Install with
. Global option:
/
.
For complete flag tables with types and defaults for every command, read
.
Quick Reference
| Command | Description |
|---|
| Run evaluations (alias for ) |
| Start an evaluation job |
| Resume an interrupted job |
| AI-powered failure summaries for a job |
| Run a single trial (debugging) |
| AI-powered summary of a single trial |
| List available datasets |
| Download a dataset |
| Scaffold a new benchmark adapter |
| Structural + AI review of an adapter |
| Scaffold a new task |
| AI quality assessment of a task |
| Launch task environment interactively |
| Analyze failing trials for a task |
| Convert Terminal-Bench tasks to Harbor |
| Browse job/trial results in web UI |
| Run successive evaluation sweeps |
| Export trace data in ATIF format |
| Clean Docker images and cache directory |
| Administrative commands (hidden) |
harbor run / harbor jobs start
The primary command.
is an alias for
.
Defaults: agent =
, n-concurrent =
, environment =
, output =
.
Essential flags:
| Flag | Short | Description |
|---|
| | Local path to task directory |
| | Dataset from registry (e.g., ) |
| | Job config YAML/JSON file |
| | Agent name (default: ). Repeatable |
| | Model identifier. Repeatable |
| | Concurrent trials (default: ) |
| | Environment backend (default: ) |
| | Output directory (default: ) |
| | Pass env var to agent: . Repeatable |
| | Agent kwarg: . Repeatable |
| | Include tasks by glob pattern. Repeatable |
| | Exclude tasks by glob. Repeatable |
| | Max tasks to run |
| | Suppress trial progress |
| | Enable debug logging |
| | Print config and exit |
| | Skip running tests |
| | Skip Docker build cache |
| | Download path from env after trial. Repeatable |
| | Scale all timeouts (default: ) |
Additional flags for environment kwargs (
), environment env vars (
), verifier kwargs (
), orchestrator kwargs (
), agent images, retry config, trace export, and per-phase timeout multipliers are in
.
Examples
bash
# Run local task with oracle (validates solution + tests)
harbor run -p ./my-task
# Run with a real agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
# Dataset evaluation with concurrency
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
# Cloud environment
harbor run -d my-dataset -a claude-code -m anthropic/claude-sonnet-4-1 -e daytona
# From config file
harbor run -c eval-config.yaml
# Pass API key to agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 \
--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
# Run subset of tasks
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 \
-t "bash-*" -l 10
# Dry run
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 --dry-run
harbor jobs resume
Resume an interrupted or partially completed job.
bash
harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
| Flag | Short | Description |
|---|
| | Path to job directory with (required) |
| | Remove trials matching this error type before resuming. Repeatable. Default: |
harbor jobs summarize
Generate AI-powered failure summaries for trials in a job.
bash
harbor jobs summarize ./jobs/my-job
harbor jobs summarize ./jobs/my-job -m sonnet --all --overwrite
| Flag | Short | Description |
|---|
| | Path to job dir or parent (positional) |
| | Model: , , (default: ) |
| | Max concurrent queries (default: ) |
| / | | Analyze all or only failed trials (default: ) |
| | Overwrite existing files |
harbor trials start
Run a single trial. Useful for debugging and task development.
Key difference from : the environment flag is
(not
), and output goes to
(default:
).
| Flag | Short | Description |
|---|
| | Path to local task directory |
| | Trial config YAML/JSON |
| | Agent name (default: ) |
| | Model for the agent |
| | Environment type (default: ) |
| | Output directory (default: ) |
| | Env var for agent: . Repeatable |
| | Agent kwarg: . Repeatable |
| | Keep environment after trial |
| | Skip running tests |
Full flag list (task kwargs, git options, etc.) in
.
bash
# Test with oracle (validates solution + tests)
harbor trials start -p ./my-task
# Test with real agent
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
# Keep container for inspection
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 --no-cleanup
The oracle agent runs
inside the environment, then the verifier runs
. If oracle does not get reward 1.0, your tests or solution have a bug.
harbor trials summarize
bash
harbor trials summarize ./trials/my-trial
harbor trials summarize ./trials/my-trial -m sonnet --overwrite
| Flag | Short | Description |
|---|
| | Path to trial directory (positional) |
| | Model: , , (default: ) |
| | Overwrite existing |
harbor datasets
harbor datasets list
bash
harbor datasets list
harbor datasets list --registry-url https://custom-registry.example.com
harbor datasets list --registry-path ./local-registry.json
Flags
and
are mutually exclusive. Default: Harbor's public registry.
harbor datasets download
bash
harbor datasets download terminal-bench@2.0
harbor datasets download terminal-bench@2.0 -o ./my-tasks --overwrite
| Flag | Short | Description |
|---|
| | Name or (positional) |
| | Download dir (default: ) |
| | Re-download even if cached |
harbor adapters
harbor adapters init
bash
harbor adapters init my-benchmark
Interactive wizard that prompts for benchmark name, adapter ID, class name, description, source URL, and license. Creates the adapter directory with template files.
harbor adapters review
bash
harbor adapters review -p adapters/my-benchmark
harbor adapters review -p adapters/my-benchmark --skip-ai -o report.md
harbor tasks
harbor tasks init
bash
harbor tasks init my-task
harbor tasks init my-task -p ./custom-tasks
| Flag | Short | Description |
|---|
| | Task name (positional) |
| | Output directory (default: ) |
| | Skip pytest in test.sh |
| | Skip solution directory |
| | Add anti-cheating canary strings |
--include-standard-metadata
| | Add full metadata template to task.toml |
harbor tasks check
AI-powered quality assessment against a rubric.
bash
harbor tasks check ./my-task
harbor tasks check ./my-task -m opus -o results.json
harbor tasks check ./my-task -r custom-rubric.toml
| Flag | Short | Description |
|---|
| | Task name or path (positional) |
| | Model: , , (default: ) |
| | Write JSON results here |
| | Custom rubric (, , , ) |
harbor tasks start-env
Launch task environment interactively for manual inspection.
bash
harbor tasks start-env -p ./my-task
harbor tasks start-env -p ./my-task -e daytona
harbor tasks start-env -p ./my-task --agent claude-code -m anthropic/claude-sonnet-4-1
| Flag | Short | Description |
|---|
| | Path to task directory |
| | Environment type (default: ) |
| | Include solution and tests (default: ) |
| / | | Interactive shell (default: ) |
| | Agent to install in environment |
| | Model for the agent |
harbor tasks debug
Analyze failing trials for a task using AI.
bash
harbor tasks debug my-task-id -m sonnet
harbor tasks debug my-task-id --job-id my-job --jobs-dir ./jobs
| Flag | Short | Description |
|---|
| | Task ID (positional) |
| | Model for analysis |
| | Specific job to analyze |
| | Directory containing jobs |
harbor tasks migrate
Convert Terminal-Bench tasks to Harbor format.
bash
harbor tasks migrate -i ./old-tasks -o ./harbor-tasks
harbor tasks migrate -i ./old-tasks -o ./harbor-tasks --cpus 2 --memory-mb 4096
| Flag | Short | Description |
|---|
| | Terminal-Bench task dir or parent |
| | Output directory |
| | Override CPUs for all tasks |
| | Override memory (MB) |
| | Override storage (MB) |
| | Override GPUs |
harbor view
Browse job/trial results in a web UI.
bash
harbor view ./jobs/my-job
harbor view ./jobs --port 9000
harbor view ./jobs --dev
| Flag | Short | Description |
|---|
| | Directory with trajectories (positional) |
| | Port or range (default: ) |
| | Bind host (default: ) |
| | Hot-reload development mode |
| | Force rebuild viewer |
| | Skip auto-build |
Shows job summaries, per-trial reward values, agent trajectories, and test output.
harbor sweeps run
Run successive sweeps. Each sweep drops tasks that already have at least one success, focusing effort on remaining failures.
bash
harbor sweeps run -c sweep-config.yaml
harbor sweeps run -c sweep-config.yaml --max-sweeps 5 --trials-per-task 3
harbor sweeps run -c sweep-config.yaml --hint "Try grep to find the file first"
| Flag | Short | Description |
|---|
| | Job config file (YAML/JSON) |
| | Max sweeps (default: ) |
| | Trials per task per sweep (default: ) |
| | Hint string passed to agent kwargs |
| | JSON mapping task name to hint |
| / | | Push exported datasets to HF Hub |
Additional export flags in
.
harbor traces export
Export ATIF trajectories as training datasets.
bash
harbor traces export -p ./jobs/my-job
harbor traces export -p ./trials --filter success --push --repo my-org/traces
harbor traces export -p ./jobs/my-job --sharegpt --episodes last
| Flag | Short | Description |
|---|
| | Path to trial dir or root containing trials |
| / | | Search recursively (default: recursive) |
| | or per trial (default: ) |
| | , , or (default: ) |
| / | | ShareGPT-formatted conversations |
| / | | Push to Hugging Face Hub |
| | HF repo id () |
Additional flags (
,
,
,
) in
.
harbor cache clean
Remove Harbor Docker images and
.
bash
harbor cache clean
harbor cache clean --force
harbor cache clean --dry
harbor cache clean --no-docker # Only clean ~/.cache/harbor
harbor cache clean --no-cache-dir # Only clean Docker images
| Flag | Short | Description |
|---|
| | Skip confirmation |
| | Preview without deleting |
| | Skip Docker image removal |
| | Skip removal |
harbor admin upload-images
Build and push task Docker images to a container registry. Updates
with
for pre-built image workflows. Hidden from
but fully functional.
bash
harbor admin upload-images -r my-registry.io --tag 20260317
harbor admin upload-images -r my-registry.io --dry-run -f "bash-*"
Key flags:
(default:
),
(required),
(default: today
),
,
,
(default: push),
,
(default:
),
--update-config/--no-update-config
,
--override-config/--no-override-config
. Full details in
.
Supported Agents
| Agent | CLI value | Notes |
|---|
| Oracle | | Runs . Default agent |
| Nop | | Does nothing (baseline) |
| Claude Code | | Needs |
| Aider | | |
| Codex | | |
| Cline CLI | | Model format: |
| Cursor CLI | | |
| Gemini CLI | | Needs or |
| Goose | | Model format: |
| Mini SWE-agent | | |
| SWE-agent | | |
| OpenCode | | |
| OpenHands | | Needs and |
| OpenHands SDK | | |
| Qwen Coder | | |
| Terminus | | Harbor's built-in agent |
| Terminus 1 | | |
| Terminus 2 | | |
Environment Backends
| Backend | CLI value | Description |
|---|
| Docker | | Default. Local containers |
| Daytona | | Cloud sandbox with Docker-in-Docker |
| E2B | | E2B cloud sandboxes |
| GKE | | Google Kubernetes Engine |
| Modal | | Serverless with GPU support |
| Runloop | | Runloop cloud environments |
Common Workflows
Create and test a new task
bash
harbor tasks init my-task
# Edit: task.toml, instruction.md, Dockerfile, tests/test.sh, solution/solve.sh
harbor tasks check ./my-task
harbor trials start -p ./my-task # oracle validates tests
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
harbor view ./trials
Run a benchmark evaluation
bash
harbor datasets list
harbor datasets download terminal-bench@2.0
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
harbor view ./jobs
harbor traces export -p ./jobs/my-job --push --repo my-org/traces
Debug a failing task
bash
harbor tasks start-env -p ./my-task # interactive shell in the container
# Inside: bash /tests/test.sh && bash /solution/solve.sh
harbor tasks debug my-task-id -m sonnet # AI analysis of failures
harbor trials start -p ./my-task # re-test with oracle
Resume a failed job
bash
harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
harbor jobs summarize ./jobs/my-job
Common Gotchas
API keys: Most agents need keys passed via
. Claude Code needs
, OpenHands needs
, Goose needs provider-specific keys. If the agent fails immediately, check the key.
Docker must be running. Harbor uses Docker for sandboxed environments. Network exhaustion from many concurrent trials can cause failures --
helps.
Model name format varies by agent. Claude Code uses
anthropic/claude-sonnet-4-1
, Cline CLI uses
(e.g.,
openrouter:anthropic/claude-opus-4.5
), Goose uses
. A
about model format usually means the wrong format for that agent.
vs : uses
/
.
uses
/
. Same short flag, different long name.
Default agent is oracle. Forgetting
gives you the oracle agent, which just runs
. Fine for task validation, not for real evaluations.
for debugging. Pass this to
to keep the container running after a failed trial so you can inspect it.
Retry defaults skip common errors. defaults to
,
,
,
,
. These usually indicate task bugs, not transient failures.