harbor-cli

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Harbor CLI Reference

Harbor CLI 参考文档

Harbor CLI manages the full evaluation lifecycle: creating tasks, running agents, viewing results. Install with

uv tool install harbor

. Global option:

--version

-v

For complete flag tables with types and defaults for every command, read

references/flags.md

Harbor CLI 管理完整的评估生命周期：创建任务、运行Agent、查看结果。使用

uv tool install harbor

进行安装。全局选项：

--version

-v

。

如需每个命令包含类型和默认值的完整参数表，请阅读

references/flags.md

。

Quick Reference

快速参考

Command	Description
`harbor run`	Run evaluations (alias for `harbor jobs start` )
`harbor jobs start`	Start an evaluation job
`harbor jobs resume`	Resume an interrupted job
`harbor jobs summarize`	AI-powered failure summaries for a job
`harbor trials start`	Run a single trial (debugging)
`harbor trials summarize`	AI-powered summary of a single trial
`harbor datasets list`	List available datasets
`harbor datasets download`	Download a dataset
`harbor adapters init`	Scaffold a new benchmark adapter
`harbor adapters review`	Structural + AI review of an adapter
`harbor tasks init`	Scaffold a new task
`harbor tasks check`	AI quality assessment of a task
`harbor tasks start-env`	Launch task environment interactively
`harbor tasks debug`	Analyze failing trials for a task
`harbor tasks migrate`	Convert Terminal-Bench tasks to Harbor
`harbor view`	Browse job/trial results in web UI
`harbor sweeps run`	Run successive evaluation sweeps
`harbor traces export`	Export trace data in ATIF format
`harbor cache clean`	Clean Docker images and cache directory
`harbor admin`	Administrative commands (hidden)

命令	描述
`harbor run`	运行评估（ `harbor jobs start` 的别名）
`harbor jobs start`	启动评估任务
`harbor jobs resume`	恢复中断的任务
`harbor jobs summarize`	基于AI生成任务失败总结
`harbor trials start`	运行单次测试（用于调试）
`harbor trials summarize`	基于AI生成单次测试总结
`harbor datasets list`	列出可用数据集
`harbor datasets download`	下载数据集
`harbor adapters init`	生成新基准适配器的脚手架
`harbor adapters review`	对适配器进行结构化+AI审核
`harbor tasks init`	生成新任务的脚手架
`harbor tasks check`	基于AI对任务进行质量评估
`harbor tasks start-env`	交互式启动任务环境
`harbor tasks debug`	分析任务中的失败测试
`harbor tasks migrate`	将Terminal-Bench任务转换为Harbor格式
`harbor view`	在Web UI中浏览任务/测试结果
`harbor sweeps run`	运行连续评估扫描
`harbor traces export`	以ATIF格式导出追踪数据
`harbor cache clean`	清理Docker镜像和缓存目录
`harbor admin`	管理命令（隐藏）

harbor run / harbor jobs start

The primary command.

harbor run

is an alias for

harbor jobs start

Defaults: agent =

oracle

, n-concurrent =

, environment =

docker

, output =

./jobs

Essential flags:

Flag	Short	Description
`--path`	`-p`	Local path to task directory
`--dataset`	`-d`	Dataset from registry (e.g., `terminal-bench@2.0` )
`--config`	`-c`	Job config YAML/JSON file
`--agent`	`-a`	Agent name (default: `oracle` ). Repeatable
`--model`	`-m`	Model identifier. Repeatable
`--n-concurrent`	`-n`	Concurrent trials (default: `1` )
`--env`	`-e`	Environment backend (default: `docker` )
`--jobs-dir`	`-o`	Output directory (default: `./jobs` )
`--agent-env`	`--ae`	Pass env var to agent: `KEY=VALUE` . Repeatable
`--agent-kwarg`	`--ak`	Agent kwarg: `key=value` . Repeatable
`--task-name`	`-t`	Include tasks by glob pattern. Repeatable
`--exclude-task-name`	`-x`	Exclude tasks by glob. Repeatable
`--n-tasks`	`-l`	Max tasks to run
`--quiet`	`-q`	Suppress trial progress
`--debug`		Enable debug logging
`--dry-run`		Print config and exit
`--disable-verification`		Skip running tests
`--no-cache`		Skip Docker build cache
`--artifact`		Download path from env after trial. Repeatable
`--timeout-multiplier`		Scale all timeouts (default: `1.0` )

Additional flags for environment kwargs (

--ek

), environment env vars (

--ee

), verifier kwargs (

--vk

), orchestrator kwargs (

--ok

), agent images, retry config, trace export, and per-phase timeout multipliers are in

references/flags.md

核心命令。

harbor run

是

harbor jobs start

的别名。

默认值： agent =

oracle

，n-concurrent =

，environment =

docker

，output =

./jobs

。

关键参数：

参数	简写	描述
`--path`	`-p`	任务目录的本地路径
`--dataset`	`-d`	注册表中的数据集（例如： `terminal-bench@2.0` ）
`--config`	`-c`	任务配置YAML/JSON文件
`--agent`	`-a`	Agent名称（默认： `oracle` ）。可重复设置
`--model`	`-m`	模型标识符。可重复设置
`--n-concurrent`	`-n`	并发测试数（默认： `1` ）
`--env`	`-e`	环境后端（默认： `docker` ）
`--jobs-dir`	`-o`	输出目录（默认： `./jobs` ）
`--agent-env`	`--ae`	向Agent传递环境变量： `KEY=VALUE` 。可重复设置
`--agent-kwarg`	`--ak`	Agent关键字参数： `key=value` 。可重复设置
`--task-name`	`-t`	通过通配符模式包含任务。可重复设置
`--exclude-task-name`	`-x`	通过通配符模式排除任务。可重复设置
`--n-tasks`	`-l`	最大运行任务数
`--quiet`	`-q`	隐藏测试进度
`--debug`		启用调试日志
`--dry-run`		打印配置后退出
`--disable-verification`		跳过测试运行
`--no-cache`		跳过Docker构建缓存
`--artifact`		测试后从环境下载路径。可重复设置
`--timeout-multiplier`		缩放所有超时时间（默认： `1.0` ）

关于环境关键字参数（

--ek

）、环境变量（

--ee

）、验证器关键字参数（

--vk

）、编排器关键字参数（

--ok

）、Agent镜像、重试配置、追踪导出和各阶段超时乘数的额外参数，请查看

references/flags.md

。

Examples

示例

bash

undefined

bash

undefined

Run local task with oracle (validates solution + tests)

使用oracle运行本地任务（验证解决方案+测试）

harbor run -p ./my-task

Run with a real agent

使用真实Agent运行

harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1

Dataset evaluation with concurrency

带并发的数据集评估

harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8

Cloud environment

云环境

harbor run -d my-dataset -a claude-code -m anthropic/claude-sonnet-4-1 -e daytona

From config file

从配置文件运行

harbor run -c eval-config.yaml

Pass API key to agent

向Agent传递API密钥

harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY

Run subset of tasks

运行任务子集

harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1
-t "bash-*" -l 10

Dry run

试运行

harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 --dry-run

undefined

harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 --dry-run

undefined

harbor jobs resume

Resume an interrupted or partially completed job.

bash

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError

Flag	Short	Description
`--job-path`	`-p`	Path to job directory with `config.json` (required)
`--filter-error-type`	`-f`	Remove trials matching this error type before resuming. Repeatable. Default: `CancelledError`

恢复中断或未完成的任务。

bash

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError

参数	简写	描述
`--job-path`	`-p`	包含 `config.json` 的任务目录路径（必填）
`--filter-error-type`	`-f`	恢复前移除匹配该错误类型的测试。可重复设置。默认： `CancelledError`

harbor jobs summarize

Generate AI-powered failure summaries for trials in a job.

bash

harbor jobs summarize ./jobs/my-job
harbor jobs summarize ./jobs/my-job -m sonnet --all --overwrite

Flag	Short	Description
`job_path`		Path to job dir or parent (positional)
`--model`	`-m`	Model: `haiku` , `sonnet` , `opus` (default: `haiku` )
`--n-concurrent`	`-n`	Max concurrent queries (default: `5` )
`--all` / `--failed`		Analyze all or only failed trials (default: `--failed` )
`--overwrite`		Overwrite existing `summary.md` files

为任务中的测试生成基于AI的失败总结。

bash

harbor jobs summarize ./jobs/my-job
harbor jobs summarize ./jobs/my-job -m sonnet --all --overwrite

参数	简写	描述
`job_path`		任务目录或父目录路径（位置参数）
`--model`	`-m`	模型： `haiku` 、 `sonnet` 、 `opus` （默认： `haiku` ）
`--n-concurrent`	`-n`	最大并发查询数（默认： `5` ）
`--all` / `--failed`		分析所有测试或仅失败测试（默认： `--failed` ）
`--overwrite`		覆盖现有 `summary.md` 文件

harbor trials start

Run a single trial. Useful for debugging and task development.

Key difference from
harbor jobs start
: the environment flag is

--environment-type

(not

--env

), and output goes to

--trials-dir

(default:

./trials

Flag	Short	Description
`--path`	`-p`	Path to local task directory
`--config`	`-c`	Trial config YAML/JSON
`--agent`	`-a`	Agent name (default: `oracle` )
`--model`	`-m`	Model for the agent
`--environment-type`	`-e`	Environment type (default: `docker` )
`--trials-dir`		Output directory (default: `./trials` )
`--agent-env`	`--ae`	Env var for agent: `KEY=VALUE` . Repeatable
`--agent-kwarg`	`--ak`	Agent kwarg: `key=value` . Repeatable
`--no-cleanup`		Keep environment after trial
`--no-verify`		Skip running tests

Full flag list (task kwargs, git options, etc.) in

references/flags.md

bash

undefined

运行单次测试。适用于调试和任务开发。

与
harbor jobs start
的关键区别：环境参数为

--environment-type

（而非

--env

），输出路径为

--trials-dir

（默认：

./trials

）。

参数	简写	描述
`--path`	`-p`	本地任务目录路径
`--config`	`-c`	测试配置YAML/JSON
`--agent`	`-a`	Agent名称（默认： `oracle` ）
`--model`	`-m`	Agent使用的模型
`--environment-type`	`-e`	环境类型（默认： `docker` ）
`--trials-dir`		输出目录（默认： `./trials` ）
`--agent-env`	`--ae`	Agent的环境变量： `KEY=VALUE` 。可重复设置
`--agent-kwarg`	`--ak`	Agent的关键字参数： `key=value` 。可重复设置
`--no-cleanup`		测试后保留环境
`--no-verify`		跳过测试运行

完整参数列表（任务关键字参数、Git选项等）请查看

references/flags.md

。

bash

undefined

Test with oracle (validates solution + tests)

使用oracle测试（验证解决方案+测试）

harbor trials start -p ./my-task

Test with real agent

使用真实Agent测试

harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1

Keep container for inspection

保留容器以便检查

harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 --no-cleanup


The oracle agent runs `solution/solve.sh` inside the environment, then the verifier runs `tests/test.sh`. If oracle does not get reward 1.0, your tests or solution have a bug.

harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 --no-cleanup


Oracle Agent会在环境中运行`solution/solve.sh`，然后验证器运行`tests/test.sh`。如果Oracle未获得1.0的奖励，说明你的测试或解决方案存在bug。

harbor trials summarize

bash

harbor trials summarize ./trials/my-trial
harbor trials summarize ./trials/my-trial -m sonnet --overwrite

Flag	Short	Description
`trial_path`		Path to trial directory (positional)
`--model`	`-m`	Model: `haiku` , `sonnet` , `opus` (default: `haiku` )
`--overwrite`		Overwrite existing `summary.md`

bash

harbor trials summarize ./trials/my-trial
harbor trials summarize ./trials/my-trial -m sonnet --overwrite

参数	简写	描述
`trial_path`		测试目录路径（位置参数）
`--model`	`-m`	模型： `haiku` 、 `sonnet` 、 `opus` （默认： `haiku` ）
`--overwrite`		覆盖现有 `summary.md`

harbor datasets

harbor datasets list

bash

harbor datasets list
harbor datasets list --registry-url https://custom-registry.example.com
harbor datasets list --registry-path ./local-registry.json

Flags

--registry-url

and

--registry-path

are mutually exclusive. Default: Harbor's public registry.

bash

harbor datasets list
harbor datasets list --registry-url https://custom-registry.example.com
harbor datasets list --registry-path ./local-registry.json

参数

--registry-url

和

--registry-path

互斥。默认：Harbor公共注册表。

harbor datasets download

bash

harbor datasets download terminal-bench@2.0
harbor datasets download terminal-bench@2.0 -o ./my-tasks --overwrite

Flag	Short	Description
`DATASET`		Name or `name@version` (positional)
`--output-dir`	`-o`	Download dir (default: `~/.cache/harbor/tasks` )
`--overwrite`		Re-download even if cached

bash

harbor datasets download terminal-bench@2.0
harbor datasets download terminal-bench@2.0 -o ./my-tasks --overwrite

参数	简写	描述
`DATASET`		名称或 `name@version` （位置参数）
`--output-dir`	`-o`	下载目录（默认： `~/.cache/harbor/tasks` ）
`--overwrite`		即使已缓存也重新下载

harbor adapters

harbor adapters init

bash

harbor adapters init my-benchmark

Interactive wizard that prompts for benchmark name, adapter ID, class name, description, source URL, and license. Creates the adapter directory with template files.

bash

harbor adapters init my-benchmark

交互式向导，提示输入基准名称、适配器ID、类名、描述、源URL和许可证。创建包含模板文件的适配器目录。

harbor adapters review

bash

harbor adapters review -p adapters/my-benchmark
harbor adapters review -p adapters/my-benchmark --skip-ai -o report.md

bash

harbor adapters review -p adapters/my-benchmark
harbor adapters review -p adapters/my-benchmark --skip-ai -o report.md

harbor tasks

harbor tasks init

bash

harbor tasks init my-task
harbor tasks init my-task -p ./custom-tasks

Flag	Short	Description
`name`		Task name (positional)
`--tasks-dir`	`-p`	Output directory (default: `.` )
`--no-pytest`		Skip pytest in test.sh
`--no-solution`		Skip solution directory
`--include-canary-strings`		Add anti-cheating canary strings
`--include-standard-metadata`		Add full metadata template to task.toml

bash

harbor tasks init my-task
harbor tasks init my-task -p ./custom-tasks

参数	简写	描述
`name`		任务名称（位置参数）
`--tasks-dir`	`-p`	输出目录（默认： `.` ）
`--no-pytest`		在test.sh中跳过pytest
`--no-solution`		跳过解决方案目录
`--include-canary-strings`		添加反作弊金丝雀字符串
`--include-standard-metadata`		向task.toml添加完整元数据模板

harbor tasks check

AI-powered quality assessment against a rubric.

bash

harbor tasks check ./my-task
harbor tasks check ./my-task -m opus -o results.json
harbor tasks check ./my-task -r custom-rubric.toml

Flag	Short	Description
`task`		Task name or path (positional)
`--model`	`-m`	Model: `sonnet` , `opus` , `haiku` (default: `sonnet` )
`--output-path`	`-o`	Write JSON results here
`--rubric_path`	`-r`	Custom rubric ( `.toml` , `.yaml` , `.yml` , `.json` )

基于评分准则的AI驱动质量评估。

bash

harbor tasks check ./my-task
harbor tasks check ./my-task -m opus -o results.json
harbor tasks check ./my-task -r custom-rubric.toml

参数	简写	描述
`task`		任务名称或路径（位置参数）
`--model`	`-m`	模型： `sonnet` 、 `opus` 、 `haiku` （默认： `sonnet` ）
`--output-path`	`-o`	将JSON结果写入此处
`--rubric_path`	`-r`	自定义评分准则（ `.toml` 、 `.yaml` 、 `.yml` 、 `.json` ）

harbor tasks start-env

Launch task environment interactively for manual inspection.

bash

harbor tasks start-env -p ./my-task
harbor tasks start-env -p ./my-task -e daytona
harbor tasks start-env -p ./my-task --agent claude-code -m anthropic/claude-sonnet-4-1

Flag	Short	Description
`--path`	`-p`	Path to task directory
`--env`	`-e`	Environment type (default: `docker` )
`--all`	`-a`	Include solution and tests (default: `true` )
`--interactive` / `--non-interactive`	`-i`	Interactive shell (default: `true` )
`--agent`		Agent to install in environment
`--model`	`-m`	Model for the agent

Full flag list in

references/flags.md

交互式启动任务环境以便手动检查。

bash

harbor tasks start-env -p ./my-task
harbor tasks start-env -p ./my-task -e daytona
harbor tasks start-env -p ./my-task --agent claude-code -m anthropic/claude-sonnet-4-1

参数	简写	描述
`--path`	`-p`	任务目录路径
`--env`	`-e`	环境类型（默认： `docker` ）
`--all`	`-a`	包含解决方案和测试（默认： `true` ）
`--interactive` / `--non-interactive`	`-i`	交互式shell（默认： `true` ）
`--agent`		在环境中安装的Agent
`--model`	`-m`	Agent使用的模型

完整参数列表请查看

references/flags.md

。

harbor tasks debug

Analyze failing trials for a task using AI.

bash

harbor tasks debug my-task-id -m sonnet
harbor tasks debug my-task-id --job-id my-job --jobs-dir ./jobs

Flag	Short	Description
`task_id`		Task ID (positional)
`--model`	`-m`	Model for analysis
`--job-id`		Specific job to analyze
`--jobs-dir`		Directory containing jobs

使用AI分析任务中的失败测试。

bash

harbor tasks debug my-task-id -m sonnet
harbor tasks debug my-task-id --job-id my-job --jobs-dir ./jobs

参数	简写	描述
`task_id`		任务ID（位置参数）
`--model`	`-m`	用于分析的模型
`--job-id`		要分析的特定任务
`--jobs-dir`		包含任务的目录

harbor tasks migrate

Convert Terminal-Bench tasks to Harbor format.

bash

harbor tasks migrate -i ./old-tasks -o ./harbor-tasks
harbor tasks migrate -i ./old-tasks -o ./harbor-tasks --cpus 2 --memory-mb 4096

Flag	Short	Description
`--input`	`-i`	Terminal-Bench task dir or parent
`--output`	`-o`	Output directory
`--cpus`		Override CPUs for all tasks
`--memory-mb`		Override memory (MB)
`--storage-mb`		Override storage (MB)
`--gpus`		Override GPUs

将Terminal-Bench任务转换为Harbor格式。

bash

harbor tasks migrate -i ./old-tasks -o ./harbor-tasks
harbor tasks migrate -i ./old-tasks -o ./harbor-tasks --cpus 2 --memory-mb 4096

参数	简写	描述
`--input`	`-i`	Terminal-Bench任务目录或父目录
`--output`	`-o`	输出目录
`--cpus`		覆盖所有任务的CPU配置
`--memory-mb`		覆盖内存配置（MB）
`--storage-mb`		覆盖存储配置（MB）
`--gpus`		覆盖GPU配置

harbor view

Browse job/trial results in a web UI.

bash

harbor view ./jobs/my-job
harbor view ./jobs --port 9000
harbor view ./jobs --dev

Flag	Short	Description
`folder`		Directory with trajectories (positional)
`--port`	`-p`	Port or range (default: `8080-8089` )
`--host`		Bind host (default: `127.0.0.1` )
`--dev`		Hot-reload development mode
`--build`		Force rebuild viewer
`--no-build`		Skip auto-build

Shows job summaries, per-trial reward values, agent trajectories, and test output.

在Web UI中浏览任务/测试结果。

bash

harbor view ./jobs/my-job
harbor view ./jobs --port 9000
harbor view ./jobs --dev

参数	简写	描述
`folder`		包含轨迹的目录（位置参数）
`--port`	`-p`	端口或端口范围（默认： `8080-8089` ）
`--host`		绑定主机（默认： `127.0.0.1` ）
`--dev`		热重载开发模式
`--build`		强制重新构建查看器
`--no-build`		跳过自动构建

展示任务总结、单测试奖励值、Agent轨迹和测试输出。

harbor sweeps run

Run successive sweeps. Each sweep drops tasks that already have at least one success, focusing effort on remaining failures.

bash

harbor sweeps run -c sweep-config.yaml
harbor sweeps run -c sweep-config.yaml --max-sweeps 5 --trials-per-task 3
harbor sweeps run -c sweep-config.yaml --hint "Try grep to find the file first"

Flag	Short	Description
`--config`	`-c`	Job config file (YAML/JSON)
`--max-sweeps`		Max sweeps (default: `3` )
`--trials-per-task`		Trials per task per sweep (default: `2` )
`--hint`		Hint string passed to agent kwargs
`--hints-file`		JSON mapping task name to hint
`--push` / `--no-push`		Push exported datasets to HF Hub

Additional export flags in

references/flags.md

运行连续扫描。每次扫描会排除已至少成功一次的任务，专注于剩余失败任务。

bash

harbor sweeps run -c sweep-config.yaml
harbor sweeps run -c sweep-config.yaml --max-sweeps 5 --trials-per-task 3
harbor sweeps run -c sweep-config.yaml --hint "Try grep to find the file first"

参数	简写	描述
`--config`	`-c`	任务配置文件（YAML/JSON）
`--max-sweeps`		最大扫描次数（默认： `3` ）
`--trials-per-task`		每次扫描每个任务的测试次数（默认： `2` ）
`--hint`		传递给Agent关键字参数的提示字符串
`--hints-file`		任务名称到提示的JSON映射
`--push` / `--no-push`		将导出的数据集推送到HF Hub

导出相关额外参数请查看

references/flags.md

。

harbor traces export

Export ATIF trajectories as training datasets.

bash

harbor traces export -p ./jobs/my-job
harbor traces export -p ./trials --filter success --push --repo my-org/traces
harbor traces export -p ./jobs/my-job --sharegpt --episodes last

Flag	Short	Description
`--path`	`-p`	Path to trial dir or root containing trials
`--recursive` / `--no-recursive`		Search recursively (default: recursive)
`--episodes`		`all` or `last` per trial (default: `all` )
`--filter`		`success` , `failure` , or `all` (default: `all` )
`--sharegpt` / `--no-sharegpt`		ShareGPT-formatted conversations
`--push` / `--no-push`		Push to Hugging Face Hub
`--repo`		HF repo id ( `org/name` )

Additional flags (

--subagents

--instruction-metadata

--verifier-metadata

--verbose

) in

references/flags.md

将ATIF轨迹导出为训练数据集。

bash

harbor traces export -p ./jobs/my-job
harbor traces export -p ./trials --filter success --push --repo my-org/traces
harbor traces export -p ./jobs/my-job --sharegpt --episodes last

参数	简写	描述
`--path`	`-p`	测试目录或包含测试的根目录
`--recursive` / `--no-recursive`		递归搜索（默认：递归）
`--episodes`		每个测试的 `all` 或 `last` （默认： `all` ）
`--filter`		`success` 、 `failure` 或 `all` （默认： `all` ）
`--sharegpt` / `--no-sharegpt`		ShareGPT格式的对话
`--push` / `--no-push`		推送到Hugging Face Hub
`--repo`		HF仓库ID（ `org/name` ）

额外参数（

--subagents

、

--instruction-metadata

、

--verifier-metadata

、

--verbose

）请查看

references/flags.md

。

harbor cache clean

Remove Harbor Docker images and

~/.cache/harbor

bash

harbor cache clean
harbor cache clean --force
harbor cache clean --dry
harbor cache clean --no-docker      # Only clean ~/.cache/harbor
harbor cache clean --no-cache-dir   # Only clean Docker images

Flag	Short	Description
`--force`	`-f`	Skip confirmation
`--dry`		Preview without deleting
`--no-docker`		Skip Docker image removal
`--no-cache-dir`		Skip `~/.cache/harbor` removal

移除Harbor Docker镜像和

~/.cache/harbor

。

bash

harbor cache clean
harbor cache clean --force
harbor cache clean --dry
harbor cache clean --no-docker      # 仅清理~/.cache/harbor
harbor cache clean --no-cache-dir   # 仅清理Docker镜像

参数	简写	描述
`--force`	`-f`	跳过确认
`--dry`		预览删除内容但不执行
`--no-docker`		跳过Docker镜像移除
`--no-cache-dir`		跳过 `~/.cache/harbor` 移除

harbor admin upload-images

Build and push task Docker images to a container registry. Updates

task.toml

with

docker_image

for pre-built image workflows. Hidden from

harbor --help

but fully functional.

bash

harbor admin upload-images -r my-registry.io --tag 20260317
harbor admin upload-images -r my-registry.io --dry-run -f "bash-*"

Key flags:

--tasks-dir / -t

(default:

tasks

--registry / -r

(required),

--tag

(default: today

YYYYMMDD

--dry-run

--filter / -f

--push/--no-push

(default: push),

--delete/--no-delete

--parallel / -n

(default:

--update-config/--no-update-config

--override-config/--no-override-config

. Full details in

references/flags.md

构建并推送任务Docker镜像到容器注册表。更新

task.toml

中的

docker_image

字段以支持预构建镜像工作流。该命令在

harbor --help

中隐藏，但功能完整。

bash

harbor admin upload-images -r my-registry.io --tag 20260317
harbor admin upload-images -r my-registry.io --dry-run -f "bash-*"

关键参数：

--tasks-dir / -t

（默认：

tasks

）、

--registry / -r

（必填）、

--tag

（默认：当日

YYYYMMDD

）、

--dry-run

、

--filter / -f

、

--push/--no-push

（默认：推送）、

--delete/--no-delete

、

--parallel / -n

（默认：

）、

--update-config/--no-update-config

、

--override-config/--no-override-config

。完整详情请查看

references/flags.md

。

Supported Agents

支持的Agent

Agent	CLI value	Notes
Oracle	`oracle`	Runs `solution/solve.sh` . Default agent
Nop	`nop`	Does nothing (baseline)
Claude Code	`claude-code`	Needs `ANTHROPIC_API_KEY`
Aider	`aider`
Codex	`codex`
Cline CLI	`cline-cli`	Model format: `provider:model-id`
Cursor CLI	`cursor-cli`
Gemini CLI	`gemini-cli`	Needs `GOOGLE_API_KEY` or `GEMINI_API_KEY`
Goose	`goose`	Model format: `provider/model_name`
Mini SWE-agent	`mini-swe-agent`
SWE-agent	`swe-agent`
OpenCode	`opencode`
OpenHands	`openhands`	Needs `LLM_API_KEY` and `LLM_MODEL`
OpenHands SDK	`openhands-sdk`
Qwen Coder	`qwen-coder`
Terminus	`terminus`	Harbor's built-in agent
Terminus 1	`terminus-1`
Terminus 2	`terminus-2`

Agent	CLI值	说明
Oracle	`oracle`	运行 `solution/solve.sh` 。默认Agent
Nop	`nop`	无操作（基准测试）
Claude Code	`claude-code`	需要 `ANTHROPIC_API_KEY`
Aider	`aider`
Codex	`codex`
Cline CLI	`cline-cli`	模型格式： `provider:model-id`
Cursor CLI	`cursor-cli`
Gemini CLI	`gemini-cli`	需要 `GOOGLE_API_KEY` 或 `GEMINI_API_KEY`
Goose	`goose`	模型格式： `provider/model_name`
Mini SWE-agent	`mini-swe-agent`
SWE-agent	`swe-agent`
OpenCode	`opencode`
OpenHands	`openhands`	需要 `LLM_API_KEY` 和 `LLM_MODEL`
OpenHands SDK	`openhands-sdk`
Qwen Coder	`qwen-coder`
Terminus	`terminus`	Harbor内置Agent
Terminus 1	`terminus-1`
Terminus 2	`terminus-2`

Environment Backends

环境后端

Backend	CLI value	Description
Docker	`docker`	Default. Local containers
Daytona	`daytona`	Cloud sandbox with Docker-in-Docker
E2B	`e2b`	E2B cloud sandboxes
GKE	`gke`	Google Kubernetes Engine
Modal	`modal`	Serverless with GPU support
Runloop	`runloop`	Runloop cloud environments

Note:

harbor run

uses

--env

-e

harbor trials start

uses

--environment-type

-e

后端	CLI值	描述
Docker	`docker`	默认。本地容器
Daytona	`daytona`	支持Docker-in-Docker的云沙箱
E2B	`e2b`	E2B云沙箱
GKE	`gke`	Google Kubernetes Engine
Modal	`modal`	支持GPU的无服务器环境
Runloop	`runloop`	Runloop云环境

注意：

harbor run

使用

--env

-e

。

harbor trials start

使用

--environment-type

-e

。

Common Workflows

常见工作流

Create and test a new task

创建并测试新任务

bash

harbor tasks init my-task

bash

harbor tasks init my-task

Edit: task.toml, instruction.md, Dockerfile, tests/test.sh, solution/solve.sh

编辑：task.toml、instruction.md、Dockerfile、tests/test.sh、solution/solve.sh

harbor tasks check ./my-task harbor trials start -p ./my-task # oracle validates tests harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 harbor view ./trials

undefined

harbor tasks check ./my-task harbor trials start -p ./my-task # oracle验证测试 harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 harbor view ./trials

undefined

Run a benchmark evaluation

运行基准评估

bash

harbor datasets list
harbor datasets download terminal-bench@2.0
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
harbor view ./jobs
harbor traces export -p ./jobs/my-job --push --repo my-org/traces

bash

harbor datasets list
harbor datasets download terminal-bench@2.0
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
harbor view ./jobs
harbor traces export -p ./jobs/my-job --push --repo my-org/traces

Debug a failing task

调试失败任务

bash

harbor tasks start-env -p ./my-task         # interactive shell in the container

bash

harbor tasks start-env -p ./my-task         # 在容器中启动交互式shell

Inside: bash /tests/test.sh && bash /solution/solve.sh

容器内执行：bash /tests/test.sh && bash /solution/solve.sh

harbor tasks debug my-task-id -m sonnet # AI analysis of failures harbor trials start -p ./my-task # re-test with oracle

undefined

harbor tasks debug my-task-id -m sonnet # AI分析失败原因 harbor trials start -p ./my-task # 使用oracle重新测试

undefined

Resume a failed job

恢复失败任务

bash

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
harbor jobs summarize ./jobs/my-job

bash

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
harbor jobs summarize ./jobs/my-job

Common Gotchas

常见注意事项

API keys: Most agents need keys passed via

--ae

. Claude Code needs

ANTHROPIC_API_KEY

, OpenHands needs

LLM_API_KEY

, Goose needs provider-specific keys. If the agent fails immediately, check the key.

Docker must be running. Harbor uses Docker for sandboxed environments. Network exhaustion from many concurrent trials can cause failures --

harbor cache clean

helps.

Model name format varies by agent. Claude Code uses

anthropic/claude-sonnet-4-1

, Cline CLI uses

provider:model-id

(e.g.,

openrouter:anthropic/claude-opus-4.5

), Goose uses

provider/model_name

. A

ValueError

about model format usually means the wrong format for that agent.

--env
vs
--environment-type
:

harbor run

uses

--env

-e

harbor trials start

uses

--environment-type

-e

. Same short flag, different long name.

Default agent is oracle. Forgetting

-a

gives you the oracle agent, which just runs

solve.sh

. Fine for task validation, not for real evaluations.

--no-cleanup
for debugging. Pass this to

harbor trials start

to keep the container running after a failed trial so you can inspect it.

Retry defaults skip common errors.

--retry-exclude

defaults to

AgentTimeoutError

VerifierTimeoutError

RewardFileNotFoundError

RewardFileEmptyError

VerifierOutputParseError

. These usually indicate task bugs, not transient failures.

API密钥： 大多数Agent需要通过

--ae

传递密钥。Claude Code需要

ANTHROPIC_API_KEY

，OpenHands需要

LLM_API_KEY

，Goose需要提供商特定密钥。如果Agent立即失败，请检查密钥是否正确。

Docker必须运行。 Harbor使用Docker提供沙箱环境。大量并发测试可能导致网络耗尽，

harbor cache clean

可解决此问题。

模型名称格式因Agent而异。 Claude Code使用

anthropic/claude-sonnet-4-1

，Cline CLI使用

provider:model-id

（例如：

openrouter:anthropic/claude-opus-4.5

），Goose使用

provider/model_name

。关于模型格式的

ValueError

通常意味着该Agent使用了错误的格式。

--env
vs
--environment-type
：

harbor run

使用

--env

-e

。

harbor trials start

使用

--environment-type

-e

。简写参数相同，但完整参数名不同。

默认Agent是oracle。 忘记添加

-a

参数会使用oracle Agent，它仅运行

solve.sh

。适用于任务验证，但不适用于真实评估。

--no-cleanup
用于调试。向

harbor trials start

传递该参数可在测试失败后保持容器运行，以便检查。

重试默认跳过常见错误。

--retry-exclude

默认排除

AgentTimeoutError

、

VerifierTimeoutError

、

RewardFileNotFoundError

、

RewardFileEmptyError

、

VerifierOutputParseError

。这些错误通常表示任务存在bug，而非临时故障。