physicalai-train-benchmarking-a-policy

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Benchmarking a Studio Policy

Studio策略的基准测试

Benchmarking evaluates a trained policy by rolling it out in a gym and scoring success. Benchmark classes live in

library/src/physicalai/benchmark/gyms/benchmark.py

(

Benchmark

PushTBenchmark

LiberoBenchmark

); results types in

benchmark/gyms/results.py

(

BenchmarkResults

TaskResult

); rollout logic in

library/src/physicalai/eval/rollout.py

(

evaluate_policy

). The library supports both direct Python API use and the

physicalai benchmark

CLI wrapper (

library/src/physicalai/cli/benchmark.py

基准测试通过在gym环境中运行训练好的策略并对成功情况打分来评估策略。Benchmark类位于

library/src/physicalai/benchmark/gyms/benchmark.py

（包含

Benchmark

、

PushTBenchmark

、

LiberoBenchmark

）；结果类型定义在

benchmark/gyms/results.py

（包含

BenchmarkResults

、

TaskResult

）；rollout逻辑在

library/src/physicalai/eval/rollout.py

（

evaluate_policy

函数）。该库既支持直接调用Python API，也支持

physicalai benchmark

CLI封装器（位于

library/src/physicalai/cli/benchmark.py

）。

Python API invocation

Python API调用

Use this path for notebooks, tests, custom scripts, or direct library integrations.

python

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.policies import ACT

policy = ACT.load_from_checkpoint("experiments/act/version_0/checkpoints/last.ckpt")
benchmark = PushTBenchmark(num_episodes=1)
results = benchmark.evaluate(policy)
print(results.summary())
results.to_json("results/benchmark/results.json")
results.to_csv("results/benchmark/results.csv")

For exported artifacts, load the Runtime-facing model first:

python

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.inference import InferenceModel

model = InferenceModel("./exports/act_policy")
results = PushTBenchmark(num_episodes=1).evaluate(model)

适用于Notebook、测试、自定义脚本或直接的库集成场景。

python

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.policies import ACT

policy = ACT.load_from_checkpoint("experiments/act/version_0/checkpoints/last.ckpt")
benchmark = PushTBenchmark(num_episodes=1)
results = benchmark.evaluate(policy)
print(results.summary())
results.to_json("results/benchmark/results.json")
results.to_csv("results/benchmark/results.csv")

对于导出的模型文件，需先加载面向Runtime的模型：

python

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.inference import InferenceModel

model = InferenceModel("./exports/act_policy")
results = PushTBenchmark(num_episodes=1).evaluate(model)

CLI invocation

CLI调用

bash

physicalai benchmark \
  --config configs/benchmark/pusht.yaml \
  --policy physicalai.policies.ACT \
  --ckpt_path experiments/act/version_0/checkpoints/last.ckpt \
  --output_dir ./results/benchmark

```
--policy
```
— policy class path.
```
--ckpt_path
```
— a
```
.ckpt
```
or an export directory.

--config

— a benchmark config (

configs/benchmark/pusht.yaml

configs/benchmark/libero.yaml

) selecting the

Benchmark

class and its settings.

```
--output_dir
```
— defaults to
```
./results/benchmark
```
.

Override benchmark settings on the CLI, e.g.

--benchmark.num_episodes 10 --benchmark.num_envs 8

bash

physicalai benchmark \
  --config configs/benchmark/pusht.yaml \
  --policy physicalai.policies.ACT \
  --ckpt_path experiments/act/version_0/checkpoints/last.ckpt \
  --output_dir ./results/benchmark

```
--policy
```
— 策略类路径。
```
--ckpt_path
```
—
```
.ckpt
```
文件路径或导出目录路径。

--config

— 基准测试配置文件（如

configs/benchmark/pusht.yaml

、

configs/benchmark/libero.yaml

），用于选择

Benchmark

类及其设置。

```
--output_dir
```
— 默认路径为
```
./results/benchmark
```
。

可通过CLI覆盖基准测试设置，例如

--benchmark.num_episodes 10 --benchmark.num_envs 8

。

Output

输出内容

Prints
```
results.summary()
```
to stdout.
Writes
```
results.json
```
and
```
results.csv
```
into
```
--output_dir
```
.

Optional video via config

video_dir

record_mode

(

all

failures

successes

none

将
```
results.summary()
```
输出到标准输出。
在
```
--output_dir
```
目录下生成
```
results.json
```
和
```
results.csv
```
文件。
可通过配置
```
video_dir
```
和
```
record_mode
```
（可选值：
```
all
```
|
```
failures
```
|
```
successes
```
|
```
none
```
）录制视频。

Workflow

工作流程

Choose API or CLI deliberately. Use the Python API for code-level tasks; use CLI for config/docs/entry-point tasks.
- Done when: the selected path matches the user's requested surface area.

Confirm the policy loads from the checkpoint/export before a full sweep:

bash

physicalai benchmark --config configs/benchmark/<suite>.yaml --policy <ClassPath> --ckpt_path <path> --benchmark.num_episodes 1

Done when: one episode runs end-to-end and a summary prints.

Run the full benchmark with the intended episode/env counts.
- Done when:
```
results.json
```
  and
```
results.csv
```
  are written and the success metric is populated.
Interpret results via
```
BenchmarkResults
```
/
```
TaskResult
```
fields; compare against a baseline checkpoint on the same config.
Record videos for qualitative review when a task regresses (
```
record_mode: failures
```
).

谨慎选择API或CLI方式。代码级任务使用Python API；配置/文档/入口点任务使用CLI。
- 完成标志：所选方式匹配用户需求场景。

在全面测试前确认策略可加载：

bash

physicalai benchmark --config configs/benchmark/<suite>.yaml --policy <ClassPath> --ckpt_path <path> --benchmark.num_episodes 1

完成标志：单轮episode完整运行并输出摘要。

使用目标episode/env数量运行完整基准测试。
- 完成标志：生成
```
results.json
```
  和
```
results.csv
```
  文件，且成功指标已填充。
通过
BenchmarkResults
/
TaskResult
字段解读结果；与相同配置下的基线checkpoint进行对比。
当任务性能退化时录制视频用于定性分析（设置
```
record_mode: failures
```
）。

Adding or changing a Benchmark

添加或修改Benchmark

Subclass

Benchmark

benchmark/gyms/

(study

PushTBenchmark

LiberoBenchmark

); the gym itself comes from

physicalai.gyms

(

pusht.py

libero.py

, …).

Add a matching config in
```
library/configs/benchmark/
```
.

Add tests under

library/tests/unit/benchmark/

Done when:
```
uv run pytest tests/unit/benchmark
```
passes and a 1-episode run succeeds.

在

benchmark/gyms/

目录下继承

Benchmark

类（可参考

PushTBenchmark

LiberoBenchmark

实现）；gym环境来自

physicalai.gyms

（如

pusht.py

、

libero.py

等）。

在
```
library/configs/benchmark/
```
目录下添加对应的配置文件。
在
```
library/tests/unit/benchmark/
```
目录下添加测试用例。
- 完成标志：
```
uv run pytest tests/unit/benchmark
```
  执行通过，且单轮episode运行成功。

Required checks

必要检查项

The policy runs from both a
```
.ckpt
```
and an export dir if both are supported paths.
The Python API path (
```
Benchmark(...).evaluate(...)
```
) and CLI wrapper agree on supported inputs for user-facing benchmark changes.
Success/episode metrics are populated (not zero/NaN by accident) and reproducible across runs.
Env/episode counts match hardware; large
```
num_envs
```
fits memory.
Heavy gym deps (e.g.
```
libero
```
,
```
robocasa
```
) are gated behind their optional extras and imported lazily.

若策略支持两种路径，则需确保从
```
.ckpt
```
文件和导出目录均可加载运行。
Python API路径（
```
Benchmark(...).evaluate(...)
```
）与CLI封装器在用户可见的基准测试变更上支持的输入需保持一致。
成功/episode指标已正确填充（避免意外为零或NaN），且多次运行结果可复现。
Env/episode数量与硬件匹配；较大的
```
num_envs
```
需符合内存限制。
重型gym依赖（如
```
libero
```
、
```
robocasa
```
）需通过可选扩展包引入，并延迟导入。

physicalai-train-benchmarking-a-policy

Original

Translation

Benchmarking a Studio Policy

Studio策略的基准测试

Python API invocation

Python API调用

CLI invocation

CLI调用

Output

输出内容

Workflow

工作流程

Adding or changing a Benchmark

添加或修改Benchmark

Required checks

必要检查项

Related skills

相关技能