agent-desktop

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

agent-desktop

agent-desktop is a CLI that lets an AI agent observe and operate desktop apps through the OS accessibility tree. It is a tool, not an agent. Every command prints one JSON envelope (

ok

data

error

). The observe, act and verify loop runs in you.

agent-desktop 是一个 CLI，可让 AI Agent 通过操作系统无障碍树观察并操作桌面应用。它是一个工具，而不是 Agent。每条命令都会打印一个 JSON 信封（

ok

、

data

或

error

）。观察、操作和验证循环由你执行。

Shell syntax

Shell 语法

Examples use a POSIX shell. In PowerShell, quote every ref (

'@s8f3k2p9:e1'

), because a bare

@token

is the splatting operator. Use

$env:NAME = 'value'

instead of

export

, and

| ConvertFrom-Json

instead of

$(...)

and

jq

. The Windows skill has the PowerShell forms.

示例使用 POSIX shell。在 PowerShell 中，必须为每个引用加引号（

'@s8f3k2p9:e1'

），因为裸

@token

会被视为 splatting 运算符。使用

$env:NAME = 'value'

代替

export

，并使用

| ConvertFrom-Json

代替

$(...)

和

jq

。Windows skill 中提供了对应的 PowerShell 写法。

First run

首次运行

Run
```
agent-desktop status
```
. It reports the platform, permissions and
```
supported_surfaces
```
.
Run
```
agent-desktop skills get platform
```
. It loads the skill for the OS the binary runs on (macOS, Windows or Linux).
For flags and syntax, run
```
agent-desktop <command> --help
```
. This skill does not repeat flag lists.

运行
```
agent-desktop status
```
。它会报告平台、权限和
```
supported_surfaces
```
。
运行
```
agent-desktop skills get platform
```
。它会为二进制文件所在的操作系统（macOS、Windows 或 Linux）加载对应的 skill。
如需了解参数和语法，请运行
```
agent-desktop <command> --help
```
。本 skill 不重复列出参数列表。

The loop

循环

Use

find

when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region

A small app needs no skeleton. Take a full
```
snapshot -i
```
.
Never skeleton a surface (
```
--surface menu
```
,
```
sheet
```
,
```
alert
```
). A surface is already focused.
An action that opens a menu, sheet or alert lists it in
```
data.surfaces
```
. Target that overlay next.
After an async step, use
```
wait
```
(see
```
commands-system.md
```
). A fixed sleep is a race.

如果你知道元素角色或确切名称，请使用

find

。否则，先梳理应用结构，再深入查看某个区域。

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region

小型应用不需要 skeleton。请执行完整的
```
snapshot -i
```
。
永远不要对 surface（
```
--surface menu
```
、
```
sheet
```
、
```
alert
```
）使用 skeleton。surface 已经处于聚焦状态。
如果操作打开了菜单、sheet 或 alert，它会列在
```
data.surfaces
```
中。下一步应定位该 overlay。
异步步骤完成后，使用
```
wait
```
（参阅
```
commands-system.md
```
）。固定等待会引入竞态条件。

Refs

引用

Output refs are qualified:
```
@<snapshot_id>:e<N>
```
, for example
```
@s8f3k2p9:e1
```
. Pass them to later commands as they are.
Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
Only addressable elements get refs. Static text and inert containers do not.
```
STALE_REF
```
: the ref no longer matches the live UI. Take a fresh snapshot and use the new ref.
```
AMBIGUOUS_TARGET
```
: more than one live element matches. Narrow the target with
```
find --role --name --exact
```
or
```
--root
```
, then use the new ref.
```
SNAPSHOT_NOT_FOUND
```
: the snapshot id expired. Snapshot again.

输出引用采用限定格式：
```
@<snapshot_id>:e<N>
```
，例如
```
@s8f3k2p9:e1
```
。请原样将其传递给后续命令。
引用仅在一次 snapshot 中有效。UI 发生任何变化后，请重新深入查看受影响的区域或获取新的 snapshot。
只有可寻址元素才会获得引用。静态文本和惰性容器不会获得引用。
```
STALE_REF
```
：引用不再与实时 UI 匹配。请获取新的 snapshot 并使用新引用。
```
AMBIGUOUS_TARGET
```
：有多个实时元素匹配。请使用
```
find --role --name --exact
```
或
```
--root
```
缩小目标范围，然后使用新引用。
```
SNAPSHOT_NOT_FOUND
```
：snapshot ID 已过期。请重新获取 snapshot。

Delivery and retry

交付与重试

Action results carry

data.disposition

and errors carry

error.disposition

, each with

delivery

and

retry

. Observation commands such as

snapshot

have none on success; their errors, and every

INVALID_ARGS

refusal, carry

not_delivered

and

retry: safe

Retry or substitute an action only when
```
disposition.retry
```
is
```
safe
```
. Only
```
not_delivered
```
is safe.
Any other delivery value means the action may have landed. A repeat can click, type or submit twice, and
```
toggle
```
can undo itself.
On a result that is not safe, read the live state (
```
get
```
,
```
is
```
, or a fresh snapshot) and decide from that.
```
details.post_state
```
helps when present.
Branch on
```
error.code
```
and
```
error.disposition
```
. Message and suggestion text can change.
```
details.retryable: false
```
means waiting cannot change the outcome. Try a different approach.

操作结果包含

data.disposition

，错误包含

error.disposition

，二者都有

delivery

和

retry

字段。

snapshot

等观察命令成功时没有这些字段；它们的错误以及每次

INVALID_ARGS

拒绝都会包含

not_delivered

和

retry: safe

。

仅当
```
disposition.retry
```
为
```
safe
```
时，才能重试或替换操作。只有
```
not_delivered
```
是安全的。
任何其他 delivery 值都意味着操作可能已经生效。重复执行可能会再次点击、输入或提交，而
```
toggle
```
甚至可能撤销自身状态。
如果结果不是安全的，请读取实时状态（
```
get
```
、
```
is
```
或新的 snapshot），然后据此决定下一步。存在
```
details.post_state
```
时，它会提供帮助。
根据
```
error.code
```
和
```
error.disposition
```
进行分支处理。消息和建议文本可能会变化。
```
details.retryable: false
```
表示等待无法改变结果。请尝试其他方法。

Headless and headed

Headless 与 Headed

Ref actions are headless by default. They use semantic accessibility calls and do not take focus, move the cursor, synthesize keys or use the clipboard. If no semantic path exists, the action fails closed. Use the global

--headed

flag only when the user explicitly permits physical input, such as double-click, hover, drag, or focus. Never switch to headed mode to recover from a headless failure. Raw coordinates (

--xy

) never imply focus. Which actions need

--headed

differs by OS: read the platform skill.

基于引用的操作默认以 headless 模式执行。它们使用语义化无障碍调用，不会获取焦点、移动光标、模拟按键或使用剪贴板。如果不存在可用的语义路径，操作会安全失败。只有当用户明确允许实体输入时，例如双击、悬停、拖拽或聚焦，才使用全局

--headed

参数。绝不能为了恢复 headless 操作失败而切换到 headed 模式。原始坐标（

--xy

）不表示会获取焦点。不同操作系统需要

--headed

的操作有所不同，请阅读平台 skill。

References

参考资料

Load one only when you need it:

agent-desktop skills get desktop <reference>

Reference	Open it when
`commands-observation`	You need `snapshot` , `find` , `get` , `is` , `screenshot` , `list-surfaces` semantics, partial trees, or the `--root` rules
`commands-interaction`	You choose between click, type, set-value, select, toggle, scroll, press and mouse commands, or you debug an action error
`commands-system`	You launch or close apps, manage windows, use the clipboard, `wait` , `batch` , sessions, multi-agent cursors, tracing, or load skills
`workflows`	You want worked patterns (forms, menus, dialogs, waiting for a new window, scroll-find, drag) and the anti-patterns list

仅在需要时加载一项：

agent-desktop skills get desktop <reference>

。

参考资料	何时打开
`commands-observation`	需要了解 `snapshot` 、 `find` 、 `get` 、 `is` 、 `screenshot` 、 `list-surfaces` 的语义、部分树或 `--root` 规则时
`commands-interaction`	需要在 click、type、set-value、select、toggle、scroll、press 和鼠标命令之间选择，或需要排查操作错误时
`commands-system`	需要启动或关闭应用、管理窗口、使用剪贴板、执行 `wait` 、 `batch` 、管理会话、多 Agent 光标、跟踪追踪信息或加载 skill 时
`workflows`	需要查看实战模式（表单、菜单、对话框、等待新窗口、滚动查找、拖拽）及其反模式列表时