agent-desktop-windows

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

agent-desktop-windows

The observe-act loop, ref system, JSON envelope, sessions and tracing are the same on every platform. This file covers only what differs on Windows.

Requires Windows 10 1809+ / Windows Server 2019+ (x64 or ARM64). Run it in the interactive desktop session of the signed-in user. Session 0 (services and SYSTEM tasks), Server Core, the secure desktop (UAC prompts, the sign-in screen), a locked desktop and another user's session are not supported: observation, actions and capture there fail or return nothing.

observe-act 循环、ref 系统、JSON envelope、会话和追踪在所有平台上均完全相同。本文件仅介绍 Windows 上的差异。

要求 Windows 10 1809+ / Windows Server 2019+（x64 或 ARM64）。应在已登录用户的交互式桌面会话中运行。不支持 Session 0（服务和 SYSTEM 任务）、Server Core、secure desktop（UAC 提示和登录界面）、锁定的桌面及其他用户的会话：在这些环境中，观察、操作和捕获会失败或不返回任何内容。

Capability Table

功能表

Group	Commands	Status on Windows
Observation	`snapshot` , `find` , `get` , `is` , `screenshot`	Works. Surfaces are `window` , `focused` , a Chromium modal as `sheet` , and an open application menu as `menu`
Surfaces	`list-surfaces`	Works. Lists `window` , `focused` and `sheet` per process, plus `menu` with `item_count` while a menu is open
Shell surfaces	`open-system-surface`	Works for `start-menu` , `taskbar` , `system-tray` , `system-tray-overflow` and `action-center` , and returns the window `snapshot --surface <kind>` reads. `quick-settings` returns `PLATFORM_NOT_SUPPORTED` naming `action-center`
Ref interaction	`click` , `right-click` , `type` , `set-value` , `clear` , `select` , `toggle` , `check` , `uncheck` , `expand` , `collapse` , `scroll` , `scroll-to`	Works, semantic in headless and `--headed` mode. `right-click` and `type` need `--headed` and return `POLICY_DENIED` headless
Multi-click and focus	`double-click` , `triple-click` , `focus`	Works with global `--headed` only
Keyboard and mouse	`press` , `hover` , `drag` , `mouse-click` , `mouse-move` , `mouse-wheel`	Works. Cursor-moving input needs `--headed` . First Contact covers `press --app`
Held input	`key-down` , `key-up` , `mouse-down` , `mouse-up`	Fails closed on every platform with `ACTION_NOT_SUPPORTED` . A headless `mouse-down` or `mouse-up` returns `POLICY_DENIED` first
App and window	`launch` , `close-app` , `list-windows` , `list-apps` , `focus-window` , `resize-window` , `move-window` , `minimize` , `maximize` , `restore`	Works. First Contact covers the name `launch` takes
Displays	`list-displays`	Works
Clipboard	`clipboard-get` , `clipboard-set` , `clipboard-clear`	Works; typed text and image content
Wait	`wait`	Works for ms, `--element` , `--window` , `--text` , `--menu` , `--menu-closed` , `--event` and `--notification`
Notifications	`list-notifications` , `dismiss-notification` , `dismiss-all-notifications` , `notification-action`	Works over the Action Center. Pass `--headed` when the center is closed
Cursor overlay	`cursor-overlay`	Works. `enable` returns `data.rendered: true` only after the renderer acknowledges over its pipe. See `references/shell-and-overlay.md`
System	`status` , `permissions` , `version` , `batch` , `skills` , `session` , `trace`	Works

分组	命令	Windows 上的状态
观察	`snapshot` 、 `find` 、 `get` 、 `is` 、 `screenshot`	支持。可用的 surface 包括 `window` 、 `focused` 、以 `sheet` 形式呈现的 Chromium 模态窗口，以及以 `menu` 形式呈现的已打开应用菜单
Surface	`list-surfaces`	支持。按进程列出 `window` 、 `focused` 和 `sheet` ；菜单打开时，还会列出包含 `item_count` 的 `menu`
Shell surface	`open-system-surface`	支持 `start-menu` 、 `taskbar` 、 `system-tray` 、 `system-tray-overflow` 和 `action-center` ，并返回可供 `snapshot --surface <kind>` 读取的窗口。 `quick-settings` 返回 `PLATFORM_NOT_SUPPORTED` ，并指明应使用 `action-center`
Ref 交互	`click` 、 `right-click` 、 `type` 、 `set-value` 、 `clear` 、 `select` 、 `toggle` 、 `check` 、 `uncheck` 、 `expand` 、 `collapse` 、 `scroll` 、 `scroll-to`	支持，在 headless 和 `--headed` 模式下均可执行语义操作。 `right-click` 和 `type` 需要 `--headed` ，在 headless 模式下返回 `POLICY_DENIED`
多击与焦点	`double-click` 、 `triple-click` 、 `focus`	仅在全局使用 `--headed` 时支持
键盘与鼠标	`press` 、 `hover` 、 `drag` 、 `mouse-click` 、 `mouse-move` 、 `mouse-wheel`	支持。需要移动光标的输入操作需要 `--headed` 。首次使用部分介绍了 `press --app`
按住输入	`key-down` 、 `key-up` 、 `mouse-down` 、 `mouse-up`	在所有平台上均会 fail closed 并返回 `ACTION_NOT_SUPPORTED` 。在 headless 模式下调用 `mouse-down` 或 `mouse-up` 时，会首先返回 `POLICY_DENIED`
应用与窗口	`launch` 、 `close-app` 、 `list-windows` 、 `list-apps` 、 `focus-window` 、 `resize-window` 、 `move-window` 、 `minimize` 、 `maximize` 、 `restore`	支持。首次使用部分介绍了 `launch` 接受的名称格式
显示器	`list-displays`	支持
剪贴板	`clipboard-get` 、 `clipboard-set` 、 `clipboard-clear`	支持；可处理文本和图片内容
等待	`wait`	支持 ms、 `--element` 、 `--window` 、 `--text` 、 `--menu` 、 `--menu-closed` 、 `--event` 和 `--notification`
通知	`list-notifications` 、 `dismiss-notification` 、 `dismiss-all-notifications` 、 `notification-action`	支持通过 Action Center 操作。Action Center 关闭时需要传入 `--headed`
光标叠加层	`cursor-overlay`	支持。 `enable` 仅会在 renderer 通过管道确认后返回 `data.rendered: true` 。参见 `references/shell-and-overlay.md`
系统	`status` 、 `permissions` 、 `version` 、 `batch` 、 `skills` 、 `session` 、 `trace`	支持

First Contact

首次使用

Quote every ref in PowerShell. PowerShell reads a bare
```
@token
```
as its splatting operator and deletes the argument before the binary sees it, so
```
set-value @s8f3k2p9:e1 hi
```
fails
```
INVALID_ARGS
```
. Write
```
set-value '@s8f3k2p9:e1' hi
```
. cmd.exe and bash need no quoting.
A window behind another window is drivable. A minimized one is not. Every element of a minimized window reports
```
offscreen
```
and ref actions fail. Run
```
restore --app <image>
```
and snapshot again. Do not use
```
focus-window
```
by reflex: it steals the user's foreground.
No permission dialog exists.
```
permissions
```
reports
```
automation
```
as
```
not_required
```
. Elevation boundaries are the only access control.
Chromium and Electron apps read thin before they settle. Pass a longer
```
--timeout-ms
```
to
```
snapshot
```
instead of concluding the tree is empty.
Blocked shortcuts.
```
alt+f4
```
,
```
win+l
```
,
```
win+d
```
,
```
alt+tab
```
,
```
win+r
```
,
```
win+x
```
,
```
win+e
```
,
```
ctrl+shift+esc
```
and their modifier supersets are refused without
```
--force
```
. In combos,
```
cmd
```
and
```
meta
```
are the Windows key, so
```
meta+c
```
is a shell shortcut. Write
```
ctrl+c
```
to copy.
type
needs
--headed
. It is keystroke synthesis through
```
SendInput
```
, so a strict-headless
```
type
```
returns
```
POLICY_DENIED
```
. Use
```
set-value
```
, or pass
```
--headed
```
. It ends
```
delivered_unverified
```
(
```
verification_scope: "unavailable"
```
) when the field exposes no readable Text selection, and can verify when one is readable: confirm with a fresh
```
get
```
.
press --app
writes to the foreground queue with no per-process targeting. A headless press to a non-frontmost app fails closed. A non-interactive caller (service, scheduled task, CI job) gets
```
delivered_unverified
```
.
launch
takes an absolute path or a bare name under System32 or the Windows directory, with or without
```
.exe
```
(
```
notepad
```
or
```
notepad.exe
```
), not a display name. Use
```
list-apps
```
to find a running app.

Notifications:

list-notifications

with the Action Center closed needs

--headed

. Single-item

dismiss-notification

and

notification-action

need

--headed

and

--expected-app

--expected-title

for the entry at INDEX.

Surfaces: an open or context menu is the
```
menu
```
surface. Windows does not advertise a
```
menubar
```
surface.
Commands that raise shell chrome take the foreground. Strict headless refuses them with
```
POLICY_DENIED
```
before anything is raised.
Stateful actions run once, then verify by re-reading.
```
delivered_verified
```
means the readback matched.
```
ACTION_FAILED
```
with
```
details.kind: "post_action_verification"
```
means the action landed but the readback differs: read
```
details.post_state
```
and do not retry. A
```
set-value
```
the app reformats reads as failed. See
```
references/troubleshooting.md
```
.
Unsigned binary warnings are expected. See
```
references/troubleshooting.md
```
.

在 PowerShell 中必须为每个 ref 加引号。 PowerShell 会将未加引号的
```
@token
```
解析为 splatting 运算符，并在二进制程序收到参数前将其删除，因此
```
set-value @s8f3k2p9:e1 hi
```
会因
```
INVALID_ARGS
```
失败。应写成
```
set-value '@s8f3k2p9:e1' hi
```
。cmd.exe 和 bash 不需要加引号。
被其他窗口遮挡的窗口仍可操作，但最小化的窗口不可操作。 最小化窗口中的每个元素都会报告
```
offscreen
```
，ref 操作也会失败。请运行
```
restore --app <image>
```
并再次执行 snapshot。不要习惯性地调用
```
focus-window
```
，因为它会抢占用户的前台窗口。
不存在权限对话框。
```
permissions
```
会将
```
automation
```
报告为
```
not_required
```
。权限提升边界是唯一的访问控制机制。
Chromium 和 Electron 应用在稳定下来之前读取到的内容较少。 应为
```
snapshot
```
传入更长的
```
--timeout-ms
```
，而不是直接判断 UI 树为空。
快捷键限制。 未使用
```
--force
```
时，
```
alt+f4
```
、
```
win+l
```
、
```
win+d
```
、
```
alt+tab
```
、
```
win+r
```
、
```
win+x
```
、
```
win+e
```
、
```
ctrl+shift+esc
```
及其包含修饰键的组合都会被拒绝。在组合键中，
```
cmd
```
和
```
meta
```
表示 Windows 键，因此
```
meta+c
```
属于 shell 快捷键。若要复制，请使用
```
ctrl+c
```
。
type
需要
--headed
。它通过
```
SendInput
```
合成按键，因此在严格 headless 模式下调用
```
type
```
会返回
```
POLICY_DENIED
```
。请改用
```
set-value
```
，或传入
```
--headed
```
。当字段未暴露可读的 Text 选择内容时，操作结果为
```
delivered_unverified
```
（
```
verification_scope: "unavailable"
```
）；存在可读内容时可以验证，请使用新的
```
get
```
确认结果。
press --app
会写入 foreground queue，不支持按进程指定目标。在 headless 模式下向非前台应用发送按键会 fail closed。非交互式调用方（服务、计划任务、CI job）会得到
```
delivered_unverified
```
。
launch
接受绝对路径，或接受位于 System32 或 Windows 目录下的名称，可以带或不带
```
.exe
```
（例如
```
notepad
```
或
```
notepad.exe
```
），但不能使用应用的显示名称。请使用
```
list-apps
```
查找正在运行的应用。
通知： Action Center 关闭时，
```
list-notifications
```
需要
```
--headed
```
。对 INDEX 位置的条目执行单项
```
dismiss-notification
```
或
```
notification-action
```
时，需要同时传入
```
--headed
```
以及
```
--expected-app
```
或
```
--expected-title
```
。
Surface： 已打开的菜单或上下文菜单属于
```
menu
```
surface。Windows 不会公开
```
menubar
```
surface。
调起 shell chrome 的命令会抢占前台。 严格 headless 模式会在调起任何界面前以
```
POLICY_DENIED
```
拒绝执行。
有状态操作执行一次后，应通过重新读取进行验证。
```
delivered_verified
```
表示 readback 与预期一致。出现
```
ACTION_FAILED
```
且
```
details.kind: "post_action_verification"
```
表示操作已经生效，但 readback 结果不同：请读取
```
details.post_state
```
，不要重试。如果应用重新格式化了
```
set-value
```
设置的值，该操作会被判定为失败。参见
```
references/troubleshooting.md
```
。
未签名二进制文件的警告属于预期现象。 参见
```
references/troubleshooting.md
```
。

What differs on Windows

Windows 上的差异

Ref actions are semantic UI Automation calls in headless and
```
--headed
```
mode, except
```
right-click
```
,
```
double-click
```
,
```
triple-click
```
,
```
type
```
and
```
focus
```
, which are physical or take focus and need
```
--headed
```
.
The cursor overlay plays its per-action travel only for headless semantic actions. A
```
--headed
```
action moves the real cursor instead.

在 headless 和
```
--headed
```
模式下，Ref 操作均为语义化 UI Automation 调用；但
```
right-click
```
、
```
double-click
```
、
```
triple-click
```
、
```
type
```
和
```
focus
```
除外。这些操作属于物理输入操作或需要获取焦点，因此需要
```
--headed
```
。
光标叠加层仅在 headless 语义操作期间播放逐次动作的移动效果。执行
```
--headed
```
操作时，它会改为移动真实光标。

References

参考资料

```
references/shell-and-overlay.md
```
: open it for
```
open-system-surface
```
, tray and taskbar surfaces, notifications, the cursor overlay, hosted (UWP) window identity and menu detection coverage.
```
references/permissions-and-elevation.md
```
: integrity levels, UIPI, blocked combos, protected processes, the cross-process interaction lease.
```
references/chromium-and-electron.md
```
: settle timing, covered-window hazard, WPF wrong-provider trap, unnamed web content.
```
references/troubleshooting.md
```
: symptom-to-cause map, action verification outcomes, saving a document headless, install warnings.

Read any file from the binary:

agent-desktop skills get agent-desktop-windows references/troubleshooting.md

```
references/shell-and-overlay.md
```
：用于查阅
```
open-system-surface
```
、托盘和任务栏 surface、通知、光标叠加层、托管（UWP）窗口标识及菜单检测覆盖范围。
```
references/permissions-and-elevation.md
```
：用于查阅完整性级别、UIPI、受限制的组合键、受保护进程和跨进程交互租约。
```
references/chromium-and-electron.md
```
：用于查阅稳定等待时序、窗口遮挡隐患、WPF 错误 provider 陷阱和未命名 Web 内容。
```
references/troubleshooting.md
```
：提供症状与原因对照表、操作验证结果、headless 模式下保存文档的方法及安装警告说明。

可从二进制程序中读取任意文件：

agent-desktop skills get agent-desktop-windows references/troubleshooting.md

。