Back to Details

agent-desktop-linux

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

agent-desktop-linux

agent-desktop-linux

The observe-act loop, refs, JSON envelope, sessions and tracing match every platform. This file covers only what differs on Linux.
Needs Ubuntu 24.04+ (x86_64 or aarch64), a graphical session of the signed-in user and a running AT-SPI bus. Other distributions need the at-spi2-core bridge fix dc0dc33; Ubuntu 22.04's bridge aborts on 
Action.GetActions
. The archive loader floor is glibc 2.35. Run the CLI inside the target apps' desktop session: the session bus, display and runtime directory come from its environment. A TTY login, an SSH shell without session variables, a container without a bus or another user's session sees no apps. GNOME is the measured desktop; X11 and Wayland sessions both work and differ in input, capture and window control. Observation and semantic actions need no helper tool.
observe-act 循环、refs、JSON envelope、sessions 和 tracing 在所有平台上均一致。本文件仅介绍 Linux 上的差异。
需要 Ubuntu 24.04+（x86_64 或 aarch64）、已登录用户的图形会话，以及正在运行的 AT-SPI 总线。其他发行版需要应用 at-spi2-core bridge 修复 dc0dc33；Ubuntu 22.04 的 bridge 在执行 
Action.GetActions
时会中止。归档加载器的最低要求为 glibc 2.35。应在目标应用的 desktop session 内运行 CLI：session bus、display 和 runtime directory 均来自该环境。TTY 登录、未设置 session 变量的 SSH shell、没有 bus 的 container，或其他用户的 session 都看不到应用。GNOME 是经过实测的 desktop；X11 和 Wayland session 均可使用，但在输入、捕获和窗口控制方面有所不同。观察和语义操作不需要 helper tool。

Capability Table

能力表

GroupCommandsStatus on Linux
Observation
snapshot
, 
find
, 
get
, 
is
Works on X11 and Wayland through AT-SPI. Supports 
window
, 
sheet
, 
alert
and 
menu
; GNOME adds shell surfaces. Virtual tray getters have limits; see the shell reference and check 
status
Surfaces
list-surfaces
Lists an app's windows, sheets and open menus; shell reads use 
snapshot --surface
Ref interaction
click
, 
right-click
, 
type
, 
set-value
, 
clear
, 
select
, 
toggle
, 
check
, 
uncheck
, 
expand
, 
collapse
, 
scroll-to
Need a readable live identity and the required AT-SPI capability. See the shell reference for menu-ref resolution limits; X11 headed fallback also needs usable geometry and does not bypass identity checks
Wheel scrolling
scroll
, 
mouse-wheel
Physical wheel input; X11 and 
--headed
only
Pointer gestures
double-click
, 
triple-click
, 
hover
, 
drag
--headed
, X11 only. On Wayland the pointer cannot be aimed and these refuse with 
ACTION_NOT_SUPPORTED
Focus
focus
Needs 
--headed
for the focus permission. Calls AT-SPI 
Component.GrabFocus
on a focusable node that is not yet 
focused
, not the pointer, and verifies the state; a GTK4 app rejects the call, so 
focus
ends 
ACTION_FAILED
grab_focus_rejected
there and 
focus-window
is the way to focus its window
Keyboard and mouse
press
, 
mouse-click
, 
mouse-move
, 
mouse-wheel
press
and text keys work on X11 and capability-detected Mutter Wayland sessions; pointer commands are X11 and 
--headed
. See 
references/display-servers-and-input.md
Held input
key-down
, 
key-up
, 
mouse-down
, 
mouse-up
Fail closed on every platform: on Linux, 
key-down
and 
key-up
with 
ACTION_NOT_SUPPORTED
; 
mouse-down
and 
mouse-up
with 
POLICY_DENIED
headless and 
ACTION_NOT_SUPPORTED
with 
--headed
App and window
launch
, 
close-app
, 
list-windows
, 
list-apps
, 
focus-window
, 
move-window
, 
resize-window
, 
minimize
, 
maximize
, 
restore
Launch and listing work on X11 and Wayland; 
focus-window
requires X11. Other window control and graceful close need an X11 client. See 
references/windows-and-lifecycle.md
Displays
list-displays
Works
Capture
screenshot
X11: a window without permission; a display only under a non-compositing manager. Wayland: portal consent, with display cropping. No selector returns the primary display. See 
references/capture-and-clipboard.md
Clipboard
clipboard-get
, 
clipboard-set
, 
clipboard-clear
Works; text, images and file lists. Wayland needs 
wl-clipboard
Wait
wait
Works for ms, 
--element
, 
--window
, 
--text
, 
--menu
, 
--menu-closed
, 
--event
. Menu waits share the menu-surface detector; 
--notification
waits for a new notification on GNOME
Notifications
list-notifications
, 
dismiss-notification
, 
dismiss-all-notifications
, 
notification-action
GNOME Shell only; list headless, mutations 
--headed
on X11, and 
dismiss-notification
and 
notification-action
also need 
--expected-app
or 
--expected-title
. Fields are read from the shell's widget layout, verified on GNOME Shell 46.0 only, and each mutation is proven by its effect. Wayland refuses the shell pointer before input. A notification owner that is not the shell or its own helper returns 
PLATFORM_NOT_SUPPORTED
. See the shell reference
Shell surfaces
open-system-surface
Only 
system-tray
and 
action-center
can work. 
system-tray
needs a StatusNotifierWatcher (else it refuses with 
watcher: absent
); 
action-center
needs proven role ancestry. 
quick-settings
refuses with 
shell_container_unproven
, 
taskbar
always refuses, and 
start-menu
and 
system-tray-overflow
are not supported, though 
--help
lists them. See the shell reference
Cursor overlay
cursor-overlay
Requires an active session; see Cursor overlay below. X11 enable returns 
data.rendered: true
; Wayland saves the setting with 
data.rendered: false
System
status
, 
permissions
, 
version
, 
batch
, 
skills
, 
session
, 
trace
Works
命令组命令Linux 上的状态
观察
snapshot
、
find
、
get
、
is
通过 AT-SPI 在 X11 和 Wayland 上均可工作。支持 
window
、
sheet
、
alert
和 
menu
；GNOME 还提供 shell surfaces。虚拟 tray getter 存在限制；请参阅 shell reference 并检查 
status
Surfaces
list-surfaces
列出应用的窗口、sheet 和已打开的菜单；shell 读取使用 
snapshot --surface
Ref 操作
click
、
right-click
、
type
、
set-value
、
clear
、
select
、
toggle
、
check
、
uncheck
、
expand
、
collapse
、
scroll-to
需要可读的实时 identity 和所需的 AT-SPI capability。请参阅 shell reference 了解 menu-ref 解析限制；X11 headed fallback 还需要可用 geometry，且不会绕过 identity 检查
滚轮滚动
scroll
、
mouse-wheel
物理滚轮输入；仅支持 X11 和 
--headed
指针手势
double-click
、
triple-click
、
hover
、
drag
仅支持 
--headed
和 X11。在 Wayland 上无法瞄准 pointer，因此这些命令会以 
ACTION_NOT_SUPPORTED
拒绝
焦点
focus
需要 
--headed
才能获得焦点权限。对尚未 
focused
且可聚焦的节点调用 AT-SPI 
Component.GrabFocus
，而不是移动指针，并验证状态；GTK4 应用会拒绝该调用，因此 
focus
会以 
ACTION_FAILED
和 
grab_focus_rejected
结束，此时应使用 
focus-window
聚焦其窗口
键盘和鼠标
press
、
mouse-click
、
mouse-move
、
mouse-wheel
press
和文本按键在 X11 以及通过 capability 检测的 Mutter Wayland session 中可用；pointer 命令仅支持 X11 和 
--headed
。参见 
references/display-servers-and-input.md
按住输入
key-down
、
key-up
、
mouse-down
、
mouse-up
在所有平台上均 fail closed：在 Linux 上，
key-down
和 
key-up
返回 
ACTION_NOT_SUPPORTED
；headless 模式下 
mouse-down
和 
mouse-up
返回 
POLICY_DENIED
，使用 
--headed
时返回 
ACTION_NOT_SUPPORTED
应用和窗口
launch
、
close-app
、
list-windows
、
list-apps
、
focus-window
、
move-window
、
resize-window
、
minimize
、
maximize
、
restore
Launch 和 listing 在 X11 和 Wayland 上均可工作；
focus-window
需要 X11。其他 window control 和 graceful close 需要 X11 client。参见 
references/windows-and-lifecycle.md
Displays
list-displays
可用
捕获
screenshot
X11：捕获窗口无需权限；仅在非合成 display manager 下捕获整个 display。Wayland：需要 portal consent，并会裁剪 display。未指定 selector 时返回主 display。参见 
references/capture-and-clipboard.md
剪贴板
clipboard-get
、
clipboard-set
、
clipboard-clear
可用；支持文本、图片和文件列表。Wayland 需要 
wl-clipboard
等待
wait
支持毫秒、
--element
、
--window
、
--text
、
--menu
、
--menu-closed
、
--event
。Menu wait 共用 menu-surface detector；
--notification
在 GNOME 上等待新通知
通知
list-notifications
、
dismiss-notification
、
dismiss-all-notifications
、
notification-action
仅支持 GNOME Shell；list 可 headless 运行，mutation 在 X11 上需要 
--headed
，而 
dismiss-notification
和 
notification-action
还需要 
--expected-app
或 
--expected-title
。字段从 shell 的 widget layout 读取，仅在 GNOME Shell 46.0 上验证过，并且每次 mutation 都通过其实际效果进行验证。Wayland 会在输入前拒绝 shell pointer。通知 owner 若不是 shell 或其自有 helper，则返回 
PLATFORM_NOT_SUPPORTED
。参见 shell reference
Shell surfaces
open-system-surface
仅 
system-tray
和 
action-center
可能工作。
system-tray
需要 StatusNotifierWatcher，否则会以 
watcher: absent
拒绝；
action-center
需要已证明的 role ancestry。
quick-settings
会以 
shell_container_unproven
拒绝，
taskbar
始终拒绝；
start-menu
和 
system-tray-overflow
不受支持，尽管 
--help
会列出它们。参见 shell reference
Cursor overlay
cursor-overlay
需要活动 session；参见下方的 Cursor overlay。X11 enable 返回 
data.rendered: true
；Wayland 会保存设置并返回 
data.rendered: false
系统
status
、
permissions
、
version
、
batch
、
skills
、
session
、
trace
可用

First Contact

首次使用

  • Run 
    agent-desktop status
    first.     An empty 
    supported_surfaces
    means the accessibility bus did not answer, not an empty desktop. Read 
    references/bus-and-permissions.md
    ; do not restart a bus you did not start.
  • --app
    takes the name 
    list-apps
    prints     (
    gnome-calculator
    , 
    nautilus
    ) or a desktop-entry name or id backed by verified executable and argument identity, never a window title or an arbitrary registry label. A desktop-entry name such as 
    Files
    is accepted once the entry exists. 
    list-windows --app
    is exact (aliases included); 
    list-apps --app
    combines a substring of the listed name with exact adapter aliases, ignoring case.
  • Window ids are opaque addresses of 34 to 70 characters, valid for the life of the window's process; run 
    list-windows
    again after a restart. 
    list-apps
    rows carry 
    bundle_id
    (desktop id when known) and 
    process_instance
    .
  • GTK4 apps nest deeply. 
    --max-depth
    and 
    --skeleton
    count semantic levels, so empty layout wrappers add output levels. 
    -i --compact
    still shows nested nameless 
    group
    nodes: read past them.
  • An empty file view does not prove an empty folder. If an icon view exposes no file items, try an available list or detailed view, then take a fresh snapshot and verify which items the application exposes.
  • A window appears before its content. Run 
    wait --window "Text Editor" --app gnome-text-editor
    first, then 
    wait --text
    , which matches a node's own name, value or description. For a web-engine app (detected from its AT-SPI toolkit metadata), a window observation waits for three equal full reads and a web-document role, or a requested depth cut. A shell with no document yet stays incomplete when the budget ends. A 
    --timeout
    shorter than the first read returns 
    TIMEOUT
    , never a partial tree.
  • Geometry-only refs require unchanged geometry. Position and size must both match, in the ref's recorded coordinate frame; a one-pixel change breaks identity. After a layout change or a window move, take a fresh snapshot instead of retrying the old ref. Indistinguishable siblings stay 
    AMBIGUOUS_TARGET
    : in a Text Editor window with three tabs, the text views share one rectangle and one state, so a ref to any of them answers 
    AMBIGUOUS_TARGET
    with 
    candidate_count
    3 until the extra tabs close. A ref belongs to the window it was taken in, so two windows with the same title each resolve their own refs.
  • Chromium and Qt apps need a launch flag or variable to appear on the bus; Electron apps need none, only 
    --no-sandbox
    where 
    chrome-sandbox
    is not setuid. See 
    references/troubleshooting.md
    .
  • Headed ref actions need a window that can take focus. A window the window manager never activated ends 
    activation_not_granted
    . See 
    references/windows-and-lifecycle.md
    .
  • A GTK4 secret field can show its text. Only a password role is withheld. Treat a GTK4 entry with hidden characters as readable by every observation command.
  • Blocked key combos refuse without 
    --force
    .     See 
    references/display-servers-and-input.md
    .
  • After opening a menu-role popup, inspect 
    data.surfaces
    .     A 
    menu
    entry leads to 
    snapshot --surface menu --app <name> -i
    . If absent, use 
    wait --menu
    and 
    list-surfaces
    before assuming the opener failed. A list-backed dropdown is part of the window, not a menu surface; see the shell reference.
  • Shell pointer input requires receiver proof. On X11 a headed shell click is proven from the shell's own tree: the receiver must be the target and stand still, or the click refuses before any pointer event. Wayland shell clicks refuse. Notifications need the GNOME notification service; tray snapshots also need a StatusNotifierWatcher.
  • The state root must be private. A group-writable or world-accessible 
    AGENT_DESKTOP_HOME
    or 
    ~/.agent-desktop
    makes 
    snapshot
    fail with 
    INTERNAL
    ; run 
    chmod 700
    on it or let the tool create it.
  • 首先运行 
    agent-desktop status
    。     
    supported_surfaces
    为空表示无障碍总线没有响应，并不表示 desktop 为空。请阅读 
    references/bus-and-permissions.md
    ；不要重启并非由你启动的 bus。
  • --app
    应使用 
    list-apps
    输出的名称    （
    gnome-calculator
    、
    nautilus
    ），或者使用以已验证的 executable 和 argument identity 为依据的 desktop-entry name 或 id；绝不要使用窗口标题或任意的 registry label。desktop-entry 名称（如 
    Files
    ）在对应 entry 已存在后即可接受。
    list-windows --app
    是精确匹配（包含 aliases）；
    list-apps --app
    会将所列名称的子串与精确的 adapter aliases 组合匹配，并忽略大小写。
  • 窗口 id 是不透明地址，长度为 34 到 70 个字符，仅在窗口进程的生命周期内有效；重启后需要再次运行 
    list-windows
    。
    list-apps
    的各行包含 
    bundle_id
    （已知时为 desktop id）和 
    process_instance
    。
  • GTK4 应用会深层嵌套。 
    --max-depth
    和 
    --skeleton
    计算语义层级，因此空的 layout wrapper 也会增加输出层级。
    -i --compact
    仍会显示嵌套的匿名 
    group
    节点：应读取并跳过这些节点。
  • 空的文件视图不能证明文件夹为空。 如果 icon view 未公开任何 file item，请尝试可用的 list view 或 detailed view，然后重新 snapshot，并验证 application 公开了哪些 item。
  • 窗口会先于其内容出现。 先运行 
    wait --window "Text Editor" --app gnome-text-editor
    ，然后运行 
    wait --text
    ；后者会匹配节点自身的 name、value 或 description。对于 web-engine app（根据其 AT-SPI toolkit metadata 检测），window observation 会等待三次相同的完整读取以及一个 web-document role，或达到请求的 depth cut。如果 shell 在预算耗尽前仍没有 document，则会保持 incomplete。
    --timeout
    若短于首次读取所需时间，将返回 
    TIMEOUT
    ，绝不会返回部分 tree。
  • 仅含 geometry 的 ref 要求 geometry 保持不变。 位置和大小都必须在 ref 记录的 coordinate frame 中匹配；一个像素的变化就会破坏 identity。布局变化或窗口移动后，应重新 snapshot，而不是重试旧 ref。无法区分的 sibling 会保持 
    AMBIGUOUS_TARGET
    ：在包含三个标签页的 Text Editor 窗口中，各 text view 共享同一 rectangle 和 state，因此指向其中任意一个的 ref 都会返回 
    AMBIGUOUS_TARGET
    和 
    candidate_count
    3，直到关闭多余标签页。ref 归属于获取它的窗口，因此两个标题相同的窗口会分别解析各自的 ref。
  • Chromium 和 Qt 应用需要 launch flag 或 variable 才能出现在 bus 上； Electron 应用不需要，仅在 
    chrome-sandbox
    未设置为 setuid 时需要 
    --no-sandbox
    。参见 
    references/troubleshooting.md
    。
  • Headed ref 操作需要一个能够接收焦点的窗口。 如果 window manager 从未激活该窗口，操作会以 
    activation_not_granted
    结束。参见 
    references/windows-and-lifecycle.md
    。
  • GTK4 secret field 可能会显示其文本。 只有 password role 会被隐藏。应将隐藏字符的 GTK4 entry 视为每个 observation command 均可读取。
  • 被阻止的组合键在没有 
    --force
    时会拒绝执行。     参见 
    references/display-servers-and-input.md
    。
  • 打开 menu-role popup 后，应检查 
    data.surfaces
    。     
    menu
    entry 可用于 
    snapshot --surface menu --app <name> -i
    。如果不存在，应先使用 
    wait --menu
    和 
    list-surfaces
    ，再假定 opener 失败。基于 list 的 dropdown 是窗口的一部分，而不是 menu surface；参见 shell reference。
  • Shell pointer 输入需要 receiver proof。 在 X11 上，headed shell click 可从 shell 自身的 tree 中得到证明：receiver 必须是目标且保持静止，否则命令会在发送任何 pointer event 前拒绝执行。Wayland shell click 会直接拒绝。通知需要 GNOME notification service；tray snapshot 还需要 StatusNotifierWatcher。
  • State root 必须保持私有。 group-writable 或 world-accessible 的 
    AGENT_DESKTOP_HOME
    或 
    ~/.agent-desktop
    会导致 
    snapshot
    以 
    INTERNAL
    失败；应对其执行 
    chmod 700
    ，或让工具自行创建。

Wait for a window

等待窗口

Wait for an app window to appear (
--window
requires a title or title substring):
bash
agent-desktop wait --window "Text Editor" --app gnome-text-editor --timeout 10000
To wait for a close that you cause, batch it so the baseline is taken before the close (Calculator has no unsaved-document prompt to interrupt it):
bash
agent-desktop batch '[{"command":"close-app","args":{"app":"gnome-calculator"}},{"command":"wait","args":{"event":"window-closed","app":"gnome-calculator","timeout":10000}}]' --stop-on-error
To watch someone else close a window, start 
agent-desktop wait --event window-closed --app gnome-text-editor --timeout 10000
before they close the window. There is no 
--window-closed
flag.
等待应用窗口出现（
--window
需要完整标题或标题子串）：
bash
agent-desktop wait --window "Text Editor" --app gnome-text-editor --timeout 10000
要等待由自己触发的关闭操作，应使用 batch，以便在关闭前获取 baseline（Calculator 没有会中断流程的未保存文档提示）：
bash
agent-desktop batch '[{"command":"close-app","args":{"app":"gnome-calculator"}},{"command":"wait","args":{"event":"window-closed","app":"gnome-calculator","timeout":10000}}]' --stop-on-error
要观察其他人关闭窗口，应在对方关闭窗口前启动 
agent-desktop wait --event window-closed --app gnome-text-editor --timeout 10000
。不存在 
--window-closed
flag。

Cursor overlay

Cursor overlay

Start a session and pass its returned ID to every overlay command:
bash
session_id=$(agent-desktop session start --name docs-example | jq -r '.data.session_id')
agent-desktop cursor-overlay enable --session "$session_id"
agent-desktop cursor-overlay disable --session "$session_id"
agent-desktop session end "$session_id"
Without a session, 
cursor-overlay enable
and 
disable
return 
INVALID_ARGS
. On X11, enable returns 
data.rendered: true
; on Wayland it saves the setting and returns 
data.rendered: false
without drawing an overlay.
session start --cursor --multi-agent
returns 
data.rendered: false
, because nothing is drawn until an agent acts. On X11 each agent id then gets its own cursor; on Wayland none is drawn.
启动 session，并将返回的 ID 传给每条 overlay 命令：
bash
session_id=$(agent-desktop session start --name docs-example | jq -r '.data.session_id')
agent-desktop cursor-overlay enable --session "$session_id"
agent-desktop cursor-overlay disable --session "$session_id"
agent-desktop session end "$session_id"
如果没有 session，
cursor-overlay enable
和 
disable
会返回 
INVALID_ARGS
。
在 X11 上，enable 返回 
data.rendered: true
；在 Wayland 上，它会保存设置并返回 
data.rendered: false
，但不会绘制 overlay。
session start --cursor --multi-agent
返回 
data.rendered: false
，因为在 agent 执行操作前不会绘制任何内容。在 X11 上，每个 agent id 随后会获得自己的 cursor；在 Wayland 上则不会绘制任何 cursor。

What differs on Linux

Linux 上的差异

  • Firefox/Gecko 
    select
    can change the selected value without firing 
    change
    or 
    input
    events, so dependent page handlers do not run. The result stays 
    delivered_unverified
    even when value readback matches. Check the dependent application effect before continuing: value readback alone is no commit proof.
  • Two 
    click
    calls on one GTK4 button less than about 250 ms apart act once. Both answer 
    ok: true
    with 
    delivered_unverified
    . To click one button several times, wait for the effect between the clicks: use 
    wait --element <ref> --predicate value --value <expected>
    on the element that shows the result, or 
    wait 300
    .
  • Ref actions are semantic AT-SPI calls that never move the pointer or change the active window. Under 
    --headed
    , ref 
    click
    , 
    right-click
    and 
    type
    try physical input first and fall back to the semantic action only after a 
    not_delivered
    refusal (Windows works the other way round). On Wayland, headed pointer commands and a headed 
    select
    refuse at once with 
    ACTION_NOT_SUPPORTED
    , cause 
    wayland_session
    ; a headed 
    type
    needs the Mutter RemoteDesktop capability. 
    references/display-servers-and-input.md
    has the full rules.
  • select
    tries the semantic route first and the pointer second; the pointer route needs an explicit 
    --headed
    request on X11. A Qt combo's semantic selection only highlights its popup row, so headless selection refuses with 
    POLICY_DENIED
    , kind 
    pointer_required
    , before changing that highlight. Headed, on a list whose toolkit index cannot be mapped to the row (a GTK4 list) it clicks the row and checks its 
    selected
    state; on a combo with no semantic opener (a libadwaita combo row) it clicks the combo open, selects the choice through the popup's own list and checks the value the combo shows. Headless, a list refuses with 
    selection_index_unproven
    , and a combo with no semantic opener with 
    POLICY_DENIED
    , kind 
    pointer_required
    , unless the combo already shows the choice. A GTK4 drop-down that opens through its own toggle takes 
    select
    without 
    --headed
    and checks the value it shows.
  • A ref can expose no safe semantic mutation route. Headless requests then return an unsupported-action error without inventing 
    hidden
    or 
    disabled
    state and without switching to physical input. GTK4 entry and password 
    Activate
    actions are one such case: use 
    type
    to edit the field, or request 
    --headed
    explicitly when a pointer click is intended.
  • Opening a native Wayland Qt combo returns 
    ACTION_NOT_SUPPORTED
    , kind 
    native_popup_requires_input
    , before invoking its action. Qt needs a user input serial to show that popup. An already open combo may be collapsed; Qt windows proven to use Xwayland retain their semantic opener.
  • Headless 
    type
    writes through 
    EditableText
    . A terminal's text node is a 
    textfield
    named 
    Terminal
    whose only action is 
    RightClick
    ; headless 
    type
    on it is 
    POLICY_DENIED
    with check 
    supported_action
    , and its value is null. It receives keys only under 
    --headed
    .
  • In key combos 
    cmd
    and 
    meta
    are the Super key. Key names are the macOS and Windows names plus X keysym names. 
    press
    takes mapped keys only; 
    type
    handles characters the layout cannot produce.
  • Window ops act on X11 clients; 
    focus-window
    and headed scoped 
    press
    require an X11 session. Headless scoped 
    press
    requires live target focus on either display server. Focusing a window on another workspace can switch the user's current desktop.
  • Linux exact FFI refs carrying bounds or a bounds hash require window-relative geometry, even when the ref also carries a name or identifier. Geometry-free identity refs are supported.
  • Firefox/Gecko 的 
    select
    可以在不触发 
    change
    或 
    input
    event 的情况下更改所选 value，因此依赖该 value 的 page handler 不会运行。即使 value readback 匹配，结果仍会保持 
    delivered_unverified
    。继续操作前应检查依赖它的 application effect：仅凭 value readback 无法证明提交成功。
  • 对同一个 GTK4 button 执行两次 
    click
    ，若间隔小于约 250 ms，则只会生效一次。两次调用都会返回 
    ok: true
    和 
    delivered_unverified
    。若要多次点击同一个 button，应在每次点击之间等待效果出现：可在显示结果的 element 上使用 
    wait --element <ref> --predicate value --value <expected>
    ，或者使用 
    wait 300
    。
  • Ref 操作是语义 AT-SPI 调用，绝不会移动 pointer 或更改 active window。在 
    --headed
    下，ref 
    click
    、
    right-click
    和 
    type
    会先尝试 physical input，仅在收到 
    not_delivered
    拒绝后才回退到 semantic action（Windows 的顺序相反）。在 Wayland 上，headed pointer command 和 headed 
    select
    会立即以 
    ACTION_NOT_SUPPORTED
    拒绝，cause 为 
    wayland_session
    ；headed 
    type
    则需要 Mutter RemoteDesktop capability。完整规则参见 
    references/display-servers-and-input.md
    。
  • select
    会先尝试 semantic route，再尝试 pointer route；在 X11 上，pointer route 需要显式指定 
    --headed
    。Qt combo 的 semantic selection 只会高亮 popup row，因此 headless selection 会在改变该高亮之前，以 
    POLICY_DENIED
    和 kind 
    pointer_required
    拒绝。在 headed 模式下，如果某个 list 的 toolkit index 无法映射到具体 row（例如 GTK4 list），则会点击该 row 并检查其 
    selected
    状态；对于没有 semantic opener 的 combo（例如 libadwaita combo row），则会点击 combo 将其打开，通过 popup 自身的 list 选择选项，并检查 combo 所显示的 value。在 headless 模式下，list 会以 
    selection_index_unproven
    拒绝；没有 semantic opener 的 combo 会以 
    POLICY_DENIED
    和 kind 
    pointer_required
    拒绝，除非 combo 已经显示了该选项。通过自身 toggle 打开的 GTK4 drop-down 无需 
    --headed
    即可执行 
    select
    ，并会检查其显示的 value。
  • 某个 ref 可能没有安全的 semantic mutation route。此时 headless 请求会返回 unsupported-action error，既不会虚构 
    hidden
    或 
    disabled
    状态，也不会切换为 physical input。GTK4 entry 和 password 的 
    Activate
    action 就属于这种情况：应使用 
    type
    编辑字段；如果明确需要 pointer click，则显式请求 
    --headed
    。
  • 打开原生 Wayland Qt combo 时，会在调用 action 前返回 
    ACTION_NOT_SUPPORTED
    和 kind 
    native_popup_requires_input
    。Qt 需要 user input serial 才能显示 popup。已经打开的 combo 可以折叠；已证明使用 Xwayland 的 Qt window 会保留其 semantic opener。
  • Headless 
    type
    通过 
    EditableText
    写入。Terminal 的 text node 是一个名为 
    Terminal
    的 
    textfield
    ，其唯一 action 是 
    RightClick
    ；headless 
    type
    会以 
    POLICY_DENIED
    拒绝，check 为 
    supported_action
    ，且其 value 为 null。只有在 
    --headed
    下，它才会接收按键。
  • 在组合键中，
    cmd
    和 
    meta
    表示 Super key。键名采用 macOS 和 Windows 的名称以及 X keysym 名称。
    press
    仅接受已映射的按键；
    type
    可处理当前 layout 无法生成的字符。
  • Window op 作用于 X11 client；
    focus-window
    和 headed scoped 
    press
    需要 X11。Headless scoped 
    press
    要求在任一 display server 上，目标都拥有实时焦点。聚焦另一个 workspace 上的窗口可能会切换用户当前的 desktop。
  • 在 Linux 上，携带 bounds 或 bounds hash 的 exact FFI ref 需要相对窗口的 geometry，即使该 ref 同时携带 name 或 identifier。无 geometry 的 identity ref 仍受支持。

References

参考资料

  • references/bus-and-permissions.md
    : bus bootstrap, the three bus states, 
    status
    and 
    permissions
    output, state-root rules.
  • references/display-servers-and-input.md
    : X11 versus Wayland table, 
    press
    , 
    type
    , pointer, wheel, drag, blocked combos, key names.
  • references/windows-and-lifecycle.md
    : 
    launch
    , 
    close-app
    , window ops, focus, workspaces, stopped apps, background 
    press
    .
  • references/capture-and-clipboard.md
    : 
    screenshot
    on X11 and Wayland, the Wayland consent dialog, clipboard owner and helper.
  • references/shell-tray-and-notifications.md
    : menus and GTK4 popovers, GNOME Overview/top bar, shell reads and input, notification identity guards, SNI and AppIndicator requirements and refusals, sandbox guidance.
  • references/troubleshooting.md
    : symptom-to-cause map, Chromium, Electron and Qt flags, toolkit limits, error codes with 
    platform_detail
    .
Read any file from the binary: 
agent-desktop skills get agent-desktop-linux references/troubleshooting.md
.
  • references/bus-and-permissions.md
    ：bus bootstrap、三种 bus state、
    status
    和 
    permissions
    输出、state-root 规则。
  • references/display-servers-and-input.md
    ：X11 与 Wayland 对照表、
    press
    、
    type
    、pointer、wheel、drag、被阻止的组合键、键名。
  • references/windows-and-lifecycle.md
    ：
    launch
    、
    close-app
    、window op、focus、workspace、已停止的应用、background 
    press
    。
  • references/capture-and-clipboard.md
    ：X11 和 Wayland 上的 
    screenshot
    、Wayland consent dialog、clipboard owner 和 helper。
  • references/shell-tray-and-notifications.md
    ：menu 和 GTK4 popover、GNOME Overview/top bar、shell 读取和输入、通知 identity guard、SNI 和 AppIndicator 要求及拒绝规则、sandbox 指南。
  • references/troubleshooting.md
    ：症状到原因的映射、Chromium、Electron 和 Qt flags、toolkit 限制、带 
    platform_detail
    的错误代码。
可从 binary 中读取任意文件： 
agent-desktop skills get agent-desktop-linux references/troubleshooting.md
。