agent-desktop-ffi

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

agent-desktop-ffi

Direct C-ABI access to PlatformAdapter operations. Build the cdylib with the

release-ffi

profile:

cargo build --profile release-ffi -p agent-desktop-ffi

The output is

target/release-ffi/libagent_desktop_ffi.dylib

(

.so

on Linux,

agent_desktop_ffi.dll

on Windows) plus a committed C header at

crates/ffi/include/agent_desktop.h

. The default

release

profile uses

panic = "abort"

: a Rust panic inside an

extern "C"

function would

SIGABRT

the host.

release-ffi

keeps

panic = "unwind"

, which the panic traps need.

Four reference topics, loaded as needed:

ownership.md: who allocates and who frees every pointer the FFI hands back.
error-handling.md: errno-style last-error contract, error codes, enum validation, panic boundary.
threading.md: host-thread contract, mutation ordering across threads and processes, adapter-bound native handles, AXIsProcessTrusted inheritance.
build-and-link.md: prebuilt archives, ABI handshake, struct size validation, C and Python examples.

直接通过 C-ABI 访问 PlatformAdapter 操作。使用

release-ffi

profile 构建 cdylib：

cargo build --profile release-ffi -p agent-desktop-ffi

输出为

target/release-ffi/libagent_desktop_ffi.dylib

（Linux 上为

.so

， Windows 上为

agent_desktop_ffi.dll

），并在

crates/ffi/include/agent_desktop.h

提供一个已提交的 C header。默认

release

profile 使用

panic = "abort"

：

extern "C"

函数内部发生 Rust panic 会让宿主进程触发

SIGABRT

。

release-ffi

保留

panic = "unwind"

，这是 panic 捕获机制所需的配置。

四个 reference topics，可按需加载：

ownership.md：FFI 返回的每个指针由谁分配、由谁释放。
error-handling.md：errno 风格的 last-error 约定、错误码、 enum 校验和 panic 边界。
threading.md：宿主线程约定、跨线程和进程的修改顺序、绑定到 adapter 的 native handles，以及 AXIsProcessTrusted 继承机制。
build-and-link.md：预构建归档、ABI 握手、结构体大小校验，以及 C 和 Python 示例。

Observe-act workflow (canonical path)

Observe-act workflow（规范路径）

ad_init(AD_ABI_VERSION_MAJOR)                    // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1")   // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)

ad_snapshot

returns the CLI envelope

{version, ok, command, data}

data.tree

carries snapshot-qualified refs for interactive elements. Read

data.complete

before trusting a tree:

false

means the observation budget ran out and the tree is partial, not an error.

ad_snapshot

always targets the focused window of

app

; window targeting, skeleton mode and

--root

drill-down are not exposed over the ABI. A bare

@eN

ref returns

AD_RESULT_ERR_INVALID_ARGS

ad_init(AD_ABI_VERSION_MAJOR)                    // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1")   // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)

ad_snapshot

返回 CLI envelope

{version, ok, command, data}

。

data.tree

携带交互式元素的 snapshot-qualified refs。在信任 tree 前，先读取

data.complete

：

false

表示 observation budget 已耗尽， tree 是部分结果，但这不是错误。

ad_snapshot

始终以

app

的当前聚焦窗口为目标；窗口定向、skeleton mode 和

--root

drill-down 均未通过 ABI 暴露。裸

@eN

ref 会返回

AD_RESULT_ERR_INVALID_ARGS

。

Core constraints

核心约束

ABI handshake. Call
```
ad_init(AD_ABI_VERSION_MAJOR)
```
once after loading the library. A mismatch returns
```
AD_RESULT_ERR_INVALID_ARGS
```
: abort.
```
ad_abi_version()
```
returns the library's major for display. The major increments on a removed symbol or a layout change; new
```
ad_*
```
symbols and error codes are additive. Pin the exact
```
libagent_desktop_ffi
```
version you link against.
Session adapters.
```
ad_adapter_create_with_session("session-id")
```
scopes refmap persistence to a session namespace, like CLI
```
--session <id>
```
. IDs are 1 to 64 ASCII alphanumeric,
```
-
```
, or
```
_
```
characters. A null ID behaves like
```
ad_adapter_create()
```
. An invalid ID returns null (read
```
ad_last_error_*
```
).
Ref actions.
```
ad_execute_by_ref
```
is the primary action entrypoint. It loads the RefStore, resolves the ref strictly (
```
STALE_REF
```
when the snapshot has no such ref or the element changed,
```
AMBIGUOUS_TARGET
```
on several candidates), runs the live actionability preflight (visible, stable, enabled, supported action, policy, editable), then dispatches. A disabled or unsupported target fails before any platform call; on
```
AD_RESULT_ERR_ACTION_FAILED
```
the check report is JSON in
```
ad_last_error_details()
```
. It auto-waits 5000 ms before the preflight;
```
ad_execute_by_ref_timeout
```
takes
```
timeout_ms
```
(
```
-1
```
= 5000 ms default,
```
0
```
= single-shot).
```
PressKey
```
defaults to
```
focus_fallback
```
; every other action, including
```
TypeText
```
, defaults to
```
headless
```
. Pass
```
AD_POLICY_KIND_FOCUS_FALLBACK
```
(1) to allow focus without cursor movement, or
```
AD_POLICY_KIND_HEADED
```
(2) to allow physical cursor and keyboard delivery.
```
ad_execute_ref_action_exact_with_policy
```
runs the same preflight from an
```
AdExactRefEntry
```
.
Generation-safe direct APIs. Every direct targeting function takes process-generation evidence:
```
AdExactRefEntry
```
,
```
AdExactWindowInfo
```
,
```
ad_list_windows_exact
```
,
```
ad_launch_app_exact
```
, and the other
```
*_exact
```
symbols.
```
AdRefEntry
```
,
```
AdWindowInfo
```
, and
```
AdSurfaceInfo
```
exist only as fields embedded in the exact structs.
```
ad_list_surfaces_exact
```
keeps the surface
```
id
```
. From
```
AD_EXACT_SURFACE_INFO_VERSION
```
2,
```
AdExactSurfaceInfo.unclassified
```
names the surface kinds its probe could not read (for example
```
sheet,menu
```
), or is null. A missing
```
sheet
```
or
```
menu
```
next to a listed kind is unknown, not absent. Only the Windows adapter fills it; macOS and Linux return null.
Native handles.
```
ad_resolve_element_exact
```
and
```
ad_find_exact
```
return an
```
AdNativeHandle
```
.
```
ad_execute_action
```
(headless, no preflight) and
```
ad_execute_action_with_policy
```
act on it directly, bypassing the ref-action pipeline. Release every handle with
```
ad_free_handle(adapter, &handle)
```
on the same adapter and thread, before
```
ad_adapter_destroy
```
, on every platform. It zeroes
```
handle.ptr
```
, so a second call is a no-op.

Action results.

AdActionResult.steps

mirrors the CLI

steps

array for activation-chain actions. Each

AdActionStep

has

label

and

outcome

strings,

mechanism

(

AD_STEP_MECHANISM_SEMANTIC_API

= 1 or

AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC

= 2, valid only when

has_mechanism

), and

verified

(valid only when

has_verified

AdActionResult.disposition

carries the same delivery/retry pair as

ad_last_error_delivery_semantics

. Release the result with

ad_free_action_result(&out)

Display discovery. Call
```
ad_list_displays
```
before using
```
AdScreenshotTarget.screen_index
```
. List order is the screenshot index order. Each
```
AdDisplayInfo
```
carries a stable display ID, bounds, primary flag, and scale. Release the list with
```
ad_display_list_free
```
.
Errors. A failing call sets thread-local last-error state; read it with the
```
ad_last_error_*
```
accessors before the next failing call.
```
ad_last_error_details()
```
returns structured JSON (the actionability report on
```
ACTION_FAILED
```
, candidate summaries on
```
AMBIGUOUS_TARGET
```
) that can contain element names, values, and window titles from the user's screen: keep it out of shared logs.
```
ad_last_error_delivery_semantics
```
reports whether the failed action was delivered and whether a retry is safe. Invalid enum discriminants return
```
AD_RESULT_ERR_INVALID_ARGS
```
. See error-handling.md.
Threading. Entrypoints may be called from any host thread. Native handles stay on the adapter and thread that created them; pass qualified refs across threads. Desktop mutations take the same cross-process interaction lease as the CLI; reads take no lease and carry finite deadlines. See threading.md for the read/read, read/mutation, and mutation/mutation matrix.
Tracing. File tracing (same JSONL contract as CLI
```
--trace
```
; segments carry
```
event
```
,
```
ts_ms
```
,
```
seq
```
, and redacted fields) is active only when the session has a manifest with
```
trace: on
```
, created by
```
session start
```
(or an equivalent manifest on disk) before the adapter is created.
```
ad_adapter_create_with_session
```
alone writes nothing. Each OS process appends to one segment,
```
~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl
```
, and a long-lived host reuses it.
```
ad_trace_show
```
(JSON envelope) and
```
ad_trace_export
```
(self-contained HTML file) read the adapter session's merged timeline;
```
limit
```
```
-1
```
= CLI default.
Log callback.
```
ad_set_log_callback(cb)
```
registers one process-wide callback for diagnostics regardless of session manifest. It receives, as JSON, the
```
tracing
```
events emitted inside later
```
ad_*
```
calls on the same thread; the FFI never installs or replaces the host process's global tracing subscriber.
```
cb
```
receives an
```
int32_t
```
level (1=ERROR ... 5=TRACE) and a
```
const char *msg
```
valid only for the duration of the call. Pass
```
NULL
```
to unregister. Sensitive field values (password, token, text, ...) are replaced with
```
{"redacted":true}
```
. The callback must not unwind across the C ABI: catch C++ exceptions and Rust panics inside it, because a violation may abort the host. It may fire from several host threads at once and briefly after a
```
NULL
```
unregister: keep the callback and any data it captures valid for the process lifetime.
Wait.
```
ad_wait(adapter, args, &out)
```
runs the CLI
```
wait
```
command. Set
```
mode.pause
```
(
```
AdOptionalU64
```
, ms),
```
mode.element
```
,
```
mode.window
```
(waits for a window with that title),
```
mode.text
```
, or one of
```
mode.surfaces.menu
```
,
```
.menu_closed
```
,
```
.notification
```
; add the
```
predicate
```
group (
```
predicate
```
,
```
value
```
,
```
action
```
,
```
count
```
) and
```
scope.timeout_ms
```
/
```
scope.app
```
. The CLI event wait (
```
--event
```
,
```
--window-id
```
) is not exposed. Zero-initialize
```
AdWaitArgs
```
and validate the struct size against
```
AD_WAIT_ARGS_SIZE
```
/
```
ad_wait_args_size()
```
before calling. The output is a
```
{version, ok, command, data}
```
envelope freed with
```
ad_free_string
```
.
```
ad_wait
```
blocks the calling thread up to
```
scope.timeout_ms
```
ms. A concurrent
```
ad_adapter_destroy
```
does not invalidate an in-flight wait.
Text input privacy. On macOS, focus-fallback or headed text insertion may briefly use the clipboard for non-ASCII text. For sensitive text, prefer
```
AD_ACTION_KIND_SET_VALUE
```
with
```
AD_POLICY_KIND_HEADLESS
```
when the target supports settable values.
ad_get_tree_exact
vs
ad_snapshot
.
```
ad_get_tree_exact
```
returns a raw flat breadth-first tree: no refs, no refmap persistence, no JSON envelope (free it with
```
ad_free_tree
```
). Use it for custom traversal or UI inspection. For observe-act agents that drive actions via
```
ad_execute_by_ref
```
, start with
```
ad_snapshot
```
.

ABI 握手。 加载库后，调用
```
ad_init(AD_ABI_VERSION_MAJOR)
```
一次。如果版本不匹配，会返回
```
AD_RESULT_ERR_INVALID_ARGS
```
，调用方应中止。
```
ad_abi_version()
```
返回库的主要版本，供显示使用。当 symbol 被移除或布局发生变化时，主要版本会递增；新的
```
ad_*
```
symbol 和错误码只做增量添加。请固定所链接的准确
```
libagent_desktop_ffi
```
版本。
Session adapters。
```
ad_adapter_create_with_session("session-id")
```
将 refmap 持久化限定在一个 session namespace 中，效果类似 CLI
```
--session <id>
```
。 ID 长度为 1 到 64 个字符，只能包含 ASCII 字母数字字符、
```
-
```
或
```
_
```
。传入 null ID 时，其行为与
```
ad_adapter_create()
```
相同。无效 ID 会返回 null（读取
```
ad_last_error_*
```
）。
Ref actions。
```
ad_execute_by_ref
```
是主要的 action entrypoint。它加载 RefStore，严格解析 ref：snapshot 中不存在该 ref 或 element 已发生变化时返回
```
STALE_REF
```
，匹配到多个候选项时返回
```
AMBIGUOUS_TARGET
```
；随后执行实时 actionability preflight（目标可见、稳定、已启用、支持该操作、符合策略且可编辑），然后进行分发。disabled 或 unsupported target 会在任何 platform call 之前失败；发生
```
AD_RESULT_ERR_ACTION_FAILED
```
时，check report 会以 JSON 形式位于
```
ad_last_error_details()
```
中。它会在 preflight 前自动等待 5000 ms；
```
ad_execute_by_ref_timeout
```
接受
```
timeout_ms
```
（
```
-1
```
= 默认 5000 ms，
```
0
```
= 单次执行）。
```
PressKey
```
默认使用
```
focus_fallback
```
；其他所有 action（包括
```
TypeText
```
）默认使用
```
headless
```
。传入
```
AD_POLICY_KIND_FOCUS_FALLBACK
```
（1）可允许在不移动光标的情况下聚焦；传入
```
AD_POLICY_KIND_HEADED
```
（2）可允许物理光标移动和键盘输入。
```
ad_execute_ref_action_exact_with_policy
```
基于
```
AdExactRefEntry
```
执行相同的 preflight。
Generation-safe direct APIs。 每个 direct targeting function 都需要 process-generation evidence：
```
AdExactRefEntry
```
、
```
AdExactWindowInfo
```
、
```
ad_list_windows_exact
```
、
```
ad_launch_app_exact
```
以及其他
```
*_exact
```
symbol。
```
AdRefEntry
```
、
```
AdWindowInfo
```
和
```
AdSurfaceInfo
```
仅作为嵌入 exact structs 的字段存在。
```
ad_list_surfaces_exact
```
会保留 surface
```
id
```
。从
```
AD_EXACT_SURFACE_INFO_VERSION
```
2 开始，
```
AdExactSurfaceInfo.unclassified
```
会列出其 probe 无法读取的 surface kinds（例如
```
sheet,menu
```
），否则为 null。对于已列出的 kind，缺少
```
sheet
```
或
```
menu
```
表示未知，而不是不存在。只有 Windows adapter 会填充该字段；macOS 和 Linux 返回 null。
Native handles。
```
ad_resolve_element_exact
```
和
```
ad_find_exact
```
返回
```
AdNativeHandle
```
。
```
ad_execute_action
```
（headless、无 preflight）和
```
ad_execute_action_with_policy
```
直接作用于该 handle，绕过 ref-action pipeline。在所有平台上，都必须在
```
ad_adapter_destroy
```
之前，使用同一个 adapter 和同一个 thread，通过
```
ad_free_handle(adapter, &handle)
```
释放每个 handle。它会将
```
handle.ptr
```
清零，因此第二次调用是 no-op。

Action results。 对于 activation-chain actions，

AdActionResult.steps

与 CLI 的

steps

array 对应。每个

AdActionStep

包含

label

和

outcome

字符串、

mechanism

（

AD_STEP_MECHANISM_SEMANTIC_API

= 1 或

AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC

= 2，仅当

has_mechanism

时有效），以及

verified

（仅当

has_verified

时有效）。

AdActionResult.disposition

携带与

ad_last_error_delivery_semantics

相同的投递/重试组合。使用

ad_free_action_result(&out)

释放 result。

显示发现。 使用
```
AdScreenshotTarget.screen_index
```
前，先调用
```
ad_list_displays
```
。列表顺序就是 screenshot index 顺序。每个
```
AdDisplayInfo
```
包含稳定的 display ID、边界、是否为主显示器的标志和缩放比例。使用
```
ad_display_list_free
```
释放列表。
错误。 调用失败时会设置 thread-local last-error state；请在下一次失败调用前读取该状态。
```
ad_last_error_details()
```
返回结构化 JSON （
```
ACTION_FAILED
```
时为 actionability report，
```
AMBIGUOUS_TARGET
```
时为 candidate summaries），其中可能包含用户屏幕上的 element names、values 和 window titles：不要将其写入共享日志。
```
ad_last_error_delivery_semantics
```
报告失败 action 是否已投递，以及重试是否安全。无效的 enum 判别值会返回
```
AD_RESULT_ERR_INVALID_ARGS
```
。参见 error-handling.md。
线程模型。 Entrypoints 可从任意宿主线程调用。Native handles 必须保留在创建它们的 adapter 和 thread 上；跨线程时应传递限定引用。桌面修改操作采用与 CLI 相同的跨进程 interaction lease；读取操作不获取 lease，且具有有限 deadline。跨线程和进程的 read/read、read/mutation 和 mutation/mutation 矩阵参见 threading.md。
Tracing。 文件 tracing（与 CLI
```
--trace
```
使用相同的 JSONL 约定； segments 包含
```
event
```
、
```
ts_ms
```
、
```
seq
```
和已脱敏字段）仅在以下情况下启用： session 拥有
```
trace: on
```
的 manifest，且该 manifest 由
```
session start
```
（或磁盘上的等效 manifest）在 adapter 创建前生成。仅调用
```
ad_adapter_create_with_session
```
不会写入任何内容。每个 OS 进程会向一个 segment 追加数据，路径为
```
~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl
```
， long-lived host 会复用该 segment。
```
ad_trace_show
```
（JSON envelope）和
```
ad_trace_export
```
（自包含 HTML 文件）读取 adapter session 的合并时间线；
```
limit
```
为
```
-1
```
时表示 CLI 默认值。
Log callback。
```
ad_set_log_callback(cb)
```
会注册一个 process-wide callback 用于诊断，无论 session manifest 是否存在。它以 JSON 形式接收同一 thread 上后续
```
ad_*
```
调用内发出的 tracing events；FFI 绝不会安装或替换宿主进程的 global tracing subscriber。
```
cb
```
接收一个
```
int32_t
```
level （1=ERROR ... 5=TRACE）和一个
```
const char *msg
```
，后者仅在调用期间有效。传入
```
NULL
```
可注销 callback。敏感字段值（password、token、text 等）会被替换为
```
{"redacted":true}
```
。Callback 不得跨 C ABI unwind：必须在内部捕获 C++ exceptions 和 Rust panics，否则可能导致宿主进程中止。它可能同时从多个宿主线程触发，也可能在
```
NULL
```
注销后短暂触发：请确保 callback 及其捕获的任何数据在进程生命周期内始终有效。
Wait。
```
ad_wait(adapter, args, &out)
```
运行 CLI
```
wait
```
command。设置
```
mode.pause
```
（
```
AdOptionalU64
```
，ms）、
```
mode.element
```
、
```
mode.window
```
（等待具有该标题的窗口）、
```
mode.text
```
，或
```
mode.surfaces.menu
```
、
```
.menu_closed
```
、
```
.notification
```
之一；添加
```
predicate
```
group（
```
predicate
```
、
```
value
```
、
```
action
```
、
```
count
```
）以及
```
scope.timeout_ms
```
/
```
scope.app
```
。CLI event wait（
```
--event
```
、
```
--window-id
```
）未通过 ABI 暴露。调用前先对
```
AdWaitArgs
```
进行零初始化，并对照
```
AD_WAIT_ARGS_SIZE
```
/
```
ad_wait_args_size()
```
验证结构体大小。输出为
```
{version, ok, command, data}
```
envelope，使用
```
ad_free_string
```
释放。
```
ad_wait
```
最多阻塞调用线程
```
scope.timeout_ms
```
ms。与其并发执行的
```
ad_adapter_destroy
```
不会使执行中的 wait 失效。
文本输入隐私。 在 macOS 上，focus-fallback 或 headed text insertion 可能会为非 ASCII 文本短暂使用剪贴板。对于敏感文本，如果 target 支持可设置值，请优先使用
```
AD_ACTION_KIND_SET_VALUE
```
搭配
```
AD_POLICY_KIND_HEADLESS
```
。
ad_get_tree_exact
vs
ad_snapshot
。
```
ad_get_tree_exact
```
返回原始的扁平 breadth-first tree：不含 refs、不持久化 refmap，也不使用 JSON envelope （使用
```
ad_free_tree
```
释放）。将其用于 custom traversal 或 UI inspection。对于通过
```
ad_execute_by_ref
```
驱动 actions 的 observe-act Agent，请从
```
ad_snapshot
```
开始。