agent-desktop-ffi

Direct C-ABI access to PlatformAdapter operations. Build the cdylib with the

release-ffi

profile:

cargo build --profile release-ffi -p agent-desktop-ffi

The output is

target/release-ffi/libagent_desktop_ffi.dylib

(

.so

on Linux,

agent_desktop_ffi.dll

on Windows) plus a committed C header at

crates/ffi/include/agent_desktop.h

. The default

release

profile uses

panic = "abort"

: a Rust panic inside an

extern "C"

function would

SIGABRT

the host.

release-ffi

keeps

panic = "unwind"

, which the panic traps need.

Four reference topics, loaded as needed:

ownership.md: who allocates and who frees every pointer the FFI hands back.
error-handling.md: errno-style last-error contract, error codes, enum validation, panic boundary.
threading.md: host-thread contract, mutation ordering across threads and processes, adapter-bound native handles, AXIsProcessTrusted inheritance.
build-and-link.md: prebuilt archives, ABI handshake, struct size validation, C and Python examples.

Observe-act workflow (canonical path)

ad_init(AD_ABI_VERSION_MAJOR)                    // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1")   // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)

ad_snapshot

returns the CLI envelope

{version, ok, command, data}

data.tree

carries snapshot-qualified refs for interactive elements. Read

data.complete

before trusting a tree:

false

means the observation budget ran out and the tree is partial, not an error.

ad_snapshot

always targets the focused window of

app

; window targeting, skeleton mode and

--root

drill-down are not exposed over the ABI. A bare

@eN

ref returns

AD_RESULT_ERR_INVALID_ARGS

Core constraints

ABI handshake. Call
```
ad_init(AD_ABI_VERSION_MAJOR)
```
once after loading the library. A mismatch returns
```
AD_RESULT_ERR_INVALID_ARGS
```
: abort.
```
ad_abi_version()
```
returns the library's major for display. The major increments on a removed symbol or a layout change; new
```
ad_*
```
symbols and error codes are additive. Pin the exact
```
libagent_desktop_ffi
```
version you link against.
Session adapters.
```
ad_adapter_create_with_session("session-id")
```
scopes refmap persistence to a session namespace, like CLI
```
--session <id>
```
. IDs are 1 to 64 ASCII alphanumeric,
```
-
```
, or
```
_
```
characters. A null ID behaves like
```
ad_adapter_create()
```
. An invalid ID returns null (read
```
ad_last_error_*
```
).
Ref actions.
```
ad_execute_by_ref
```
is the primary action entrypoint. It loads the RefStore, resolves the ref strictly (
```
STALE_REF
```
when the snapshot has no such ref or the element changed,
```
AMBIGUOUS_TARGET
```
on several candidates), runs the live actionability preflight (visible, stable, enabled, supported action, policy, editable), then dispatches. A disabled or unsupported target fails before any platform call; on
```
AD_RESULT_ERR_ACTION_FAILED
```
the check report is JSON in
```
ad_last_error_details()
```
. It auto-waits 5000 ms before the preflight;
```
ad_execute_by_ref_timeout
```
takes
```
timeout_ms
```
(
```
-1
```
= 5000 ms default,
```
0
```
= single-shot).
```
PressKey
```
defaults to
```
focus_fallback
```
; every other action, including
```
TypeText
```
, defaults to
```
headless
```
. Pass
```
AD_POLICY_KIND_FOCUS_FALLBACK
```
(1) to allow focus without cursor movement, or
```
AD_POLICY_KIND_HEADED
```
(2) to allow physical cursor and keyboard delivery.
```
ad_execute_ref_action_exact_with_policy
```
runs the same preflight from an
```
AdExactRefEntry
```
.
Generation-safe direct APIs. Every direct targeting function takes process-generation evidence:
```
AdExactRefEntry
```
,
```
AdExactWindowInfo
```
,
```
ad_list_windows_exact
```
,
```
ad_launch_app_exact
```
, and the other
```
*_exact
```
symbols.
```
AdRefEntry
```
,
```
AdWindowInfo
```
, and
```
AdSurfaceInfo
```
exist only as fields embedded in the exact structs.
```
ad_list_surfaces_exact
```
keeps the surface
```
id
```
. From
```
AD_EXACT_SURFACE_INFO_VERSION
```
2,
```
AdExactSurfaceInfo.unclassified
```
names the surface kinds its probe could not read (for example
```
sheet,menu
```
), or is null. A missing
```
sheet
```
or
```
menu
```
next to a listed kind is unknown, not absent. Only the Windows adapter fills it; macOS and Linux return null.
Native handles.
```
ad_resolve_element_exact
```
and
```
ad_find_exact
```
return an
```
AdNativeHandle
```
.
```
ad_execute_action
```
(headless, no preflight) and
```
ad_execute_action_with_policy
```
act on it directly, bypassing the ref-action pipeline. Release every handle with
```
ad_free_handle(adapter, &handle)
```
on the same adapter and thread, before
```
ad_adapter_destroy
```
, on every platform. It zeroes
```
handle.ptr
```
, so a second call is a no-op.

Action results.

AdActionResult.steps

mirrors the CLI

steps

array for activation-chain actions. Each

AdActionStep

has

label

and

outcome

strings,

mechanism

(

AD_STEP_MECHANISM_SEMANTIC_API

= 1 or

AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC

= 2, valid only when

has_mechanism

), and

verified

(valid only when

has_verified

AdActionResult.disposition

carries the same delivery/retry pair as

ad_last_error_delivery_semantics

. Release the result with

ad_free_action_result(&out)

Display discovery. Call
```
ad_list_displays
```
before using
```
AdScreenshotTarget.screen_index
```
. List order is the screenshot index order. Each
```
AdDisplayInfo
```
carries a stable display ID, bounds, primary flag, and scale. Release the list with
```
ad_display_list_free
```
.
Errors. A failing call sets thread-local last-error state; read it with the
```
ad_last_error_*
```
accessors before the next failing call.
```
ad_last_error_details()
```
returns structured JSON (the actionability report on
```
ACTION_FAILED
```
, candidate summaries on
```
AMBIGUOUS_TARGET
```
) that can contain element names, values, and window titles from the user's screen: keep it out of shared logs.
```
ad_last_error_delivery_semantics
```
reports whether the failed action was delivered and whether a retry is safe. Invalid enum discriminants return
```
AD_RESULT_ERR_INVALID_ARGS
```
. See error-handling.md.
Threading. Entrypoints may be called from any host thread. Native handles stay on the adapter and thread that created them; pass qualified refs across threads. Desktop mutations take the same cross-process interaction lease as the CLI; reads take no lease and carry finite deadlines. See threading.md for the read/read, read/mutation, and mutation/mutation matrix.
Tracing. File tracing (same JSONL contract as CLI
```
--trace
```
; segments carry
```
event
```
,
```
ts_ms
```
,
```
seq
```
, and redacted fields) is active only when the session has a manifest with
```
trace: on
```
, created by
```
session start
```
(or an equivalent manifest on disk) before the adapter is created.
```
ad_adapter_create_with_session
```
alone writes nothing. Each OS process appends to one segment,
```
~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl
```
, and a long-lived host reuses it.
```
ad_trace_show
```
(JSON envelope) and
```
ad_trace_export
```
(self-contained HTML file) read the adapter session's merged timeline;
```
limit
```
```
-1
```
= CLI default.
Log callback.
```
ad_set_log_callback(cb)
```
registers one process-wide callback for diagnostics regardless of session manifest. It receives, as JSON, the
```
tracing
```
events emitted inside later
```
ad_*
```
calls on the same thread; the FFI never installs or replaces the host process's global tracing subscriber.
```
cb
```
receives an
```
int32_t
```
level (1=ERROR ... 5=TRACE) and a
```
const char *msg
```
valid only for the duration of the call. Pass
```
NULL
```
to unregister. Sensitive field values (password, token, text, ...) are replaced with
```
{"redacted":true}
```
. The callback must not unwind across the C ABI: catch C++ exceptions and Rust panics inside it, because a violation may abort the host. It may fire from several host threads at once and briefly after a
```
NULL
```
unregister: keep the callback and any data it captures valid for the process lifetime.
Wait.
```
ad_wait(adapter, args, &out)
```
runs the CLI
```
wait
```
command. Set
```
mode.pause
```
(
```
AdOptionalU64
```
, ms),
```
mode.element
```
,
```
mode.window
```
(waits for a window with that title),
```
mode.text
```
, or one of
```
mode.surfaces.menu
```
,
```
.menu_closed
```
,
```
.notification
```
; add the
```
predicate
```
group (
```
predicate
```
,
```
value
```
,
```
action
```
,
```
count
```
) and
```
scope.timeout_ms
```
/
```
scope.app
```
. The CLI event wait (
```
--event
```
,
```
--window-id
```
) is not exposed. Zero-initialize
```
AdWaitArgs
```
and validate the struct size against
```
AD_WAIT_ARGS_SIZE
```
/
```
ad_wait_args_size()
```
before calling. The output is a
```
{version, ok, command, data}
```
envelope freed with
```
ad_free_string
```
.
```
ad_wait
```
blocks the calling thread up to
```
scope.timeout_ms
```
ms. A concurrent
```
ad_adapter_destroy
```
does not invalidate an in-flight wait.
Text input privacy. On macOS, focus-fallback or headed text insertion may briefly use the clipboard for non-ASCII text. For sensitive text, prefer
```
AD_ACTION_KIND_SET_VALUE
```
with
```
AD_POLICY_KIND_HEADLESS
```
when the target supports settable values.
ad_get_tree_exact
vs
ad_snapshot
.
```
ad_get_tree_exact
```
returns a raw flat breadth-first tree: no refs, no refmap persistence, no JSON envelope (free it with
```
ad_free_tree
```
). Use it for custom traversal or UI inspection. For observe-act agents that drive actions via
```
ad_execute_by_ref
```
, start with
```
ad_snapshot
```
.

agent-desktop-ffi

NPX Install

Tags

SKILL.md Content

agent-desktop-ffi

Observe-act workflow (canonical path)

Core constraints