sim-use

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

0. Preflight

0. 预检步骤

Before first interaction with a device, run the preflight check:

bash

python3 scripts/preflight.py --device <UDID>

This verifies sim-use is installed, the device is reachable, and the daemon is healthy. If you don't have the script, do the checks manually:

```
sim-use --version
```
— confirm sim-use is on PATH.
```
sim-use devices
```
— confirm the target device is listed and booted/connected.
```
sim-use ui --device <UDID>
```
— confirm you can read the screen.

--device

is optional when only one simulator is booted or one daemon is running. For Android, run

sim-use android init --device <serial>

once to install the bridge APK.

首次与设备交互前，请运行预检检查：

bash

python3 scripts/preflight.py --device <UDID>

此命令会验证sim-use已安装、设备可连接且守护进程运行正常。若没有该脚本，请手动执行以下检查：

```
sim-use --version
```
— 确认sim-use已添加至PATH。
```
sim-use devices
```
— 确认目标设备已列出且处于启动/连接状态。
```
sim-use ui --device <UDID>
```
— 确认可读取屏幕内容。

当仅启动了一个模拟器或运行了一个守护进程时，

--device

参数为可选。对于Android设备，需运行一次

sim-use android init --device <serial>

来安装桥接APK。

1. The observe-act loop

1. 观察-执行循环

Every interaction follows the same cycle: observe → act → verify.

每次交互都遵循相同的流程：观察 → 执行 → 验证。

Observe

观察

bash

sim-use ui --device <UDID>

Read the outline. Each element has an

@N

alias and optionally a

#<id>

identifier. List cells carry

#N

(dominant list) or

#N@M

(scoped).

Frames in the JSON output (

--json

entries[].frame

screen

) are in platform-native units — iOS points, Android pixels. Key off the envelope's

platform

field before doing math on coordinates across platforms. Always pair

--json

with

--no-raw

— see Keeping output small below.

bash

sim-use ui --device <UDID>

读取屏幕大纲。每个元素都有一个

@N

别名，可选带有

#<id>

标识符。列表单元格带有

#N

（主列表）或

#N@M

（范围限定）标记。

JSON输出中的帧信息（

--json

模式下的

entries[].frame

、

screen

）采用平台原生单位——iOS为点（points），Android为像素（pixels）。在跨平台进行坐标计算前，请先查看返回结果中的

platform

字段。使用

--json

时务必搭配

--no-raw

参数——详见下方「精简输出内容」部分。

Act

执行

Pick a selector, in order of preference:

Selector	When to use
`tap @N`	Right after `ui` . Fastest, cache-backed.
`tap #<id>`	Stable across minor layout changes. Paste from the outline.
`tap --label 'X'`	Scripted flows. Combine with `--wait-timeout` for transitions.
`tap --label-regex '...'`	Dynamic labels with counters/timestamps. Anchor with `^...$` .
`tap --label-contains 'X'`	Substring match when exact label is unknown.
`tap -x N -y N` / `tap --point x,y`	Last resort for elements with no AX data.

Disambiguate collisions with

--element-type

--frame minY=0.7r

(see

references/cheatsheet.md

按优先级选择选择器：

选择器	使用场景
`tap @N`	刚执行完 `ui` 命令后使用。速度最快，基于缓存。
`tap #<id>`	适用于小幅度布局变更后的稳定操作。直接从屏幕大纲中复制标识符。
`tap --label 'X'`	脚本化流程。可搭配 `--wait-timeout` 参数处理页面过渡。
`tap --label-regex '...'`	带有计数器/时间戳的动态标签。使用 `^...$` 锚定匹配。
`tap --label-contains 'X'`	精确标签未知时，使用子字符串匹配。
`tap -x N -y N` / `tap --point x,y`	无辅助功能（AX）数据元素的最后选择方案。

可通过

--element-type

或

--frame minY=0.7r

参数消除选择器冲突（详见

references/cheatsheet.md

）。

Verify

验证

Always verify after acting — commands are fire-and-forget:

bash

sim-use ui --device <UDID>       # read the new screen state
sim-use screenshot --device <UDID> --output after.png

执行操作后务必进行验证——命令为“触发即遗忘”模式：

bash

sim-use ui --device <UDID>       # 读取新的屏幕状态
sim-use screenshot --device <UDID> --output after.png

Keeping output small

精简输出内容

Every byte of command output you read costs context. Defaults that keep the loop cheap:

Prefer the default text outline over
```
--json
```
. The outline carries everything a tap needs (
```
@N
```
/
```
#<id>
```
aliases, roles, frames, states); reach for
```
--json
```
when you need structured fields for coordinate math (
```
entries[].frame
```
,
```
screen
```
) or full untruncated text (the outline truncates labels at 60 graphemes,
```
value=
```
at 30).
When you do use
```
--json
```
, add
```
--no-raw
```
.
```
data.raw
```
is the raw accessibility tree — typically the bulk of the envelope's bytes, and useful only for debugging sim-use itself.
One
```
ui
```
per action: the Verify read of step N is the Observe read of step N+1. Don't run a second
```
ui
```
in between.
Verify with the text outline, not a screenshot. Reading a screenshot costs several times more than a typical outline; take one only when the check is genuinely visual (colors, images, layout).
On iOS, to wait out a transition, prefer
```
tap --label 'X' --wait-timeout 3
```
(polls for the element) over re-running
```
ui
```
in a loop. Android
```
tap
```
has no
```
--wait-timeout
```
; use
```
sleep
```
between commands instead.
For a known multi-step sequence on iOS, use
```
sim-use ios batch
```
(see
```
references/batch-reference.md
```
) — one invocation, one output.

读取的命令输出每一字节都会占用上下文资源。以下默认设置可降低循环成本：

优先使用默认文本大纲而非
```
--json
```
模式。文本大纲包含点击操作所需的全部信息（
```
@N
```
/
```
#<id>
```
别名、角色、帧信息、状态）；仅当需要结构化字段进行坐标计算（
```
entries[].frame
```
、
```
screen
```
）或完整未截断文本（文本大纲会将标签截断为60个字符，
```
value=
```
字段截断为30个字符）时，才使用
```
--json
```
模式。
使用
```
--json
```
模式时，添加
```
--no-raw
```
参数。
```
data.raw
```
是原始辅助功能树——通常占返回结果的大部分字节，仅对调试sim-use本身有用。
每个操作对应一次
```
ui
```
命令：第N步的验证读取即为第N+1步的观察读取。不要在中间重复执行
```
ui
```
命令。
使用文本大纲进行验证，而非截图。读取截图的成本是普通文本大纲的数倍；仅当检查内容为纯视觉元素（颜色、图片、布局）时才进行截图。
在iOS上等待页面过渡时，优先使用
```
tap --label 'X' --wait-timeout 3
```
（轮询元素）而非循环执行
```
ui
```
命令。Android的
```
tap
```
命令无
```
--wait-timeout
```
参数；可在命令间使用
```
sleep
```
命令替代。
对于iOS上已知的多步骤序列，使用
```
sim-use ios batch
```
命令（详见
```
references/batch-reference.md
```
）——一次调用即可完成所有步骤并返回结果。

Common moves

常用操作

Task	Command
Scroll down	`sim-use gesture scroll-up --device <UDID>` (scroll-up = content moves up = page down)
Type text	`sim-use type 'hello' --device <UDID>`
Paste unicode	`sim-use paste 'こんにちは 🎉' --device <UDID>` (iOS: needs hardware keyboard)
Hardware button	`sim-use button home --device <UDID>`
Android back	`sim-use button back --device <UDID>`
Wait for animation	`sleep 0.4` between commands, or `--pre-delay 0.5`
Toggle/switch	`sim-use tap @N --duration 0.05 --device <UDID>` (UISwitch needs a brief hold)
Swipe	`sim-use swipe --from 50,500 --to 350,500 --device <UDID>`
Pinch zoom in	`sim-use gesture pinch-out --device <UDID>` (two-finger spread)
Rotate	`sim-use gesture rotate-cw --angle 90 --device <UDID>`
Record evidence GIF	`sim-use record-video --output demo.gif --device <UDID>` — stop with SIGINT/SIGTERM (never SIGKILL); transcodes after stop; auto-plays inline in PRs

任务	命令
向下滚动	`sim-use gesture scroll-up --device <UDID>` （scroll-up = 内容向上移动 = 页面向下滚动）
输入文本	`sim-use type 'hello' --device <UDID>`
粘贴Unicode文本	`sim-use paste 'こんにちは 🎉' --device <UDID>` （iOS：需启用硬件键盘）
硬件按键操作	`sim-use button home --device <UDID>`
Android返回键	`sim-use button back --device <UDID>`
等待动画完成	在命令间添加 `sleep 0.4` ，或使用 `--pre-delay 0.5` 参数
切换开关	`sim-use tap @N --duration 0.05 --device <UDID>` （UISwitch需要短暂按住）
滑动操作	`sim-use swipe --from 50,500 --to 350,500 --device <UDID>`
捏合放大	`sim-use gesture pinch-out --device <UDID>` （双指张开）
旋转屏幕	`sim-use gesture rotate-cw --angle 90 --device <UDID>`
录制演示GIF	`sim-use record-video --output demo.gif --device <UDID>` — 使用SIGINT/SIGTERM停止录制（禁止使用SIGKILL）；停止后会自动转码；可在PR中自动内嵌播放

2. Pitfalls

2. 常见问题

Quick symptom index — see

references/pitfalls.md

for detailed recipes.

Symptom	Cause	Fix
`tap --label` hits wrong element	Label collision (e.g. header and tab bar share text)	Add `--frame minY=0.7r` or `--element-type` to narrow
`tap @N` fails after navigation	Alias cache is stale	Re-run `ui` before tapping
`App:` line shows wrong app	System layer (alert, share sheet) is on top	Dismiss it first, then re-run `ui`
`multipleMatches` error	Several elements share the selector	Use `--frame` , `--element-type` , or a more specific selector
Tap lands but nothing happens	Animation in progress, or element not yet interactive	Add `--pre-delay 0.3` or `--wait-timeout 3`
iOS: `paste` drops text	Soft keyboard only; HID Cmd+V is ignored	Use `paste --via-menu --target-id <id>`
Android: `paste` denied	Background clipboard access blocked	Use `type` instead
Outline shows `U+FFFC` in label	iOS icon placeholder character	Match with `--label-regex` excluding the prefix
`[i] … covers ~N% of the screen` warning (text output, or `--json` top-level `advisory` key)	The selector resolved to a near-full-screen wrapper (common on Flutter/canvas UIs) and the tap hit its center, likely missing the intended control	Re-run `ui` and target the control via `@N` / `#<id>` , or pass explicit `-x/-y` / `--point`
`[i] Screen orientation could not be confirmed…` / `…coordinates may be stale…` advisory	Device/app is rotated (the `App:` header shows a tag like `(landscape-right)` ) and orientation self-calibration couldn't verify the mapping, or the `@N` snapshot predates a rotation	Re-run `ui` and tap again; selectors handle rotation automatically once calibration succeeds. Explicit `-x/-y` / `--point` is device-native portrait space by default — on `swipe` / `touch` , pass `--coordinate-space ui` to use outline (visual-space) coordinates on a rotated device

快速症状索引——详细解决方案请见

references/pitfalls.md

。

症状	原因	修复方案
`tap --label` 命中错误元素	标签冲突（如标题栏和标签栏使用相同文本）	添加 `--frame minY=0.7r` 或 `--element-type` 参数缩小范围
导航后 `tap @N` 执行失败	别名缓存已过期	点击前重新执行 `ui` 命令
`App:` 行显示错误应用	系统层（弹窗、分享面板）处于顶层	先关闭系统层元素，再重新执行 `ui` 命令
出现 `multipleMatches` 错误	多个元素匹配同一选择器	使用 `--frame` 、 `--element-type` 或更具体的选择器
点击生效但无反应	动画正在进行，或元素尚未可交互	添加 `--pre-delay 0.3` 或 `--wait-timeout 3` 参数
iOS： `paste` 命令丢失文本	仅使用软键盘；HID Cmd+V被忽略	使用 `paste --via-menu --target-id <id>` 命令
Android： `paste` 命令被拒绝	后台剪贴板访问被阻止	使用 `type` 命令替代
大纲中标签显示 `U+FFFC`	iOS图标占位符字符	使用 `--label-regex` 参数排除前缀进行匹配
出现 `[i] … covers ~N% of the screen` 警告（文本输出或 `--json` 模式下的顶层 `advisory` 键）	选择器匹配到近乎全屏的容器（Flutter/画布UI中常见），点击命中容器中心，可能未击中目标控件	重新执行 `ui` 命令并通过 `@N` / `#<id>` 定位目标控件，或传入明确的 `-x/-y` / `--point` 参数
出现 `[i] Screen orientation could not be confirmed…` / `…coordinates may be stale…` 提示	设备/应用已旋转（ `App:` 标题栏显示 `(landscape-right)` 等标记），且方向自校准无法验证映射关系，或 `@N` 快照早于旋转操作	重新执行 `ui` 命令后再次点击；校准成功后选择器会自动处理旋转。默认情况下，明确的 `-x/-y` / `--point` 参数使用设备原生竖屏坐标——在旋转设备上执行 `swipe` / `touch` 操作时，传入 `--coordinate-space ui` 参数可使用大纲（视觉空间）坐标

3. Crash awareness

3. 崩溃感知

See

references/crash-awareness.md

for the full protocol. Summary:

sim-use watches for the target process disappearing between commands. When it detects a crash:

================ PROCESS DISAPPEARED ================
com.example.app (pid 12345) was alive at the previous command and is GONE now.

On Android,

ui

also detects the AOSP system crash dialog directly from the accessibility tree.

Mandatory response:

STOP. Do not silently relaunch or continue.
Report the crash to the user with the banner text.
Wait for instructions before proceeding.

After an intentional relaunch, call

sim-use app-state --reset

to clear the signal.

完整协议请见

references/crash-awareness.md

。摘要：

sim-use会监控命令间目标进程是否消失。检测到崩溃时会输出：

================ PROCESS DISAPPEARED ================
com.example.app (pid 12345) was alive at the previous command and is GONE now.

在Android上，

ui

命令还会直接从辅助功能树中检测AOSP系统崩溃弹窗。

强制响应流程：

立即停止操作。请勿静默重启或继续执行。
将崩溃提示横幅内容告知用户。
等待用户指令后再继续。

主动重启应用后，调用

sim-use app-state --reset

命令清除崩溃信号。

4. Escalation

4. 升级处理

Stop and ask the user when:

A selector collision cannot be resolved with available disambiguators.
Preflight fails and autofix does not recover.
The task requires a destructive action (deleting data, uninstalling an app).
You've retried the same action 3 times without progress.

出现以下情况时，请停止操作并询问用户：

无法通过现有参数解决选择器冲突。
预检失败且自动修复无法恢复。
任务需要执行破坏性操作（删除数据、卸载应用）。
同一操作重试3次仍无进展。

5. Exit checklist

5. 完成检查清单

Before reporting a task as complete:

Run
```
sim-use ui
```
(or
```
screenshot
```
) to capture the final state.
Confirm the screen matches the intended outcome.
If the outcome is ambiguous, show the final
```
ui
```
output or screenshot to the user.

报告任务完成前，请执行以下操作：

运行
```
sim-use ui
```
（或
```
screenshot
```
）命令捕获最终状态。
确认屏幕状态与预期结果一致。
若结果不明确，将最终的
```
ui
```
输出或截图展示给用户。