sim-use

0. Preflight

Before first interaction with a device, run the preflight check:

bash

python3 scripts/preflight.py --device <UDID>

This verifies sim-use is installed, the device is reachable, and the daemon is healthy. If you don't have the script, do the checks manually:

```
sim-use --version
```
— confirm sim-use is on PATH.
```
sim-use devices
```
— confirm the target device is listed and booted/connected.
```
sim-use ui --device <UDID>
```
— confirm you can read the screen.

--device

is optional when only one simulator is booted or one daemon is running. For Android, run

sim-use android init --device <serial>

once to install the bridge APK.

1. The observe-act loop

Every interaction follows the same cycle: observe → act → verify.

Observe

bash

sim-use ui --device <UDID>

Read the outline. Each element has an

@N

alias and optionally a

#<id>

identifier. List cells carry

#N

(dominant list) or

#N@M

(scoped).

Frames in the JSON output (

--json

entries[].frame

screen

) are in platform-native units — iOS points, Android pixels. Key off the envelope's

platform

field before doing math on coordinates across platforms. Always pair

--json

with

--no-raw

— see Keeping output small below.

Act

Pick a selector, in order of preference:

Selector	When to use
`tap @N`	Right after `ui` . Fastest, cache-backed.
`tap #<id>`	Stable across minor layout changes. Paste from the outline.
`tap --label 'X'`	Scripted flows. Combine with `--wait-timeout` for transitions.
`tap --label-regex '...'`	Dynamic labels with counters/timestamps. Anchor with `^...$` .
`tap --label-contains 'X'`	Substring match when exact label is unknown.
`tap -x N -y N` / `tap --point x,y`	Last resort for elements with no AX data.

Disambiguate collisions with

--element-type

--frame minY=0.7r

(see

references/cheatsheet.md

Verify

Always verify after acting — commands are fire-and-forget:

bash

sim-use ui --device <UDID>       # read the new screen state
sim-use screenshot --device <UDID> --output after.png

Keeping output small

Every byte of command output you read costs context. Defaults that keep the loop cheap:

Prefer the default text outline over
```
--json
```
. The outline carries everything a tap needs (
```
@N
```
/
```
#<id>
```
aliases, roles, frames, states); reach for
```
--json
```
when you need structured fields for coordinate math (
```
entries[].frame
```
,
```
screen
```
) or full untruncated text (the outline truncates labels at 60 graphemes,
```
value=
```
at 30).
When you do use
```
--json
```
, add
```
--no-raw
```
.
```
data.raw
```
is the raw accessibility tree — typically the bulk of the envelope's bytes, and useful only for debugging sim-use itself.
One
```
ui
```
per action: the Verify read of step N is the Observe read of step N+1. Don't run a second
```
ui
```
in between.
Verify with the text outline, not a screenshot. Reading a screenshot costs several times more than a typical outline; take one only when the check is genuinely visual (colors, images, layout).
On iOS, to wait out a transition, prefer
```
tap --label 'X' --wait-timeout 3
```
(polls for the element) over re-running
```
ui
```
in a loop. Android
```
tap
```
has no
```
--wait-timeout
```
; use
```
sleep
```
between commands instead.
For a known multi-step sequence on iOS, use
```
sim-use ios batch
```
(see
```
references/batch-reference.md
```
) — one invocation, one output.

Common moves

Task	Command
Scroll down	`sim-use gesture scroll-up --device <UDID>` (scroll-up = content moves up = page down)
Type text	`sim-use type 'hello' --device <UDID>`
Paste unicode	`sim-use paste 'こんにちは 🎉' --device <UDID>` (iOS: needs hardware keyboard)
Hardware button	`sim-use button home --device <UDID>`
Android back	`sim-use button back --device <UDID>`
Wait for animation	`sleep 0.4` between commands, or `--pre-delay 0.5`
Toggle/switch	`sim-use tap @N --duration 0.05 --device <UDID>` (UISwitch needs a brief hold)
Swipe	`sim-use swipe --from 50,500 --to 350,500 --device <UDID>`
Pinch zoom in	`sim-use gesture pinch-out --device <UDID>` (two-finger spread)
Rotate	`sim-use gesture rotate-cw --angle 90 --device <UDID>`
Record evidence GIF	`sim-use record-video --output demo.gif --device <UDID>` — stop with SIGINT/SIGTERM (never SIGKILL); transcodes after stop; auto-plays inline in PRs

2. Pitfalls

Quick symptom index — see

references/pitfalls.md

for detailed recipes.

Symptom	Cause	Fix
`tap --label` hits wrong element	Label collision (e.g. header and tab bar share text)	Add `--frame minY=0.7r` or `--element-type` to narrow
`tap @N` fails after navigation	Alias cache is stale	Re-run `ui` before tapping
`App:` line shows wrong app	System layer (alert, share sheet) is on top	Dismiss it first, then re-run `ui`
`multipleMatches` error	Several elements share the selector	Use `--frame` , `--element-type` , or a more specific selector
Tap lands but nothing happens	Animation in progress, or element not yet interactive	Add `--pre-delay 0.3` or `--wait-timeout 3`
iOS: `paste` drops text	Soft keyboard only; HID Cmd+V is ignored	Use `paste --via-menu --target-id <id>`
Android: `paste` denied	Background clipboard access blocked	Use `type` instead
Outline shows `U+FFFC` in label	iOS icon placeholder character	Match with `--label-regex` excluding the prefix
`[i] … covers ~N% of the screen` warning (text output, or `--json` top-level `advisory` key)	The selector resolved to a near-full-screen wrapper (common on Flutter/canvas UIs) and the tap hit its center, likely missing the intended control	Re-run `ui` and target the control via `@N` / `#<id>` , or pass explicit `-x/-y` / `--point`
`[i] Screen orientation could not be confirmed…` / `…coordinates may be stale…` advisory	Device/app is rotated (the `App:` header shows a tag like `(landscape-right)` ) and orientation self-calibration couldn't verify the mapping, or the `@N` snapshot predates a rotation	Re-run `ui` and tap again; selectors handle rotation automatically once calibration succeeds. Explicit `-x/-y` / `--point` is device-native portrait space by default — on `swipe` / `touch` , pass `--coordinate-space ui` to use outline (visual-space) coordinates on a rotated device

3. Crash awareness

See

references/crash-awareness.md

for the full protocol. Summary:

sim-use watches for the target process disappearing between commands. When it detects a crash:

================ PROCESS DISAPPEARED ================
com.example.app (pid 12345) was alive at the previous command and is GONE now.

On Android,

ui

also detects the AOSP system crash dialog directly from the accessibility tree.

Mandatory response:

STOP. Do not silently relaunch or continue.
Report the crash to the user with the banner text.
Wait for instructions before proceeding.

After an intentional relaunch, call

sim-use app-state --reset

to clear the signal.

4. Escalation

Stop and ask the user when:

A selector collision cannot be resolved with available disambiguators.
Preflight fails and autofix does not recover.
The task requires a destructive action (deleting data, uninstalling an app).
You've retried the same action 3 times without progress.

5. Exit checklist

Before reporting a task as complete:

Run
```
sim-use ui
```
(or
```
screenshot
```
) to capture the final state.
Confirm the screen matches the intended outcome.
If the outcome is ambiguous, show the final
```
ui
```
output or screenshot to the user.

sim-use

NPX Install

Tags

SKILL.md Content

0. Preflight

1. The observe-act loop

Observe

Act

Verify

Keeping output small

Common moves

2. Pitfalls

3. Crash awareness

4. Escalation

5. Exit checklist