device-interaction

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Device Interaction

设备交互

TRIGGER when: user asks to verify/test/check if the app works on device, after implementing a UI-affecting feature that needs device verification, user says "does it work", "test this", "check on device", user reports UI doesn't work as expected, need to debug touch/interaction issues. DO NOT TRIGGER when: user asks about unit tests only, build-only requests without device testing, code review without device testing, simulator configuration questions, changes that don't affect UI (e.g. comments, refactors, non-UI logic).

触发场景：用户询问是否在设备上验证/测试/检查应用运行情况、实现了需要设备验证的UI相关功能后、用户说“它能用吗”“测试这个”“在设备上检查”、用户反馈UI未按预期工作、需要调试触摸/交互问题。不触发场景：用户仅询问单元测试、仅构建不涉及设备测试的请求、不涉及设备测试的代码审查、模拟器配置问题、不影响UI的更改（如注释、重构、非UI逻辑）。

For the Main Agent

主Agent说明

This is a SUBAGENT skill. Invoke it via the Agent tool when device verification is needed. If there is an open session for that work, provide that session identifier to a subagent for exclusive use by that subagent.

Agent tool:
- subagent_type: "general-purpose"
- description: "Verify login feature works"
- prompt: "Using the device-interaction skill, verify that the login feature works correctly on session <session-identifier>. Launch the app, capture screenshot and UI hierarchy, check that the login button is visible and tappable, and report if the implementation is working correctly."

After implementing a UI-affecting feature, invoke this skill to verify the implementation works on a device.

这是一个SUBAGENT技能。 当需要设备验证时，通过Agent工具调用它。如果存在该工作的开放会话，请将会话标识符提供给子Agent，供其专属使用。

Agent tool:
- subagent_type: "general-purpose"
- description: "Verify login feature works"
- prompt: "Using the device-interaction skill, verify that the login feature works correctly on session <session-identifier>. Launch the app, capture screenshot and UI hierarchy, check that the login button is visible and tappable, and report if the implementation is working correctly."

在实现影响UI的功能后，调用此技能验证该功能在设备上是否正常工作。

Session Lifecycle

会话生命周期

DeviceInteractionStartSession (do this early, runs in the background)
  → DeviceInteractionInstallAndRun (after each code change; includes building)
    → DeviceEventSynthesize (interact + observe, repeatable)
  → DeviceInteractionEndSession (when done — keeping sessions open is resource-heavy)

DeviceInteractionStartSession（尽早执行，在后台运行）
  → DeviceInteractionInstallAndRun（每次代码变更后执行；包含构建步骤）
    → DeviceEventSynthesize（交互+观察，可重复执行）
  → DeviceInteractionEndSession（完成后执行——保持会话开启会占用大量资源）

DeviceInteractionStartSession tool

DeviceInteractionStartSession工具

Device Discovery

设备发现

When opening a new device interaction session, pass a device identifier to select a device, or omit it to use the current destination. Pass any non-matching value to get a list of available targets.

开启新的设备交互会话时，可传入设备标识符来选择设备，或省略该参数以使用当前目标设备。传入不匹配的值可获取可用目标设备列表。

DeviceInteractionInstallAndRun tool

DeviceInteractionInstallAndRun工具

Optional Parameters

可选参数

```
commandLineArguments
```
— arguments passed to the app at launch. Use
```
$(inherited)
```
as a token to preserve the scheme's existing arguments (e.g.
```
["$(inherited)", "--reset-state"]
```
to add an extra argument at the end).
```
environmentVariables
```
— key/value pairs set in the app's environment at launch. Use
```
"$(inherited)"
```
as a key to preserve the scheme's existing environment variables (e.g.
```
{"$(inherited)": "", "DEBUG_MODE": "1"}
```
).

Omit both parameters to leave the scheme's arguments and environment unchanged.

Prefer these parameters over editing the scheme directly. They are applied only for that one run and have no lasting effect on the user's configuration.

```
commandLineArguments
```
—— 应用启动时传入的命令行参数。使用
```
$(inherited)
```
作为令牌保留scheme的现有参数（例如：
```
["$(inherited)", "--reset-state"]
```
在末尾添加额外参数）。
```
environmentVariables
```
—— 应用启动时设置的环境变量键值对。使用
```
"$(inherited)"
```
作为键保留scheme的现有环境变量（例如：
```
{"$(inherited)": "", "DEBUG_MODE": "1"}
```
）。

省略这两个参数将保持scheme的参数和环境变量不变。

优先使用这些参数，而非直接编辑scheme。 它们仅对单次运行生效，不会对用户配置产生持久影响。

For the Subagent

子Agent说明

ALWAYS report UI issues that might be caused by code: overlapping or unreadable text, unexpectedly cropped image/text, wrong colors etc.

务必报告可能由代码导致的UI问题：文本重叠或无法读取、图片/文本意外被裁剪、颜色错误等。

DeviceEventSynthesize tool

DeviceEventSynthesize工具

This tool allows performing an interaction and observing the state of a device.

此工具可执行交互操作并观察设备状态。

Reading Hierarchy Files

读取层级文件

The hierarchy files include calculated center positions for each element:

UIView {{100, 200}, {50, 30}}, center: {125.0, 215.0}
  UIButton "Login" {{110, 205}, {30, 20}}, center: {125.0, 215.0}

```
{100, 200}
```
- origin position
```
{50, 30}
```
- width and height
```
center: {125.0, 215.0}
```
- calculated center point (best for tapping)

Always prefer the center coordinates for touch events.

Warning: Elements marked

isRemoteLeafPlaceholder

do not report child elements — interacting with them requires falling back to screenshot-estimated coordinates.

层级文件包含每个元素的计算中心位置：

UIView {{100, 200}, {50, 30}}, center: {125.0, 215.0}
  UIButton "Login" {{110, 205}, {30, 20}}, center: {125.0, 215.0}

```
{100, 200}
```
- 原点位置
```
{50, 30}
```
- 宽高
```
center: {125.0, 215.0}
```
- 计算出的中心点（最适合用于点击操作）

触摸事件始终优先使用中心坐标。

注意：标记为

isRemoteLeafPlaceholder

的元素不会报告子元素——与它们交互时需退回到通过截图估算的坐标。

Interaction Command Syntax

交互命令语法

The

interactionCommand

parameter accepts a command syntax:

Command	Description
`t <x> <y> [duration]`	Tap at coordinates with optional hold duration
`d <x> <y>`	Double tap
`t <x1> <y1> f <x2> <y2> [duration]`	Swipe from (x1,y1) to (x2,y2)
`b h/p/u/d [duration]`	Hardware button: h=Home, p=Power, u=VolUp, d=VolDown
`sender keyboard kbd <text>`	Type text; must be the last command in the chain — all content after `kbd` is taken verbatim (multiple spaces preserved). For special characters use `\u{XXXX}` Unicode escapes: `\u{000A}` (return/newline), `\u{0009}` (tab)
`w duration`	Wait for a duration without any work
`orientation faceDown/faceUp/landscapeLeft/landscapeRight/portrait/portraitUpsideDown`	Set device orientation

Examples:

```
"t 100 200"
```
- Tap at (100, 200)
```
"d 200 300"
```
- Double tap at (200, 300)
```
"t 200 600 f 200 200 0.3"
```
- Swipe up (scroll to the content below)
```
"t 200 200 f 200 600 0.3"
```
- Swipe down (scroll to the content above)
```
"b h"
```
- Press home button
```
"b h b h"
```
- Press home button twice to go to the app switcher
```
"b h w 0 b h"
```
- Wake and unlock a device (non-passcode devices only)
```
"sender keyboard kbd hello world"
```
- Type text with spaces
```
"sender keyboard kbd hello   world"
```
- Type text preserving multiple spaces
```
"sender keyboard kbd submit\u{000A}"
```
- Type text then press Return/submit
```
"w 0.3"
```
- Wait for 0.3s
```
"orientation landscapeLeft"
```
- Rotate device to landscape

interactionCommand

参数接受以下命令语法：

命令	描述
`t <x> <y> [duration]`	点击指定坐标，可选择按住时长
`d <x> <y>`	双击指定坐标
`t <x1> <y1> f <x2> <y2> [duration]`	从(x1,y1)滑动到(x2,y2)
`b h/p/u/d [duration]`	硬件按钮：h=Home，p=Power，u=音量+，d=音量-
`sender keyboard kbd <text>`	输入文本；必须是命令链中的最后一个命令 —— `kbd` 之后的所有内容均按原样读取（保留多个空格）。特殊字符使用 `\u{XXXX}` Unicode转义： `\u{000A}` （回车/换行）、 `\u{0009}` （制表符）
`w duration`	等待指定时长，不执行任何操作
`orientation faceDown/faceUp/landscapeLeft/landscapeRight/portrait/portraitUpsideDown`	设置设备方向

示例：

```
"t 100 200"
```
- 点击(100, 200)
```
"d 200 300"
```
- 双击(200, 300)
```
"t 200 600 f 200 200 0.3"
```
- 向上滑动（滚动查看下方内容）
```
"t 200 200 f 200 600 0.3"
```
- 向下滑动（滚动查看上方内容）
```
"b h"
```
- 按下Home键
```
"b h b h"
```
- 连续按两次Home键进入应用切换器
```
"b h w 0 b h"
```
- 唤醒并解锁设备（仅适用于无密码设备）
```
"sender keyboard kbd hello world"
```
- 输入带空格的文本
```
"sender keyboard kbd hello   world"
```
- 输入文本并保留多个空格
```
"sender keyboard kbd submit\u{000A}"
```
- 输入文本后按回车/提交
```
"w 0.3"
```
- 等待0.3秒
```
"orientation landscapeLeft"
```
- 将设备旋转至横向左方向

Standard Subagent Workflow

标准子Agent工作流程

Before any interaction, always capture and read the hierarchy (and screenshot). After any interaction, capture again and verify the result. For complex components (like toggles or switches), look at nested elements (like

Switch

Slider

) — nearby elements might correspond to the actual control. When done, report findings to the main agent.

To capture without interacting, use DeviceEventSynthesize with an empty interactionCommand.
Never guess positions from screenshots alone — always use hierarchy center coordinates.
If not confident or thumbnail resolution is insufficient, analyze the full-size screenshot.

在任何交互操作前，务必捕获并读取层级结构（以及截图）。在任何交互操作后，再次捕获并验证结果。对于复杂组件（如开关或滑块），查看嵌套元素（如

Switch

或

Slider

）——附近的元素可能对应实际控件。完成后，将结果报告给主Agent。

若仅需捕获不进行交互，使用空的interactionCommand调用DeviceEventSynthesize。
切勿仅通过截图猜测位置——始终使用层级结构的中心坐标。
若不确定或缩略图分辨率不足，分析全尺寸截图。

Timing and Retries

时序与重试

App launch: After starting a session, the app may take a few seconds to load. Capture the hierarchy and check it has meaningful UI elements before interacting. If the hierarchy is mostly empty or shows a launch screen, capture again before proceeding.
After interaction: If a tap or swipe doesn't produce the expected change, recapture the hierarchy and retry the interaction once (the element may have shifted during an animation). If it still fails after one retry, report the failure rather than retrying indefinitely.
Loading states: If the hierarchy shows a spinner or loading indicator, capture again after a brief pause. Do not interact with elements that are still loading.

应用启动：启动会话后，应用可能需要几秒加载。在交互前捕获层级结构，检查是否包含有意义的UI元素。如果层级结构大多为空或显示启动画面，在继续操作前重新捕获。
交互后：如果点击或滑动未产生预期变化，重新捕获层级结构并重试一次交互（元素可能在动画过程中移位）。如果重试一次后仍失败，报告失败而非无限重试。
加载状态：如果层级结构显示加载 spinner 或加载指示器，短暂暂停后重新捕获。不要与仍在加载的元素交互。

Judging Success vs Failure

判定成功与失败

When verifying, distinguish between these categories:

Functional bug (always report): element doesn't respond to tap, navigation goes to wrong screen, crash, data not displayed, missing expected UI element.
Visual/layout bug (always report): overlapping text, truncated labels, elements rendered off-screen, wrong colors, broken alignment.
Transient state (do NOT report as bug): loading spinners, brief animations, keyboard appearing/dismissing. Capture again after the transition completes.
Unexpected exits (always report): crashes, application exits. To identify, track process id and capture process's standard output.
Expected behavior (do NOT report as bug): empty states with placeholder text, disabled buttons when form is incomplete, permission dialogs.

验证时，区分以下类别：

功能性bug（务必报告）：元素对点击无响应、导航至错误页面、崩溃、数据未显示、缺少预期UI元素。
视觉/布局bug（务必报告）：文本重叠、标签截断、元素渲染到屏幕外、颜色错误、对齐错乱。
临时状态（勿作为bug报告）：加载spinner、短暂动画、键盘弹出/收起。过渡完成后重新捕获。
意外退出（务必报告）：崩溃、应用退出。可通过跟踪进程ID并捕获进程标准输出来识别。
预期行为（勿作为bug报告）：带占位文本的空状态、表单未完成时按钮禁用、权限弹窗。

Error Handling

错误处理

If application is not visible, retry once, as this might be caused by a slow device.
If tap target unclear, re-read hierarchy data for correct center coordinates.
You can inspect runtime logs to troubleshoot. If you suspect timing bugs, suggest to the main agent that temporarily adding
```
print
```
statements in the relevant code may help diagnose the issue.
Report issues back to the main agent with details and suggestions.

如果应用不可见，重试一次，这可能是设备运行缓慢导致的。
如果点击目标不明确，重新读取层级结构数据获取正确的中心坐标。
可检查运行时日志进行故障排查。如果怀疑是时序bug，建议主Agent在相关代码中临时添加
```
print
```
语句以帮助诊断问题。
将问题详情和建议反馈给主Agent。