codex-plusplus-ios-simulator-tweak

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Codex++ iOS Simulator Tweak

Codex++ iOS Simulator 插件

Skill by ara.so — Codex Skills collection.

A Codex++ tweak that adds an iOS Simulator tab to Codex's right panel. It mirrors a booted iOS simulator without opening

Simulator.app

, forwards touch and keyboard input, and lets you annotate simulator UI elements directly into Codex comments.

由ara.so开发的Skill——Codex Skills合集成员。

这是一款Codex++插件，可为Codex右侧面板添加iOS Simulator标签页。它无需打开

Simulator.app

即可镜像已启动的iOS模拟器，转发触摸和键盘输入，还能直接在Codex注释中标注模拟器UI元素。

What It Does

功能特性

Embeds a live iOS simulator view in Codex's right panel
Mirrors simulator output through CoreSimulator IOSurface (headless)
Forwards touch, drag, swipe, keyboard, and hardware button input
Provides UI element annotations using the accessibility tree
Device picker to switch between installed simulators
Auto-boot capability when no simulator is running
Screenshot and hardware control (Home, Lock, Side Button, Siri)

在Codex右侧面板中嵌入实时iOS模拟器视图
通过CoreSimulator IOSurface实现无头模式下的模拟器输出镜像
支持转发触摸、拖拽、滑动、键盘及硬件按钮输入
借助无障碍树提供UI元素标注功能
设备选择器，可在已安装的模拟器间切换
无模拟器运行时自动启动功能
截图及硬件控制（Home键、锁屏键、侧边按钮、Siri按钮）

Installation

安装步骤

Prerequisites

前置条件

macOS with full Xcode installed (not just Command Line Tools)
At least one iOS simulator runtime/device downloaded
Xcode command-line tools properly configured:

bash

sudo xcode-select -s /Applications/Xcode.app

Verify Xcode path:

bash

xcode-select -p

安装完整Xcode的macOS系统（仅安装命令行工具不行）
至少下载一个iOS模拟器运行时/设备
正确配置Xcode命令行工具：

bash

sudo xcode-select -s /Applications/Xcode.app

验证Xcode路径：

bash

xcode-select -p

Should output: /Applications/Xcode.app/Contents/Developer

应输出：/Applications/Xcode.app/Contents/Developer

undefined

undefined

Install Codex++

安装Codex++

First install Codex++, the Codex extension framework.

首先安装Codex++，这是Codex的扩展框架。

Install the Tweak

安装插件

Clone or download this tweak into the Codex++ tweaks directory:

bash

mkdir -p ~/Library/Application\ Support/codex-plusplus/tweaks/
cd ~/Library/Application\ Support/codex-plusplus/tweaks/
git clone https://github.com/b-nnett/codex-plusplus-ios-simulator.git ios-simulator

Or manually place the folder:

text

~/Library/Application Support/codex-plusplus/tweaks/ios-simulator/

The tweak runs a preflight check on first launch and shows fix hints if dependencies are missing.

将此插件克隆或下载到Codex++插件目录：

bash

mkdir -p ~/Library/Application\ Support/codex-plusplus/tweaks/
cd ~/Library/Application\ Support/codex-plusplus/tweaks/
git clone https://github.com/b-nnett/codex-plusplus-ios-simulator.git ios-simulator

或手动将文件夹放置到以下路径：

text

~/Library/Application Support/codex-plusplus/tweaks/ios-simulator/

插件首次启动时会运行预检查，若依赖缺失会显示修复提示。

Usage

使用方法

Opening the Simulator Panel

打开模拟器面板

Via UI:

Click the
```
+
```
menu in Codex's right panel
Select iOS Simulator (appears below the divider)

Via Keyboard:

Press
```
Cmd+Y
```

通过UI操作：

点击Codex右侧面板的
```
+
```
菜单
选择iOS Simulator（显示在分隔线下方）

通过键盘快捷键：

按下
```
Cmd+Y
```

Controls

操作控制

Once the panel is open:

Tap/Click: Click anywhere on the simulator screen
Drag/Swipe: Click and drag
Keyboard: Type directly (focus is captured)
Hardware Buttons:
- Home button
- Lock button
- Side button
- Siri button
Screenshot: Capture current simulator state
Device Picker: Dropdown to switch simulators
Auto-boot: Toggle to automatically boot simulator when none is running

面板打开后：

点击/轻触：点击模拟器屏幕任意位置
拖拽/滑动：点击并拖动
键盘：直接输入（焦点已捕获）
硬件按钮:
- Home键
- 锁屏键
- 侧边按钮
- Siri按钮
截图：捕获当前模拟器状态
设备选择器：下拉菜单切换模拟器
自动启动：切换开关，无模拟器运行时自动启动

Annotations

标注功能

Annotation mode allows you to reference specific UI elements in Codex comments:

Click the annotation button in the simulator panel
Click on a UI element in the simulator
Write your comment in Codex's native comment UI

The tweak automatically includes:

Element label and accessibility role
Simulator device ID
Element frame coordinates
Marker point
Viewport size

Example annotation payload:

javascript

{
  "element": {
    "label": "Sign In",
    "role": "Button",
    "frame": { "x": 120, "y": 450, "width": 175, "height": 44 }
  },
  "simulator": "iPhone 15 Pro",
  "marker": { "x": 207.5, "y": 472 },
  "viewport": { "width": 393, "height": 852 }
}

Use cases:

"Fix this button layout" → points to specific button
"Why is this label truncated?" → includes label frame and text
"Adjust spacing here" → marks exact coordinates

标注模式可让你在Codex注释中引用特定UI元素：

点击模拟器面板中的标注按钮
点击模拟器中的某个UI元素
在Codex原生注释界面中撰写你的评论

插件会自动包含以下信息：

元素标签及无障碍角色
模拟器设备ID
元素框架坐标
标记点
视口尺寸

标注示例数据：

javascript

{
  "element": {
    "label": "Sign In",
    "role": "Button",
    "frame": { "x": 120, "y": 450, "width": 175, "height": 44 }
  },
  "simulator": "iPhone 15 Pro",
  "marker": { "x": 207.5, "y": 472 },
  "viewport": { "width": 393, "height": 852 }
}

使用场景：

"修复此按钮布局" → 指向特定按钮
"为什么这个标签被截断了？" → 包含标签框架和文本
"调整此处间距" → 标记精确坐标

File Structure

文件结构

text

ios-simulator/
├── index.js                    # Main Codex++ tweak entry point
├── helpers/
│   ├── sim-capture.swift       # Headless frame capture helper
│   └── sim-input.m             # Touch, keyboard, hardware button helper
├── manifest.json               # Tweak metadata
└── README.md

text

ios-simulator/
├── index.js                    # Codex++插件主入口文件
├── helpers/
│   ├── sim-capture.swift       # 无头帧捕获辅助工具
│   └── sim-input.m             # 触摸、键盘、硬件按钮辅助工具
├── manifest.json               # 插件元数据
└── README.md

Key Code Patterns

核心代码模式

Tweak Entry Point (index.js)

插件入口文件（index.js）

The main tweak file exports a Codex++ tweak object:

javascript

module.exports = {
  name: 'iOS Simulator',
  icon: 'phone.fill',
  
  // Called when panel is opened
  onActivate(panel) {
    panel.setTitle('iOS Simulator');
    initializeSimulator(panel);
  },
  
  // Called when panel is closed
  onDeactivate() {
    stopHelpers();
    cleanupResources();
  },
  
  // Panel UI setup
  renderPanel(container) {
    const canvas = document.createElement('canvas');
    const controls = createControlBar();
    container.append(canvas, controls);
    return { canvas, controls };
  }
};

主插件文件导出一个Codex++插件对象：

javascript

module.exports = {
  name: 'iOS Simulator',
  icon: 'phone.fill',
  
  // 面板打开时调用
  onActivate(panel) {
    panel.setTitle('iOS Simulator');
    initializeSimulator(panel);
  },
  
  // 面板关闭时调用
  onDeactivate() {
    stopHelpers();
    cleanupResources();
  },
  
  // 面板UI设置
  renderPanel(container) {
    const canvas = document.createElement('canvas');
    const controls = createControlBar();
    container.append(canvas, controls);
    return { canvas, controls };
  }
};

Capturing Simulator Frames (sim-capture.swift)

捕获模拟器帧（sim-capture.swift）

The Swift helper uses CoreSimulator to capture IOSurface frames:

swift

import Foundation
import CoreSimulator
import IOSurface

// Connect to simulator framebuffer
let device = SimDevice(udid: deviceUDID)
let surface = device.surface // IOSurface

// Capture loop
while isRunning {
    let surfaceRef = device.io.surface
    let baseAddress = IOSurfaceGetBaseAddress(surfaceRef)
    let width = IOSurfaceGetWidth(surfaceRef)
    let height = IOSurfaceGetHeight(surfaceRef)
    
    // Write raw frame to stdout
    fwrite(baseAddress, 1, width * height * 4, stdout)
    fflush(stdout)
    
    usleep(16667) // ~60fps
}

Swift辅助工具使用CoreSimulator捕获IOSurface帧：

swift

import Foundation
import CoreSimulator
import IOSurface

// 连接到模拟器帧缓冲
let device = SimDevice(udid: deviceUDID)
let surface = device.surface // IOSurface

// 捕获循环
while isRunning {
    let surfaceRef = device.io.surface
    let baseAddress = IOSurfaceGetBaseAddress(surfaceRef)
    let width = IOSurfaceGetWidth(surfaceRef)
    let height = IOSurfaceGetHeight(surfaceRef)
    
    // 将原始帧写入标准输出
    fwrite(baseAddress, 1, width * height * 4, stdout)
    fflush(stdout)
    
    usleep(16667) // ~60fps
}

Sending Input Events (sim-input.m)

发送输入事件（sim-input.m）

The Objective-C helper sends touch and keyboard events:

objc

#import <Foundation/Foundation.h>
#import <SimulatorKit/SimulatorKit.h>

// Send touch event
- (void)sendTouchAtX:(CGFloat)x y:(CGFloat)y phase:(NSString*)phase {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    SimDeviceIOClient *io = device.io;
    
    SimDeviceIOTouchEvent *event = [SimDeviceIOTouchEvent new];
    event.x = x;
    event.y = y;
    event.phase = [self phaseFromString:phase]; // began, moved, ended
    
    [io sendTouchEvent:event];
}

// Send keyboard input
- (void)sendKeyPress:(NSString*)key {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    [device.io sendKeyboardEvent:key];
}

// Hardware buttons
- (void)pressHomeButton {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    [device.io pressButton:SimDeviceIOButtonHome];
}

Objective-C辅助工具发送触摸和键盘事件：

objc

#import <Foundation/Foundation.h>
#import <SimulatorKit/SimulatorKit.h>

// 发送触摸事件
- (void)sendTouchAtX:(CGFloat)x y:(CGFloat)y phase:(NSString*)phase {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    SimDeviceIOClient *io = device.io;
    
    SimDeviceIOTouchEvent *event = [SimDeviceIOTouchEvent new];
    event.x = x;
    event.y = y;
    event.phase = [self phaseFromString:phase]; // began, moved, ended
    
    [io sendTouchEvent:event];
}

// 发送键盘输入
- (void)sendKeyPress:(NSString*)key {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    [device.io sendKeyboardEvent:key];
}

// 硬件按钮
- (void)pressHomeButton {
    SimDevice *device = [self deviceWithUDID:deviceUDID];
    [device.io pressButton:SimDeviceIOButtonHome];
}

Device Management

设备管理

List available simulators:

javascript

const { execSync } = require('child_process');

function getAvailableSimulators() {
  const output = execSync('xcrun simctl list devices --json', { encoding: 'utf8' });
  const data = JSON.parse(output);
  
  const devices = [];
  for (const runtime in data.devices) {
    for (const device of data.devices[runtime]) {
      if (device.isAvailable) {
        devices.push({
          udid: device.udid,
          name: device.name,
          state: device.state,
          runtime: runtime
        });
      }
    }
  }
  return devices;
}

Boot a simulator:

javascript

function bootSimulator(udid) {
  execSync(`xcrun simctl boot ${udid}`, { encoding: 'utf8' });
}

Shutdown:

javascript

function shutdownSimulator(udid) {
  execSync(`xcrun simctl shutdown ${udid}`, { encoding: 'utf8' });
}

列出可用模拟器：

javascript

const { execSync } = require('child_process');

function getAvailableSimulators() {
  const output = execSync('xcrun simctl list devices --json', { encoding: 'utf8' });
  const data = JSON.parse(output);
  
  const devices = [];
  for (const runtime in data.devices) {
    for (const device of data.devices[runtime]) {
      if (device.isAvailable) {
        devices.push({
          udid: device.udid,
          name: device.name,
          state: device.state,
          runtime: runtime
        });
      }
    }
  }
  return devices;
}

启动模拟器：

javascript

function bootSimulator(udid) {
  execSync(`xcrun simctl boot ${udid}`, { encoding: 'utf8' });
}

关闭模拟器：

javascript

function shutdownSimulator(udid) {
  execSync(`xcrun simctl shutdown ${udid}`, { encoding: 'utf8' });
}

Configuration

配置说明

The tweak compiles helper binaries on first use and caches them:

text

~/Library/Caches/co.bennett.ios-simulator/
├── sim-capture          # Compiled Swift helper
└── sim-input            # Compiled Objective-C helper

No network requests are made. All compilation happens locally.

插件首次使用时会编译辅助二进制文件并缓存：

text

~/Library/Caches/co.bennett.ios-simulator/
├── sim-capture          # 编译后的Swift辅助工具
└── sim-input            # 编译后的Objective-C辅助工具

无需网络请求，所有编译均在本地完成。

manifest.json

json

{
  "name": "iOS Simulator",
  "version": "1.0.0",
  "description": "Headless iOS Simulator for Codex++",
  "entry": "index.js",
  "permissions": [
    "process",
    "filesystem"
  ],
  "shortcuts": [
    {
      "key": "cmd+y",
      "action": "toggleSimulator"
    }
  ]
}

json

{
  "name": "iOS Simulator",
  "version": "1.0.0",
  "description": "Headless iOS Simulator for Codex++",
  "entry": "index.js",
  "permissions": [
    "process",
    "filesystem"
  ],
  "shortcuts": [
    {
      "key": "cmd+y",
      "action": "toggleSimulator"
    }
  ]
}

Troubleshooting

故障排查

Preflight Check Failed

预检查失败

Error: "Xcode not found"

bash

undefined

错误： "Xcode not found"

bash

undefined

Ensure Xcode is installed

确保已安装Xcode

xcode-select -p

If pointing to Command Line Tools, fix with:

如果指向命令行工具，执行以下命令修复：

sudo xcode-select -s /Applications/Xcode.app


**Error:** "No simulators available"

```bash

sudo xcode-select -s /Applications/Xcode.app


**错误：** "No simulators available"

```bash

List available simulators

列出可用模拟器

xcrun simctl list devices

Download iOS runtimes in Xcode:

在Xcode中下载iOS运行时：

Xcode → Settings → Platforms → [Download iOS runtime]

Xcode → 设置 → 平台 → [下载iOS运行时]

undefined

undefined

Helper Compilation Fails

辅助工具编译失败

Error: "Swift compiler not found"

bash

undefined

错误： "Swift compiler not found"

bash

undefined

Verify swiftc is available

验证swiftc是否可用

which swiftc

Should be under Xcode.app/Contents/Developer/

路径应在Xcode.app/Contents/Developer/下

If not, run: sudo xcode-select -s /Applications/Xcode.app

若不在，执行：sudo xcode-select -s /Applications/Xcode.app


**Error:** "Framework not found: SimulatorKit"

```bash


**错误：** "Framework not found: SimulatorKit"

```bash

SimulatorKit is a private framework included with Xcode

SimulatorKit是Xcode附带的私有框架

Ensure full Xcode is installed, not just Command Line Tools

确保安装的是完整Xcode，而非仅命令行工具

Check framework exists:

检查框架是否存在：

ls /Applications/Xcode.app/Contents/Developer/Library/PrivateFrameworks/SimulatorKit.framework

undefined

ls /Applications/Xcode.app/Contents/Developer/Library/PrivateFrameworks/SimulatorKit.framework

undefined

Simulator Won't Boot

模拟器无法启动

Error: "Unable to boot device in current state: Booted"

The simulator is already running. Use the device picker to select it, or shut it down first:

bash

xcrun simctl shutdown <UDID>

Error: "Failed to boot device"

Check simulator logs:

bash

xcrun simctl spawn booted log stream --predicate 'subsystem == "com.apple.CoreSimulator"'

错误： "Unable to boot device in current state: Booted"

模拟器已在运行。使用设备选择器选择该模拟器，或先关闭它：

bash

xcrun simctl shutdown <UDID>

错误： "Failed to boot device"

查看模拟器日志：

bash

xcrun simctl spawn booted log stream --predicate 'subsystem == "com.apple.CoreSimulator"'

No Frame Output

无帧输出

Symptom: Black screen in Codex panel

Verify simulator is booted:
bash
```
xcrun simctl list devices | grep Booted
```
Check helper process is running:
bash
```
ps aux | grep sim-capture
```
Restart the panel (close and reopen with
```
Cmd+Y
```
)

症状： Codex面板显示黑屏

验证模拟器已启动：
bash
```
xcrun simctl list devices | grep Booted
```
检查辅助工具进程是否运行：
bash
```
ps aux | grep sim-capture
```
重启面板（关闭后用
```
Cmd+Y
```
重新打开）

Touch Input Not Working

触摸输入无效

Ensure simulator window is not open in
```
Simulator.app
```
(conflicting input)
Check
```
sim-input
```
helper is running:
bash
```
ps aux | grep sim-input
```
Verify device UDID matches between capture and input helpers

确保未在
```
Simulator.app
```
中打开模拟器窗口（输入冲突）
检查
```
sim-input
```
辅助工具是否运行：
bash
```
ps aux | grep sim-input
```
验证捕获和输入辅助工具使用的设备UDID是否一致

Annotations Return Empty Labels

标注返回空标签

Cause: App under test has poor accessibility labeling

Solution: Add accessibility labels to your UI elements:

swift

// SwiftUI
Text("Sign In")
    .accessibilityLabel("Sign In Button")
    .accessibilityIdentifier("signInButton")

// UIKit
button.accessibilityLabel = "Sign In Button"
button.accessibilityIdentifier = "signInButton"

原因： 被测应用的无障碍标签设置不完善

解决方案： 为UI元素添加无障碍标签：

swift

// SwiftUI
Text("Sign In")
    .accessibilityLabel("Sign In Button")
    .accessibilityIdentifier("signInButton")

// UIKit
button.accessibilityLabel = "Sign In Button"
button.accessibilityIdentifier = "signInButton"

Common Workflows

常见工作流

Testing a SwiftUI App

测试SwiftUI应用

Open simulator panel (
```
Cmd+Y
```
)
Select device from picker (e.g., "iPhone 15 Pro")

Launch your app from Xcode or command line:

bash

xcrun simctl launch <UDID> com.yourcompany.yourapp

Interact with UI in Codex panel
Annotate elements to document issues

打开模拟器面板（
```
Cmd+Y
```
）
从选择器中选择设备（如“iPhone 15 Pro”）

从Xcode或命令行启动你的应用：

bash

xcrun simctl launch <UDID> com.yourcompany.yourapp

在Codex面板中与UI交互
标注元素以记录问题

Debugging Layout Issues

调试布局问题

Enable annotation mode
Click on the element with incorrect layout
Write comment: "This button is clipped on iPhone SE"
The annotation includes frame and viewport size for the agent to analyze

启用标注模式
点击布局异常的元素
撰写评论：“此按钮在iPhone SE上被截断”
标注信息包含框架和视口尺寸，供Agent分析

Recording Interaction Sequences

记录交互序列

javascript

// Custom extension to record touch sequences
const touches = [];

panel.on('touch', (x, y, phase) => {
  touches.push({ x, y, phase, timestamp: Date.now() });
});

panel.on('annotate', () => {
  // Include touch sequence in annotation
  return { touches, viewport, element };
});

javascript

// 自定义扩展以记录触摸序列
const touches = [];

panel.on('touch', (x, y, phase) => {
  touches.push({ x, y, phase, timestamp: Date.now() });
});

panel.on('annotate', () => {
  // 在标注中包含触摸序列
  return { touches, viewport, element };
});

Privacy & Security

隐私与安全

No Screen Recording permission required (uses CoreSimulator IOSurface directly)
No network requests (all helpers compiled locally)
Helper processes stopped when tweak deactivates
Cached binaries stored in user cache directory only

无需屏幕录制权限（直接使用CoreSimulator IOSurface）
无网络请求（所有辅助工具均在本地编译）
插件停用后辅助进程自动停止
缓存二进制文件仅存储在用户缓存目录

License

许可证

MIT