open-computer-use

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Open Computer Use

Overview

概述

Open Computer Use exposes Computer Use as a local CLI and stdio MCP server. It is not Codex.app-specific; adapt the commands and MCP config to the agent runtime you are operating in.

The macOS runtime requires macOS 14.0 or later. Windows and Linux use their own platform runtimes and are not subject to this macOS minimum.

It supports the same core tool surface across macOS, Linux, and Windows:

list_apps

get_app_state

click

perform_secondary_action

scroll

drag

type_text

press_key

, and

set_value

The macOS runtime additionally implements

select_text

, which selects the given text inside a text element or places the text cursor before / after it (

selection

text

cursor_before

cursor_after

). Pass

text

exactly as it appears in the accessibility tree, plus

prefix

suffix

when it is not unique; an ambiguous target fails closed instead of silently selecting the first match. JavaScript exposes

app.selectText(elementIndex, text, options?)

. Unreadable or mismatched range verification throws, including when full action read-back is off. Windows and Linux runtimes do not implement this tool yet.

macOS additionally exposes

query

: a targeted accessibility lookup that finds controls by text and/or role without rendering a tree or taking a screenshot.

Open Computer Use 将 Computer Use 作为本地 CLI 和 stdio MCP 服务器提供。它并非 Codex.app 专用；应根据当前操作的 Agent 运行时调整命令和 MCP 配置。

macOS 运行时要求 macOS 14.0 或更高版本。Windows 和 Linux 使用各自的平台运行时，不受这一 macOS 最低版本要求的限制。

它在 macOS、Linux 和 Windows 上支持相同的核心工具集合：

list_apps

、

get_app_state

、

click

、

perform_secondary_action

、

scroll

、

drag

、

type_text

、

press_key

和

set_value

。

macOS 运行时还实现了

select_text

，它可以在文本元素内选择指定文本，或者将文本光标放置在其之前或之后（

selection

：

text

、

cursor_before

、

cursor_after

）。请按 accessibility tree 中的原样传入

text

；如果文本不是唯一的，还应传入

prefix

或

suffix

。目标不明确时会安全失败（fail closed），而不是静默选择第一个匹配项。JavaScript 暴露了

app.selectText(elementIndex, text, options?)

。范围验证无法读取或不匹配时会抛出异常，即使完整操作回读已关闭也是如此。Windows 和 Linux 运行时目前尚未实现此工具。

macOS 还提供

query

：一种有针对性的 accessibility 查找，无需渲染树或截取屏幕截图，即可按文本和/或角色查找控件。

Core Workflow

核心工作流

On macOS, run
```
sw_vers -productVersion
```
before invoking the CLI and require macOS 14.0 or later. On older versions, explain that the runtime cannot launch; do not recommend
```
doctor
```
or permission changes as a fix for binary incompatibility.
Check the CLI is installed with
```
open-computer-use -h
```
or
```
ocu -h
```
. If installation or setup is missing, read references/installation.md.
On supported macOS versions, run
```
open-computer-use doctor
```
before the first real GUI task. If permissions are missing, ask the user to approve Accessibility and Screen Recording in the onboarding UI.
Inspect available apps before acting:
```
open-computer-use call list_apps
```
.
Capture current UI state with
```
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
```
. The default state is usually enough for UI operation.
When the task needs longer semantic text, such as chat history, email bodies, document text, or long form content, call
```
get_app_state
```
with
```
text_limit: 1000
```
or
```
text_limit: "max"
```
.
When visible long pages or lists appear incomplete even after scrolling, call
```
get_app_state
```
with a larger
```
max_tree_nodes
```
or
```
max_tree_depth
```
.
Prefer element-targeted actions using
```
element_index
```
from the latest
```
get_app_state
```
result.
For multi-step CLI work, use
```
open-computer-use call --calls '<json-array>'
```
so one process can reuse the latest element index mapping.
For agent runtimes that support local MCP servers, configure
```
open-computer-use mcp
```
or
```
ocu mcp
```
and call the exposed Computer Use tools directly. Read references/usage.md.
For direct code-first orchestration, run
```
ocu capabilities
```
and then use one-shot
```
ocu js
```
or persistent
```
ocu repl
```
. Read references/usage.md.
If communication, permission, or desktop-session access fails, read references/troubleshooting.md.

在 macOS 上，调用 CLI 前运行
```
sw_vers -productVersion
```
，并要求 macOS 14.0 或更高版本。对于较旧的版本，应说明运行时无法启动；不要将
```
doctor
```
或权限更改作为二进制文件不兼容问题的解决方案。
使用
```
open-computer-use -h
```
或
```
ocu -h
```
检查 CLI 是否已安装。如果尚未安装或完成设置，请阅读 references/installation.md。
在受支持的 macOS 版本上，首次执行实际 GUI 任务前运行
```
open-computer-use doctor
```
。如果缺少权限，请让用户在 onboarding UI 中批准 Accessibility 和 Screen Recording。
执行操作前检查可用应用：
```
open-computer-use call list_apps
```
。
使用
```
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
```
捕获当前 UI 状态。默认状态通常足以执行 UI 操作。
当任务需要更长的语义文本，例如聊天记录、邮件正文、文档文本或长篇内容时，调用
```
get_app_state
```
并设置
```
text_limit: 1000
```
或
```
text_limit: "max"
```
。
如果即使滚动后，可见的长页面或列表仍显示不完整，请调用
```
get_app_state
```
，并增大
```
max_tree_nodes
```
或
```
max_tree_depth
```
。
优先使用基于元素定位的操作，并采用最新
```
get_app_state
```
结果中的
```
element_index
```
。
对于多步骤 CLI 工作，使用
```
open-computer-use call --calls '<json-array>'
```
，使一个进程能够复用最新的元素索引映射。
对于支持本地 MCP 服务器的 Agent 运行时，配置
```
open-computer-use mcp
```
或
```
ocu mcp
```
，然后直接调用已暴露的 Computer Use 工具。阅读 references/usage.md。
对于以代码优先的编排方式，运行
```
ocu capabilities
```
，然后使用一次性
```
ocu js
```
或持久化
```
ocu repl
```
。阅读 references/usage.md。
如果通信、权限或桌面会话访问失败，请阅读 references/troubleshooting.md。

Operating Rules

操作规则

Treat the target desktop as the user's real session. Do not inspect password managers, unrelated private content, or sensitive apps unless the user explicitly asked for that task.
Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes.
Do not assume Codex.app plugin helpers are available. Use the installed
```
open-computer-use
```
/
```
ocu
```
CLI or an explicit MCP config.
Use indexes returned by
```
get_app_state
```
or macOS
```
query
```
in the same native session; do not guess indexes across sessions or after large UI changes.
Prefer semantic actions and
```
set_value
```
for editable controls. Use coordinate
```
click
```
,
```
scroll
```
, and
```
drag
```
only when the element tree does not expose a safer target.
On macOS, do not enable
```
OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1
```
unless the user explicitly requested
```
click_method: "global"
```
, a
```
drag
```
that must drive a window-server drag session (window move, drag-select text, Finder drag-and-drop), or other diagnostic behavior that may move the real pointer. Without it
```
drag
```
reports
```
Drag delivered via app_post
```
and those operations have no effect; see
```
references/usage.md
```
for alternatives.
On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.

将目标桌面视为用户的真实会话。除非用户明确要求执行相关任务，否则不要检查密码管理器、无关的私密内容或敏感应用。
在发送、删除、购买、批准、上传或进行其他外部可见更改之前，必须先询问用户。
不要假设 Codex.app 插件辅助工具可用。应使用已安装的
```
open-computer-use
```
/
```
ocu
```
CLI，或显式配置 MCP。
在同一个原生会话中使用
```
get_app_state
```
或 macOS
```
query
```
返回的索引；不要跨会话猜测索引，也不要在 UI 发生大幅变化后猜测索引。
对可编辑控件，优先使用语义操作和
```
set_value
```
。仅当元素树未提供更安全的操作目标时，才使用基于坐标的
```
click
```
、
```
scroll
```
和
```
drag
```
。
在 macOS 上，除非用户明确要求
```
click_method: "global"
```
、必须驱动 window-server drag session 的
```
drag
```
（移动窗口、拖选文本、Finder 拖放），或其他可能移动真实指针的诊断行为，否则不要启用
```
OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1
```
。未启用时，
```
drag
```
会报告
```
Drag delivered via app_post
```
，且这些操作不会产生效果；替代方案见
```
references/usage.md
```
。
在 Windows 和 Linux 上，假设 GUI 自动化可用之前，请确认命令正在已登录的桌面会话中运行。

Common CLI Actions

常见 CLI 操作

open-computer-use -h
ocu -h
ocu capabilities --json
ocu js 'nodeRepl.write(6 * 7)'
ocu repl
open-computer-use doctor
open-computer-use call list_apps
ocu call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":1000}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":"max"}'
open-computer-use call get_app_state --args '{"app":"Google Chrome","max_tree_nodes":3000,"max_tree_depth":96}'
open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}'
open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'

For a short sequence that reuses state in one process:

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

open-computer-use -h
ocu -h
ocu capabilities --json
ocu js 'nodeRepl.write(6 * 7)'
ocu repl
open-computer-use doctor
open-computer-use call list_apps
ocu call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":1000}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":"max"}'
open-computer-use call get_app_state --args '{"app":"Google Chrome","max_tree_nodes":3000,"max_tree_depth":96}'
open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}'
open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'

以下是在一个进程内复用状态的简短操作序列：

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

MCP Usage

MCP 使用

For runtimes that can launch local MCP servers over stdio, use:

toml

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

Read references/usage.md for JSON config examples, direct tool-call patterns, and platform notes.

对于能够通过 stdio 启动本地 MCP 服务器的运行时，请使用：

toml

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

有关 JSON 配置示例、直接工具调用模式和平台说明，请阅读 references/usage.md。

References

参考资料

references/installation.md: one-time CLI install, agent MCP install commands, and macOS permissions.
references/usage.md: MCP config, direct CLI calls, sequencing, and platform behavior.
references/troubleshooting.md: permission, desktop-session, app discovery, and action failures.

references/installation.md：一次性 CLI 安装、Agent MCP 安装命令和 macOS 权限设置。
references/usage.md：MCP 配置、直接 CLI 调用、操作时序和平台行为。
references/troubleshooting.md：权限、桌面会话、应用发现和操作失败问题。