jev-desktop
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chinesejev-desktop
jev-desktop
This skill requires a clone of the agent-desktop repository (the scripts live in and ship in no installed package) and a .
scripts/jev/TYPESAFE_API_KEYTwo entry points over the same screen reading. takes a goal and drives
until it is met. takes one step and hands the plan back to you. Neither
puts the tree in your context.
run.mjsact.mjs此 skill 需要克隆 agent-desktop 仓库(脚本位于 ,不会随任何已安装包一起分发),并设置 。
scripts/jev/TYPESAFE_API_KEY两个入口基于同一套屏幕读取机制。 接收一个目标并持续驱动,直至目标达成。 每次执行一个步骤,并将计划交还给你。二者都不会将树放入你的上下文。
run.mjsact.mjsThe loop
循环
sh
node scripts/jev/run.mjs --app Finder "open the Applications folder"
node scripts/jev/run.mjs --app TextEdit --cursor --text "A line to write." "write the supplied sentence into the document"Each turn reads the screen, asks for one operation and a target for that
operation in the same request, carries it out, and reads again. It prints one
JSON line per turn and one when it stops (shown trimmed).
json
{"turn":{"step":1,"operation":"CLICK","target":"treeitem \"Applications\"",
"confidence":1,"ok":true,"delivery":"delivered_verified","changed":true}}
{"stop":"done","confidence":0.96}The operations are , , , , ,
, , , , , and . Only the
ones something on screen can receive are offered: for a region that
reports it holds more, only after a . always scrolls
down. is for and for and .
CLICKTYPE_TEXTCHECKUNCHECKEXPANDCOLLAPSESCROLLDRILLWIDENWAITDONEBLOCKEDDRILLWIDENDRILLSCROLLdeliverywaitedWAITlookedDRILLWIDENThe target is asked once per operation. is chosen from the
elements that advertise , from those that advertise
, and so on. Only the head matching the chosen operation is read, so a
target can never be incompatible with the verb that acts on it.
click_targetClicktype_text_targetSetValueLooking is an operation. A window is read with first, so a dense
document costs the same first look as a small panel. A region that was cut off
reports how much it holds, and pins it as the root for later turns.
gives the whole window back.
--skeletonDRILLWIDENCHECKUNCHECKYou supply the text. Nothing here writes a value, so a run never puts a
string on screen that you did not choose. Pass once per value and they
are consumed in order, or pass a function as the option and it is asked
for each field with that field's description. When there is no value left to
give, is not offered at all, so the run stops rather than inventing
one.
--texttextTYPE_TEXTText goes in by whichever route the application accepts. A direct value
write () is one verified call. Only when the application refuses it
and the refusal says a retry is safe does the value go through the clipboard and
a paste (macOS only). The clipboard is put back when the run ends, or
emptied if it held anything but text. One key press per character is never used:
it drops characters and loses capitals.
set-valuecmd+vWhat leaves the machine. Every turn posts a description of the screen to the
decision endpoint (TypeSafe's by default, or whatever
points at): each element's role, its accessible name or description, up
to sixty characters of the value it holds, its state, the window title, and the
recent actions, excluding supplied text and detailed execution evidence. That is enough to send the contents of a private document or a
filled form to the configured remote service. Secure text fields are already
withheld by agent-desktop and never reach the request. Pass to
either script to withhold what every other field holds as well; targeting gets
harder, because a value is often the only thing that tells two unnamed rows
apart. The default is to include values. redacts explicit supplied
strings from action turns and later recent-action history, including when later
target names or execution evidence repeat them. Live screen descriptions remain
separate: if the application displays that text, its accessible name or field
value may still reach the remote service under the settings above.
preserves its exact executable in stdout, so that field can include a
value supplied with ; stdout may also contain accessible names and
element paths. Choose the endpoint and output destination with that exposure in
mind.
api.typesafe.aiTYPESAFE_BASE_URL--no-valuesrun.mjsact.mjsargv--textA step that is hard to undo is not taken quietly. An ordinary step needs 0.70
confidence in its target, one rated destructive (0.50 or more) needs 0.90, and
below 0.55 nothing runs. When the bar is not met the run stops and names the
candidate it would have acted on, so you decide instead of it. gets the
rating in its single request. rates risk with a second model call
(counted toward the 80), only for targets at 0.70 up to 0.90, so by default a
destructive step at target confidence 0.90 or higher runs unrated. If that call
returns no number, the step counts as destructive. Pass
to or to stop for confirmation on every step rated
destructive, regardless of confidence.
act.mjsrun.mjs--confirm-destructiverun.mjsact.mjs--cursorIt stops on , on , on a confidence too low for the risk, after 40
operations (, and count), after 80 model calls (risk
ratings count), or after three turns in a row that changed nothing ( turns
do not count). It also stops with when an operation fails, and
with when no element advertises an
action.
DONEBLOCKEDWAITDRILLWIDENWAITaction_failednothing on this screen can be acted onA screen with more actionable elements than a choice can carry says so, in the
request and in the turn it reports (), and the policy is told to look
inside a region rather than call the goal impossible.
truncated--root @refsh
node scripts/jev/run.mjs --app Finder "open the Applications folder"
node scripts/jev/run.mjs --app TextEdit --cursor --text "A line to write." "write the supplied sentence into the document"每一轮都会读取屏幕,在同一个请求中询问一项操作及该操作的目标,执行后再重新读取。每轮打印一行 JSON,停止时再打印一行(下面经过精简)。
json
{"turn":{"step":1,"operation":"CLICK","target":"treeitem \"Applications\"",
"confidence":1,"ok":true,"delivery":"delivered_verified","changed":true}}
{"stop":"done","confidence":0.96}可用操作包括 、、、、、、、、、、 和 。仅提供屏幕内容能够接收的操作:对于报告包含更多内容的区域可使用 ,而 仅可在 之后使用。 始终向下滚动。对于 , 为 ;对于 和 ,则为 。
CLICKTYPE_TEXTCHECKUNCHECKEXPANDCOLLAPSESCROLLDRILLWIDENWAITDONEBLOCKEDDRILLWIDENDRILLSCROLLWAITdeliverywaitedDRILLWIDENlooked每个操作只询问一次目标。 从声明支持 的元素中选择, 从声明支持 的元素中选择,依此类推。只读取与所选操作匹配的 head,因此目标绝不会与执行它的动词不兼容。
click_targetClicktype_text_targetSetValue查看本身是一项操作。 首先使用 读取窗口,因此密集文档的首次查看成本与小面板相同。被截断的区域会报告其包含的内容量,而 会将其固定为后续各轮的根节点。 会重新返回整个窗口。
--skeletonDRILLWIDEN使用 和 ,而不是 toggle。 两者都是幂等的:已经处于目标状态的复选框会保持不变。
CHECKUNCHECK由你提供文本。 这里不会写入任何值,因此一次运行绝不会在屏幕上写入你未选择的内容。每个值传递一次 ,它们会按顺序使用;也可以将函数作为 选项传入,此时会针对每个字段询问该字段的描述。当没有剩余可提供的值时,根本不会提供 ,因此运行会停止,而不是自行编造一个值。
--texttextTYPE_TEXT文本通过应用接受的方式输入。 直接值写入()是一次经过验证的调用。只有当应用拒绝该写入且拒绝信息明确表示重试安全时,文本才会通过剪贴板和 粘贴(仅限 macOS)输入。运行结束时会恢复剪贴板原内容;如果其中原本保存的不是文本,则会清空剪贴板。绝不会逐字符发送按键,因为这会丢失字符和大写字母。
set-valuecmd+v会离开本机的数据。 每一轮都会向决策端点(默认是 TypeSafe 的 ,或 指向的任何地址)发送屏幕描述:每个元素的 role、可访问名称或描述、最多六十个字符的值、状态、窗口标题以及近期操作,但不包含所提供的文本和详细执行证据。这些信息足以将私密文档内容或已填写表单发送至配置的远程服务。安全文本字段已由 agent-desktop 屏蔽,绝不会进入请求。可向任一脚本传入 ,以同时屏蔽所有其他字段持有的值;这会使定位变得更难,因为值往往是区分两个未命名行的唯一依据。默认包含值。 会从操作轮次及后续近期操作历史中隐去明确提供的字符串,即使后续目标名称或执行证据再次出现这些字符串也一样。实时屏幕描述仍单独存在:如果应用显示了相应文本,其可访问名称或字段值仍可能根据上述设置发送到远程服务。 会在 stdout 中保留其精确的可执行 ,因此该字段可能包含通过 提供的值;stdout 还可能包含可访问名称和元素路径。选择端点和输出目的地时,应将这种数据暴露风险纳入考虑。
api.typesafe.aiTYPESAFE_BASE_URL--no-valuesrun.mjsact.mjsargv--text难以撤销的步骤不会在无人察觉的情况下执行。 普通步骤要求目标置信度达到 0.70;被评定为破坏性(不低于 0.50)的步骤要求达到 0.90;低于 0.55 时不会执行任何操作。未达到门槛时,运行会停止并指出原本会操作的候选项,由你作出决定。 在其单次请求中获得该评级。 仅针对目标置信度处于 0.70 至 0.90 之间的目标,通过第二次模型调用评估风险(计入 80 次调用上限),因此默认情况下,目标置信度达到或超过 0.90 的破坏性步骤无需评级便执行。如果该调用未返回数值,则该步骤按破坏性处理。向 或 传入 ,可使每个被评定为破坏性的步骤都停止并等待确认,无论其置信度如何。
act.mjsrun.mjsrun.mjsact.mjs--confirm-destructive--cursor它会在以下情况停止:遇到 、遇到 、置信度低于风险要求、执行 40 次操作后(、 和 均计入)、发起 80 次模型调用后(风险评级计入),或连续三轮没有产生任何变化后( 轮次不计入)。操作失败时,它也会以 停止;当屏幕上没有任何元素声明支持操作时,则以 停止。
DONEBLOCKEDWAITDRILLWIDENWAITaction_failednothing on this screen can be acted on当屏幕中的可操作元素多于一次 choice 可承载的数量时,请求和所报告的轮次()中都会明确说明这一点,同时 policy 会收到指示:应查看某个区域内部,而不是直接判定目标无法实现。
truncated当你已知某个区域时,使用 可从该区域内部开始。
--root @refOne step at a time
一次处理一个步骤
You keep the goal, the plan and the memory. You send one sentence. You get back
one small object.
sh
node scripts/jev/act.mjs --app TextEdit --execute \
--text "Morning over the dock." "type this text into the main writing area"json
{
"ok": true,
"intent": "type this text into the main writing area",
"app": "TextEdit", "window": "Untitled 3", "window_id": "w-4521", "surface": "sheet",
"element": { "ref": "@s1:e139", "role": "textfield", "name": "Save As:",
"where": "sheet \"save\" > group" },
"command": "set-value",
"argv": ["set-value", "@s1:e139", "poem.txt"],
"decision": "act",
"why": null,
"confidence": { "target": 0.99, "command": 0.88 },
"gates": { "present": 0.94, "destructive": 0.31, "needs_text": 0.87 },
"runner_up": [{ "ref": "@s1:e137", "what": "textfield", "p": 0.02 }],
"notes": [],
"executed": { "ok": true, "delivery": "delivered_verified" }
}你保留目标、计划和记忆。你发送一句话,收到一个小对象。
sh
node scripts/jev/act.mjs --app TextEdit --execute \
--text "Morning over the dock." "type this text into the main writing area"json
{
"ok": true,
"intent": "type this text into the main writing area",
"app": "TextEdit", "window": "Untitled 3", "window_id": "w-4521", "surface": "sheet",
"element": { "ref": "@s1:e139", "role": "textfield", "name": "Save As:",
"where": "sheet \"save\" > group" },
"command": "set-value",
"argv": ["set-value", "@s1:e139", "poem.txt"],
"decision": "act",
"why": null,
"confidence": { "target": 0.99, "command": 0.88 },
"gates": { "present": 0.94, "destructive": 0.31, "needs_text": 0.87 },
"runner_up": [{ "ref": "@s1:e137", "what": "textfield", "p": 0.02 }],
"notes": [],
"executed": { "ok": true, "delivery": "delivered_verified" }
}Calling it
调用方式
| Flag | Meaning |
|---|---|
| Required. |
| Read one window of the app, by the id |
| Withhold accessible field values from the decision request. Supported by both scripts; the default includes values. |
| |
| Text the intent needs. Jev returns choices, never strings. |
| Resolve inside one container instead of the whole window. |
| |
Environment: is required (both scripts fail with
). and are covered
below. sets the paste read-back deadline in ms (default 2000, 0
to 60000).
TYPESAFE_API_KEYTYPESAFE_API_KEY unsetTYPESAFE_BASE_URLTYPESAFE_MODELJEV_VERIFY_MSargvdouble-clickhover--headedscrollPhrase the intent as an action, not as an element. resolves to . names no
operation and can resolve to . Describe the target the way a person would,
, not the way the tree
names it.
"type this text into the main writing area"type"the main writing area"focus"the field holding the name the file will be saved under"| Flag | 含义 |
|---|---|
| 必填。 |
| 按 ID 读取应用的某个窗口,该 ID 由 |
| 从决策请求中屏蔽可访问字段值。两个脚本都支持;默认包含值。 |
| 仅限 |
| 意图所需的文本。Jev 返回候选项,而不会返回字符串。 |
| 在一个容器内解析,而不是在整个窗口内解析。 |
| 仅限 |
环境:两个脚本都要求设置 (未设置时会因 而失败)。下文会介绍 和 。 设置粘贴回读截止时间,单位为 ms(默认 2000,范围 0 到 60000)。
TYPESAFE_API_KEYTYPESAFE_API_KEY unsetTYPESAFE_BASE_URLTYPESAFE_MODELJEV_VERIFY_MSargvdouble-clickhover--headedscroll将意图表述为操作,而不是元素。 会解析为 。 没有指定任何操作,因此可能解析为 。应像人一样描述目标,例如 ,而不是按照树的命名方式描述。
"type this text into the main writing area"type"the main writing area"focus"the field holding the name the file will be saved under"Reading decision
decision解读 decision
decision | What it means | What you do |
|---|---|---|
| One element clearly matches and the confidence clears the bar for this action's risk. | Nothing. With |
| The match is plausible but under the bar. | Ask the user, or re-phrase the intent and call again. |
| Jev answered | The screen is not where you think. Open the surface you need, then call again. |
| The command takes text and none was supplied. | Call again with |
whydecisionactnullrunner_up | 含义 | 你该做什么 |
|---|---|---|
| 一个元素明确匹配,且置信度达到该操作风险对应的门槛。 | 无需操作。使用 |
| 匹配有可能成立,但未达到门槛。 | 询问用户,或重新表述意图后再次调用。 |
| Jev 回答 | 当前屏幕并非你预期的位置。打开所需 surface 后再次调用。 |
| 命令需要文本,但未提供文本。 | 使用 |
whyactdecisionactnullrunner_upWhat the script decides, not Jev
由脚本而非 Jev 决定的内容
- Risk sets the bar. The answer says what; confidence says whether to act.
An ordinary action needs 0.70. One that Jev rates at 0.50 or more needs 0.90: writing a file, deleting, sending, confirming a warning. Below 0.55 nothing acts at all.
destructive - The command is reconciled against the element. Both questions are
answered in parallel and neither sees the other, so code checks the chosen
verb against the element's advertised actions. A readonly combobox gets
, never
click. Aset-valuepayload with a non-text verb corrects the verb, because the caller supplying text is evidence Jev does not have.--text - An open surface wins. When a sheet, alert, menu or popover is up, the
script reads the window, then that surface, and offers the surface's elements
instead of the window's. In the window tree those
elements carry and would all be dropped.
offscreen - Disabled, hidden and action-less elements are never offered, nor a cell inside a treeitem (the row beside it acts). Everything else is, including unnamed rows: a row is told apart by the value it holds, and a Choice does better with the full list than with a shortlist.
- 风险决定门槛。 答案说明执行什么;置信度决定是否执行。普通操作需要 0.70。被 Jev 评定为 且评分不低于 0.50 的操作需要 0.90,例如写入文件、删除、发送或确认警告。低于 0.55 时不会执行任何操作。
destructive - 命令会与元素进行核对。 两个问题会并行回答,且彼此不可见,因此代码会将选定的动词与元素声明支持的操作进行检查。只读 combobox 会得到 ,绝不会得到
click。如果set-value携带的文本与非文本动词不匹配,系统会修正该动词,因为调用方提供文本这一证据是 Jev 无法获得的。--text - 打开的 surface 优先。 当 sheet、alert、menu 或 popover 打开时,脚本会先读取窗口,再读取该 surface,并提供 surface 中的元素,而不是窗口中的元素。在窗口树中,这些元素会带有 标记,从而全部被丢弃。
offscreen - 禁用、隐藏及不支持操作的元素绝不会提供,treeitem 内部的 cell 也不会提供(由其旁边的 row 执行操作)。其他所有元素均可提供,包括未命名的行:行与行之间通过其持有的值进行区分,而 Choice 使用完整列表的效果优于使用候选项短列表。
How the request is shaped
请求的组织方式
act.mjs| Question | Type | Asks |
|---|---|---|
| choice | Which element the intent refers to, plus |
| choice | Which of the 16 interaction verbs it asks for. |
| noul | Is the thing on this screen at all? |
| noul | Would this be hard to undo? |
| noul | Does this need text from the caller? |
A Choice accepts 255 options, so up to 254 elements go in one pass (both
scripts). When the first pass lands under 0.70, the top five are
re-asked with richer descriptions. A screen with more than 254 elements says so
in ; use .
act.mjsnotes--root @refact.mjs| 问题 | 类型 | 询问内容 |
|---|---|---|
| choice | 意图指向哪个元素,以及 |
| choice | 意图要求执行 16 个交互动词中的哪一个。 |
| noul | 目标是否确实位于当前屏幕上。 |
| noul | 操作是否难以撤销。 |
| noul | 操作是否需要调用方提供文本。 |
一个 Choice 最多接受 255 个候选项,因此单次最多可传入 254 个元素(两个脚本均如此)。当 的首次执行结果低于 0.70 时,会使用更丰富的描述重新询问前五个候选项。元素超过 254 个的屏幕会在 中说明;此时应使用 。
act.mjsnotes--root @refDeciding through OpenRouter
通过 OpenRouter 作出决策
The decision endpoint is , defaulting to TypeSafe's native
. Configured endpoints must use HTTPS;
HTTP is allowed only for or a literal loopback IP, which keeps local
test routers usable. URL credentials are rejected, and redirects fail closed so
the bearer key cannot be forwarded to another endpoint. Point it at OpenRouter's Decisions
router to use an OpenRouter key as :
TYPESAFE_BASE_URLhttps://api.typesafe.ai/v1/systemonelocalhostTYPESAFE_API_KEYsh
TYPESAFE_BASE_URL=https://openrouter.ai/api/alpha/decisions \
TYPESAFE_MODEL=typesafe/jev-1.13 \
TYPESAFE_API_KEY=sk-or-... \
node scripts/jev/run.mjs --app Finder "open the Applications folder"TYPESAFE_MODELjev-latest~typesafe/<name>typesafe/jev-1.13typesafe/jev-latest400 Model does not exist决策端点由 指定,默认使用 TypeSafe 原生的 。配置的端点必须使用 HTTPS;仅允许对 或字面量回环 IP 使用 HTTP,以便本地测试路由器继续可用。URL 凭据会被拒绝,重定向也会采用 fail-closed 策略,从而防止 bearer key 被转发到其他端点。要将 OpenRouter key 作为 使用,可将端点指向 OpenRouter 的 Decisions router:
TYPESAFE_BASE_URLhttps://api.typesafe.ai/v1/systemonelocalhostTYPESAFE_API_KEYsh
TYPESAFE_BASE_URL=https://openrouter.ai/api/alpha/decisions \
TYPESAFE_MODEL=typesafe/jev-1.13 \
TYPESAFE_API_KEY=sk-or-... \
node scripts/jev/run.mjs --app Finder "open the Applications folder"TYPESAFE_MODELjev-latest~typesafe/<name>typesafe/jev-1.13typesafe/jev-latest400 Model does not existKnown limits
已知限制
- reads the screen once per call.
act.mjsreads it before and after each action, and skips the read afterrun.mjsorDRILL. Without a pinned region, each read takes a second snapshot when a sheet, alert, menu or popover is open. A read takes about 3 s on a dense app;WIDENis far cheaper.--root - does not retry: a 429 ends the call.
act.mjsretries 429, 503 and 529 twice, after 500 ms then 1000 ms, before it fails.run.mjs - Held input is unavailable, so no sustained key and no drag with hold. See
.
crates/core/src/commands/input_hold_policy.rs - is not resolved here. It needs no element, so send it directly.
press
- 每次调用只读取一次屏幕。
act.mjs会在每个操作前后读取屏幕,并在run.mjs或DRILL后跳过读取。如果没有固定区域,当 sheet、alert、menu 或 popover 打开时,每次读取都需要额外创建一次快照。在密集应用中,一次读取约需 3 秒;使用WIDEN的成本要低得多。--root - 不会重试:遇到 429 时调用即结束。
act.mjs会在失败前分别等待 500 ms 和 1000 ms,对 429、503 和 529 各重试两次。run.mjs - 无法使用 held input,因此不能执行持续按键,也不能执行带 hold 的拖拽。请参阅 。
crates/core/src/commands/input_hold_policy.rs - 此处不会解析 。它不需要元素,因此请直接发送。
press
Files
文件
| File | Holds |
|---|---|
| The operations, how a screen is described, and when to stop. No application is touched here. |
| Which elements a command can reach, which surface is open, how an element is described. Shared by both entry points. |
| The only code that speaks to agent-desktop. |
| The loop and its command line. |
| The single-step resolver. |
| File | 内容 |
|---|---|
| 操作、屏幕描述方式以及停止条件。此处不会接触任何应用。 |
| 某个命令可访问哪些元素、哪个 surface 处于打开状态,以及如何描述元素。由两个入口共享。 |
| 唯一与 agent-desktop 通信的代码。 |
| 循环及其命令行。 |
| 单步解析器。 |
Checks
检查
WAITrun.mjsaction_failedset-valueBINARY_TOO_OLDrun.mjsdone{"ok":false,"error":…}act.mjsexecuted.codeNOT_EXECUTEDdecisionactAfter a paste the field is read back. If it does not hold the requested text the run stops with and does not type again. The read-back compares the exact string, so a field that normalizes text (trims, changes case or line endings) reports a failure even when the text landed. The read is retried until the deadline before it fails.
TEXT_VERIFICATION_FAILEDJEV_VERIFY_MSsh
node scripts/jev/act.test.mjs
node scripts/jev/run.test.mjs
node scripts/jev/run.integration.test.mjs
node scripts/jev/evidence.test.mjsTurns keep native , and for the caller; they are excluded from model history. A verified step with and means no input was needed. A failed postcondition keeps and any nested action evidence.
stepspost_statedetailsAlreadyInStateoutcome: "skipped"delivery: "not_delivered"ok: false在 中, 会暂停 250 ms,然后再次读取屏幕。操作失败时会立即以 停止运行,并保留错误代码和 delivery 状态,且不会重复该操作。焦点设置、剪贴板写入或粘贴失败时,也会以相同方式结束运行。如果二进制文件在 失败后未报告任何错误处置状态,则会以 失败。
run.mjsWAITaction_failedset-valueBINARY_TOO_OLD只有当运行以 停止时, 才退出 0;其他所有停止情况及任何错误()均退出 1,使用错误则退出 2。除非执行的命令失败或未被执行,否则 退出 0;当 不是 时, 为 。错误退出 1,使用错误退出 2。
donerun.mjs{"ok":false,"error":…}act.mjsdecisionactexecuted.codeNOT_EXECUTED粘贴后会回读字段。如果字段未持有请求的文本,运行会以 停止,且不会再次输入。回读会比较精确字符串,因此即使文本实际已写入,对文本进行规范化处理的字段,例如去除空白、改变大小写或换行符,也会报告失败。读取会持续重试,直到达到 截止时间后才失败。
TEXT_VERIFICATION_FAILEDJEV_VERIFY_MSsh
node scripts/jev/act.test.mjs
node scripts/jev/run.test.mjs
node scripts/jev/run.integration.test.mjs
node scripts/jev/evidence.test.mjs各轮会为调用方保留原生的 、 和 ;这些内容会从模型历史中排除。经过验证的 步骤若带有 和 ,表示无需输入。失败的后置条件会保留 以及任何嵌套的操作证据。
stepspost_statedetailsAlreadyInStateoutcome: "skipped"delivery: "not_delivered"ok: false