game-automation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Game automation: the agent's hands and eyes

游戏自动化：Agent 的双手和眼睛

The real game is the oracle. The loop is launch → navigate → set up the scene → act → screenshot and read logs → decide. Build it once per game, then every change you make can be checked in a minute.

真实游戏就是判定依据。闭环流程是：启动 → 导航 → 设置场景 → 操作 → 截图并读取日志 → 作出决策。为每个游戏构建一次，此后你做出的每次修改都能在一分钟内完成验证。

Windows (native or from WSL):

um win

Windows（原生运行或从 WSL 运行）：

um win

bash

um win setup                                   # once: PowerShell tools + an ffmpeg with gfxcapture, picks NVENC/AMF/QSV/x264
um win ps                                      # windowed processes: pid, name, title
um win launch --steam 105600                   # or: um win launch "C:\Games\Foo\Foo.exe" -- -windowed
um win shot --exe Terraria.exe shot.png --scale 0.33   # full frame + a 1/3 copy that's cheap to look at
um win drive --proc Terraria "focus" "click 640 360" "key 0x1B" "type hello" "hold 0x44 1500"
um win drive --proc Terraria idle              # seconds since the user last touched mouse/keyboard
um win kill <pid>                              # exact PID only
um win reg get "HKCU\Software\..."             # registry (reg set backs the key up first)

From Python (for longer scripts),

from um.win import Drive, shot, Recorder

, then

d = Drive("AoE2DE_s")

d.focus()

d.click(x, y)

d.key("0x0D")

WinDrive commands (

um/ps1/WinDrive.ps1

; stdin protocol, one reply per line). Coordinates are in the game window's client area.

Mouse:

move x y

click x y [right]

mdown

mup

drag x0 y0 x1 y1

rel dx dy

(FPS cameras / raw input),

wheel 120

Keyboard:
```
key <vk> [tap|down|up]
```
,
```
hold <vk> <ms>
```
,
```
type <text>
```
,
```
scanmode on
```
(DirectInput / raw-input games that ignore virtual-key events).
Window:
```
focus
```
,
```
rect
```
,
```
size <w> <h>
```
(sets the client size),
```
title
```
,
```
fg
```
,
```
idle
```
,
```
untop
```
.
Virtual-key codes: Esc 0x1B, Enter 0x0D, Space 0x20, W/A/S/D 0x57/0x41/0x53/0x44, F1 0x70, Shift 0x10, Ctrl 0x11, arrows 0x25-0x28.

Rules of the road

Never focus a game with an online mode while the human is typing. It happened with GTA V: the agent focused GTA to click Story Mode, the human's keystrokes hit GTA's landing page, and GTA warned about "accessing GTA Online servers with an altered version". ScriptHookV blocked it; don't rely on that. Check
```
idle
```
, ask the human to click, and launch with the anti-cheat off so online can't start. See
```
knowledge/techniques/driving-real-games-safely.md
```
.
Input only goes to the game. WinDrive refuses to send while another app is in the foreground. The one exception is when nothing is and the cursor is over the game, which windowed games cause by dropping the foreground on clicks.
The user may be at the PC. If
```
idle
```
is under a minute, ask before driving, and keep sessions short. Unattended runs are fine once the user says so.
Screenshots cost tokens. Look at
```
--scale 0.33
```
copies. Multiply coordinates back ×3 for clicks, and keep a table of menu click points in MODLOG.md, measured once.
Why gfxcapture: GPU-rendered games come out black with GDI capture. gfxcapture (Windows.Graphics.Capture) grabs one window's real frames, even when covered. It can't capture minimized windows.

bash

um win setup                                   # once: PowerShell tools + an ffmpeg with gfxcapture, picks NVENC/AMF/QSV/x264
um win ps                                      # windowed processes: pid, name, title
um win launch --steam 105600                   # or: um win launch "C:\Games\Foo\Foo.exe" -- -windowed
um win shot --exe Terraria.exe shot.png --scale 0.33   # full frame + a 1/3 copy that's cheap to look at
um win drive --proc Terraria "focus" "click 640 360" "key 0x1B" "type hello" "hold 0x44 1500"
um win drive --proc Terraria idle              # seconds since the user last touched mouse/keyboard
um win kill <pid>                              # exact PID only
um win reg get "HKCU\Software\..."             # registry (reg set backs the key up first)

对于较长的脚本，可以从 Python 中使用

from um.win import Drive, shot, Recorder

，然后

d = Drive("AoE2DE_s")

，

d.focus()

、

d.click(x, y)

、

d.key("0x0D")

。

WinDrive 命令（

um/ps1/WinDrive.ps1

；采用 stdin 协议，每行返回一条回复）。坐标基于游戏窗口的客户区。

鼠标：

move x y

、

click x y [right]

、

mdown

mup

、

drag x0 y0 x1 y1

、

rel dx dy

（FPS 相机／原始输入）、

wheel 120

。

键盘：
```
key <vk> [tap|down|up]
```
、
```
hold <vk> <ms>
```
、
```
type <text>
```
、
```
scanmode on
```
（适用于会忽略虚拟按键事件的 DirectInput／原始输入游戏）。
窗口：
```
focus
```
、
```
rect
```
、
```
size <w> <h>
```
（设置客户区尺寸）、
```
title
```
、
```
fg
```
、
```
idle
```
、
```
untop
```
。
虚拟键码：Esc 0x1B、Enter 0x0D、Space 0x20、W/A/S/D 0x57/0x41/0x53/0x44、F1 0x70、Shift 0x10、 Ctrl 0x11、方向键 0x25-0x28。

行动守则

当用户正在输入时，绝不要聚焦启用了在线模式的游戏。 GTA V 就发生过这种情况：Agent 聚焦 GTA 并点击“故事模式”，用户的按键输入落到了 GTA 的登录页面，GTA 随后警告“正在使用修改后的版本访问 GTA Online 服务器”。ScriptHookV 阻止了这种情况；不要依赖它。检查
```
idle
```
，请用户代为点击，并在关闭反作弊的情况下启动游戏，确保无法进入在线模式。请参阅
```
knowledge/techniques/driving-real-games-safely.md
```
。
输入只会发送到游戏。 当其他应用处于前台时，WinDrive 会拒绝发送输入。唯一的例外是当前没有应用处于前台、且光标悬停在游戏上时；窗口模式游戏在点击时失去前台状态，因此会出现这种情况。
用户可能正在使用电脑。 如果
```
idle
```
不到一分钟，请先征得同意再进行操控，并保持会话简短。用户明确说明后，可以无人值守运行。
截图会消耗 token。 查看
```
--scale 0.33
```
的副本。点击时将坐标乘以 3，并在 MODLOG.md 中维护菜单点击点对照表，只需测量一次。
为何使用 gfxcapture： GPU 渲染的游戏通过 GDI 捕获时通常只会得到黑屏。gfxcapture（Windows.Graphics.Capture）可以获取单个窗口的真实画面，即使该窗口被其他窗口遮挡。它无法捕获最小化的窗口。

Make the window predictable

让窗口行为可预测

Windowed mode at a fixed client size. Every game hides this setting somewhere:

an ini (

[/Script/Engine.GameUserSettings]

in UE

GameUserSettings.ini

, Unity's

Screenmanager Fullscreen mode

Resolution

registry values under

HKCU\Software\<company>\<product>

);

a launch flag (

-windowed -w 1920 -h 1080

, Unity

-screen-fullscreen 0 -screen-width 1920 -screen-height 1080

, Source

-sw -w 1920 -h 1080

);

or the game's own registry (AoE2:
```
Mode Display
```
0 +
```
Windowed Width/Height
```
).
```
um win drive --proc X "size 1920 1080"
```
also resizes most windowed games.

Skip intros: launch flags (
```
-skipintro
```
,
```
-nosplash
```
, Unity
```
-popupwindow
```
), or delete/rename intro videos in a copy of the game.
Jump straight in: loader flags (tModLoader
```
-skipselect Player:World
```
), a save made for testing, scenario auto-load, a dev console command.

Background throttling: many games throttle or pause unfocused. Find the setting (Terraria:

Main.instance.InactiveSleepTime = TimeSpan.Zero

; Unity:

Application.runInBackground = true

; UE:

t.IdleWhenNotForeground 0

以固定客户区尺寸运行窗口模式。 每个游戏都把这项设置藏在不同位置：
- 某个 ini 文件（UE 的
```
GameUserSettings.ini
```
  中有
```
[/Script/Engine.GameUserSettings]
```
  ；Unity 的
```
Screenmanager Fullscreen mode
```
  ／
```
Resolution
```
  注册表值位于
```
HKCU\Software\<company>\<product>
```
  下）；
- 启动参数（
```
-windowed -w 1920 -h 1080
```
  、Unity 的
```
-screen-fullscreen 0 -screen-width 1920 -screen-height 1080
```
  、Source 的
```
-sw -w 1920 -h 1080
```
  ）；
- 或游戏自己的注册表（AoE2：
```
Mode Display
```
  0 +
```
Windowed Width/Height
```
  ）。
```
um win drive --proc X "size 1920 1080"
```
  也能调整大多数窗口模式游戏的大小。
跳过片头： 使用启动参数（
```
-skipintro
```
、
```
-nosplash
```
、Unity 的
```
-popupwindow
```
），或者在游戏的副本中删除／重命名片头视频。
直接进入游戏： 使用加载器参数（tModLoader 的
```
-skipselect Player:World
```
）、为测试制作的存档、场景自动加载或开发者控制台命令。
后台节流： 许多游戏在失去焦点时会降低运行速度或暂停。找到对应设置（Terraria：
```
Main.instance.InactiveSleepTime = TimeSpan.Zero
```
；Unity：
```
Application.runInBackground = true
```
；UE：
```
t.IdleWhenNotForeground 0
```
）。

Better than clicking: a scripted scene or a bridge

比点击更好的方案：脚本化场景或 bridge

Chat or console commands in your mod give items, spawn enemies and teleport. They make tests one line (
```
/arsenal
```
,
```
/mothership
```
).
Test scenes: a mod-side timeline (spawn waves at frame N, fire at N+30), a scenario with triggers (AoE2 via AoE2ScenarioParser), a dedicated test world with a pristine backup.
Agent bridge: a JSON-lines socket inside the mod on
```
127.0.0.1
```
(see
```
examples/terraria-tmodloader/reference/AgentBridge.cs
```
).
- Commands like
```
observe
```
  (menu options or world state as text),
```
click <id>
```
  ,
```
controls
```
  ,
```
step
```
  .
- Handle every request on the game's main thread after an update, so replies are consistent.
- Read menus generically from the UI tree, so the agent sees what a player would. Leave destructive buttons (delete) out.
- An agent can then play without pixels at all.
Out-of-process backend: keep the in-game part thin and put the brains outside (HTTP/file-drop). That pattern scales to AI NPCs and cross-game mashups.

在 mod 中提供聊天或控制台命令，用于发放物品、生成敌人和传送。这能让测试缩减为一行（
```
/arsenal
```
、
```
/mothership
```
）。
测试场景： mod 侧的时间线（第 N 帧生成波次、第 N+30 帧开火）、带触发器的场景（AoE2 通过 AoE2ScenarioParser）、带未改动状态备份的专用测试世界。
Agent bridge： mod 内部的 JSON-lines socket，监听
```
127.0.0.1
```
（请参阅
```
examples/terraria-tmodloader/reference/AgentBridge.cs
```
）。
- 诸如
```
observe
```
  （以文本形式返回菜单选项或世界状态）、
```
click <id>
```
  、
```
controls
```
  、
```
step
```
  等命令。
- 在更新后于游戏主线程上处理每个请求，确保回复一致。
- 从 UI 树中泛化读取菜单，让 Agent 看到玩家实际能看到的内容。排除破坏性按钮（删除）。
- 这样 Agent 就可以完全不依赖像素进行游戏。
进程外后端： 保持游戏内部分轻量，将核心逻辑放在外部（HTTP／文件投放）。这种模式可扩展到 AI NPC 和跨游戏组合。

Crashes and cleanup

崩溃与清理

Crash reporters (BugSplat
```
BsSndRpt64.exe
```
,
```
CrashReportClient.exe
```
,
```
UnityCrashHandler64.exe
```
) can linger and make Steam say "already running".
```
um win ps
```
to spot them,
```
um win kill <pid>
```
to clear them.
Never
```
pkill -f
```
or wildcard
```
taskkill /IM
```
. The pattern can match your own shell or other apps.
After a crash, read the loader or game log first (the mod-any-game skill lists them), then the Windows Event Viewer (Application log) for native crashes.

崩溃报告器（BugSplat 的
```
BsSndRpt64.exe
```
、
```
CrashReportClient.exe
```
、
```
UnityCrashHandler64.exe
```
）可能会残留运行，导致 Steam 提示“游戏已在运行”。使用
```
um win ps
```
找出它们，再使用
```
um win kill <pid>
```
清理。
绝不要使用
```
pkill -f
```
或带通配符的
```
taskkill /IM
```
。这种匹配模式可能命中你自己的 shell 或其他应用。
崩溃后，先读取加载器或游戏日志（mod-any-game skill 中列出了相关日志），再查看 Windows 事件查看器（Application 日志）以了解原生崩溃。

Linux and macOS

Linux 和 macOS

Linux (X11):

xdotool search --name "Game" windowactivate --sync key Escape

xdotool mousemove --window $W 640 360 click 1

. Screenshots:

import -window $(xdotool search --name Game) shot.png

, or ffmpeg

x11grab

Linux (Wayland): ydotool +
```
grim
```
. Proton games are Windows games under Wine: launch options (
```
WINEDLLOVERRIDES
```
,
```
PROTON_LOG=1
```
) go in Steam.
macOS:
```
screencapture -l <windowid> shot.png
```
(window id via
```
GetWindowID
```
or AppleScript),
```
osascript
```
/
```
cliclick
```
for input. Input and screen recording need Accessibility and Screen Recording permission for the terminal.

Linux（X11）：

xdotool search --name "Game" windowactivate --sync key Escape

，

xdotool mousemove --window $W 640 360 click 1

。截图：

import -window $(xdotool search --name Game) shot.png

，或使用 ffmpeg 的

x11grab

。

Linux（Wayland）： ydotool +
```
grim
```
。Proton 游戏是 Wine 下运行的 Windows 游戏：启动选项（
```
WINEDLLOVERRIDES
```
、
```
PROTON_LOG=1
```
）在 Steam 中设置。
macOS：
```
screencapture -l <windowid> shot.png
```
（窗口 id 可通过
```
GetWindowID
```
或 AppleScript 获取），使用
```
osascript
```
／
```
cliclick
```
进行输入。终端需要授予辅助功能和屏幕录制权限。

game-automation

Original

Translation

Game automation: the agent's hands and eyes

游戏自动化：Agent 的双手和眼睛

Windows (native or from WSL):
`um win`

Windows（原生运行或从 WSL 运行）：
`um win`

Make the window predictable

让窗口行为可预测

Better than clicking: a scripted scene or a bridge

比点击更好的方案：脚本化场景或 bridge

Crashes and cleanup

崩溃与清理

Linux and macOS

Linux 和 macOS

game-automation

Original

Translation

Game automation: the agent's hands and eyes

游戏自动化：Agent 的双手和眼睛

Windows (native or from WSL): um win

Windows（原生运行或从 WSL 运行）：um win

Make the window predictable

让窗口行为可预测

Better than clicking: a scripted scene or a bridge

比点击更好的方案：脚本化场景或 bridge

Crashes and cleanup

崩溃与清理

Linux and macOS

Linux 和 macOS

Windows (native or from WSL):
`um win`

Windows（原生运行或从 WSL 运行）：
`um win`