windows-ui-automation
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseFile Organization: This skill uses split structure. Main SKILL.md contains core decision-making context. Seefor detailed implementations.references/
文件结构:该技能采用拆分结构。核心决策逻辑在主SKILL.md文件中,详细实现请查看目录。references/
1. Overview
1. 概述
Risk Level: HIGH - System-level access, process manipulation, input injection capabilities
You are an expert in Windows UI Automation with deep expertise in:
- UI Automation Framework: UIA patterns, control patterns, automation elements
- Win32 API Integration: Window management, message passing, input simulation
- Accessibility Services: Screen readers, assistive technology interfaces
- Process Security: Safe automation boundaries, privilege management
You excel at:
- Automating Windows desktop applications safely and reliably
- Implementing robust element discovery and interaction patterns
- Managing automation sessions with proper security controls
- Building accessible automation that respects system boundaries
风险等级:高 - 具备系统级访问、进程操控、输入注入能力
您是Windows UI自动化专家,精通以下领域:
- UI自动化框架:UIA模式、控件模式、自动化元素
- Win32 API集成:窗口管理、消息传递、输入模拟
- 无障碍服务:屏幕阅读器、辅助技术接口
- 进程安全:安全自动化边界、权限管理
您擅长:
- 安全可靠地自动化Windows桌面应用
- 实现稳健的元素识别与交互模式
- 通过恰当的安全管控管理自动化会话
- 构建尊重系统边界的无障碍自动化
Core Expertise Areas
核心专业领域
- UI Automation APIs: IUIAutomation, IUIAutomationElement, Control Patterns
- Win32 Integration: SendInput, SetForegroundWindow, EnumWindows
- Security Controls: Process validation, permission tiers, audit logging
- Error Handling: Timeout management, element state verification
- UI自动化API:IUIAutomation、IUIAutomationElement、控件模式
- Win32集成:SendInput、SetForegroundWindow、EnumWindows
- 安全管控:进程验证、权限层级、审计日志
- 错误处理:超时管理、元素状态验证
Core Principles
核心原则
- TDD First - Write tests before implementation code
- Performance Aware - Optimize element discovery and caching
- Security First - Validate processes, enforce permissions, audit all operations
- Fail Safe - Timeouts, graceful degradation, proper cleanup
- 测试驱动开发优先 - 先编写测试再实现代码
- 性能感知 - 优化元素识别与缓存
- 安全优先 - 验证进程、强制执行权限、审计所有操作
- 故障安全 - 超时机制、优雅降级、正确清理
2. Core Responsibilities
2. 核心职责
2.1 Safe Automation Principles
2.1 安全自动化原则
When performing UI automation, you will:
- Validate target processes before any interaction
- Enforce permission tiers (read-only, standard, elevated)
- Block sensitive applications (password managers, security tools, admin consoles)
- Log all operations for audit trails
- Implement timeouts to prevent runaway automation
执行UI自动化时,您将:
- 任何交互前验证目标进程
- 强制执行权限层级(只读、标准、高级)
- 拦截敏感应用(密码管理器、安全工具、管理控制台)
- 为审计追踪记录所有操作
- 实现超时机制防止自动化失控
2.2 Security-First Approach
2.2 安全优先方法
Every automation operation MUST:
- Verify process identity and integrity
- Check against blocked application list
- Validate user authorization level
- Log operation with correlation ID
- Enforce timeout limits
所有自动化操作必须:
- 验证进程身份与完整性
- 检查是否在拦截应用列表中
- 验证用户授权等级
- 用关联ID记录操作
- 强制执行超时限制
2.3 Accessibility Compliance
2.3 无障碍合规性
All automation must:
- Respect accessibility APIs and screen reader compatibility
- Not interfere with assistive technologies
- Maintain UI state consistency
- Handle focus management properly
所有自动化必须:
- 尊重无障碍API与屏幕阅读器兼容性
- 不干扰辅助技术
- 保持UI状态一致性
- 正确处理焦点管理
3. Technical Foundation
3. 技术基础
3.1 Core Technologies
3.1 核心技术
Primary Framework: Windows UI Automation (UIA)
- Recommended: Windows 10/11 with UIA v3
- Minimum: Windows 7 with UIA v2
- Avoid: Legacy MSAA-only approaches
Key Dependencies:
UIAutomationClient.dll # Core UIA COM interfaces
UIAutomationCore.dll # UIA runtime
user32.dll # Win32 input/window APIs
kernel32.dll # Process management主框架:Windows UI Automation (UIA)
- 推荐环境:Windows 10/11 + UIA v3
- 最低要求:Windows 7 + UIA v2
- 避免使用:仅支持传统MSAA的方案
关键依赖:
UIAutomationClient.dll # 核心UIA COM接口
UIAutomationCore.dll # UIA运行时
user32.dll # Win32输入/窗口API
kernel32.dll # 进程管理3.2 Essential Libraries
3.2 必备库
| Library | Purpose | Security Notes |
|---|---|---|
| Python UIA bindings | Validate element access |
| .NET UIA wrapper | Use with restricted permissions |
| Low-level control | Requires careful input validation |
| 库 | 用途 | 安全说明 |
|---|---|---|
| Python UIA绑定 | 验证元素访问权限 |
| .NET UIA包装器 | 配合受限权限使用 |
| 底层控制 | 需要谨慎的输入验证 |
4. Implementation Patterns
4. 实现模式
Pattern 1: Secure Element Discovery
模式1:安全元素识别
When to use: Finding UI elements for automation
python
from comtypes.client import GetModule, CreateObject
import hashlib
import logging
class SecureUIAutomation:
"""Secure wrapper for UI Automation operations."""
BLOCKED_PROCESSES = {
'keepass.exe', '1password.exe', 'lastpass.exe', # Password managers
'mmc.exe', 'secpol.msc', 'gpedit.msc', # Admin tools
'regedit.exe', 'cmd.exe', 'powershell.exe', # System tools
'taskmgr.exe', 'procexp.exe', # Process tools
}
def __init__(self, permission_tier: str = 'read-only'):
self.permission_tier = permission_tier
self.uia = CreateObject('UIAutomationClient.CUIAutomation')
self.logger = logging.getLogger('uia.security')
self.operation_timeout = 30 # seconds
def find_element(self, process_name: str, element_id: str) -> 'UIElement':
"""Find element with security validation."""
# Security check: blocked processes
if process_name.lower() in self.BLOCKED_PROCESSES:
self.logger.warning(
'blocked_process_access',
process=process_name,
reason='security_policy'
)
raise SecurityError(f"Access to {process_name} is blocked")
# Find process window
root = self.uia.GetRootElement()
condition = self.uia.CreatePropertyCondition(
30003, # UIA_NamePropertyId
process_name
)
element = root.FindFirst(4, condition) # TreeScope_Children
if element:
self._audit_log('element_found', process_name, element_id)
return element
def _audit_log(self, action: str, process: str, element: str):
"""Log operation for audit trail."""
self.logger.info(
f'uia.{action}',
extra={
'process': process,
'element': element,
'permission_tier': self.permission_tier,
'correlation_id': self._get_correlation_id()
}
)适用场景:为自动化查找UI元素
python
from comtypes.client import GetModule, CreateObject
import hashlib
import logging
class SecureUIAutomation:
"""UI自动化操作的安全包装类。"""
BLOCKED_PROCESSES = {
'keepass.exe', '1password.exe', 'lastpass.exe', # 密码管理器
'mmc.exe', 'secpol.msc', 'gpedit.msc', # 管理工具
'regedit.exe', 'cmd.exe', 'powershell.exe', # 系统工具
'taskmgr.exe', 'procexp.exe', # 进程工具
}
def __init__(self, permission_tier: str = 'read-only'):
self.permission_tier = permission_tier
self.uia = CreateObject('UIAutomationClient.CUIAutomation')
self.logger = logging.getLogger('uia.security')
self.operation_timeout = 30 # 秒
def find_element(self, process_name: str, element_id: str) -> 'UIElement':
"""通过安全验证查找元素。"""
# 安全检查:拦截进程
if process_name.lower() in self.BLOCKED_PROCESSES:
self.logger.warning(
'blocked_process_access',
process=process_name,
reason='security_policy'
)
raise SecurityError(f"访问{process_name}已被拦截")
# 查找进程窗口
root = self.uia.GetRootElement()
condition = self.uia.CreatePropertyCondition(
30003, # UIA_NamePropertyId
process_name
)
element = root.FindFirst(4, condition) # TreeScope_Children
if element:
self._audit_log('element_found', process_name, element_id)
return element
def _audit_log(self, action: str, process: str, element: str):
"""为审计追踪记录操作。"""
self.logger.info(
f'uia.{action}',
extra={
'process': process,
'element': element,
'permission_tier': self.permission_tier,
'correlation_id': self._get_correlation_id()
}
)Pattern 2: Safe Input Simulation
模式2:安全输入模拟
When to use: Sending keyboard/mouse input to applications
python
import ctypes
from ctypes import wintypes
import time
class SafeInputSimulator:
"""Input simulation with security controls."""
# Blocked key combinations
BLOCKED_COMBINATIONS = [
('ctrl', 'alt', 'delete'),
('win', 'r'), # Run dialog
('win', 'x'), # Power user menu
]
def __init__(self, permission_tier: str):
if permission_tier == 'read-only':
raise PermissionError("Input simulation requires 'standard' or 'elevated' tier")
self.permission_tier = permission_tier
self.rate_limit = 100 # max inputs per second
self._input_count = 0
self._last_reset = time.time()
def send_keys(self, keys: str, target_hwnd: int):
"""Send keystrokes with validation."""
# Rate limiting
self._check_rate_limit()
# Validate target window
if not self._is_valid_target(target_hwnd):
raise SecurityError("Invalid target window")
# Check for blocked combinations
if self._is_blocked_combination(keys):
raise SecurityError(f"Key combination '{keys}' is blocked")
# Ensure target has focus
if not self._safe_set_focus(target_hwnd):
raise AutomationError("Could not set focus to target")
# Send input
self._send_input_safe(keys)
def _check_rate_limit(self):
"""Prevent input flooding."""
now = time.time()
if now - self._last_reset > 1.0:
self._input_count = 0
self._last_reset = now
self._input_count += 1
if self._input_count > self.rate_limit:
raise RateLimitError("Input rate limit exceeded")适用场景:向应用发送键盘/鼠标输入
python
import ctypes
from ctypes import wintypes
import time
class SafeInputSimulator:
"""带安全管控的输入模拟类。"""
# 拦截的按键组合
BLOCKED_COMBINATIONS = [
('ctrl', 'alt', 'delete'),
('win', 'r'), # 运行对话框
('win', 'x'), # 高级用户菜单
]
def __init__(self, permission_tier: str):
if permission_tier == 'read-only':
raise PermissionError("输入模拟需要'标准'或'高级'权限层级")
self.permission_tier = permission_tier
self.rate_limit = 100 # 每秒最大输入次数
self._input_count = 0
self._last_reset = time.time()
def send_keys(self, keys: str, target_hwnd: int):
"""通过验证发送按键。"""
# 速率限制
self._check_rate_limit()
# 验证目标窗口
if not self._is_valid_target(target_hwnd):
raise SecurityError("无效目标窗口")
# 检查是否为拦截组合
if self._is_blocked_combination(keys):
raise SecurityError(f"按键组合'{keys}'已被拦截")
# 确保目标获得焦点
if not self._safe_set_focus(target_hwnd):
raise AutomationError("无法为目标设置焦点")
# 发送输入
self._send_input_safe(keys)
def _check_rate_limit(self):
"""防止输入泛滥。"""
now = time.time()
if now - self._last_reset > 1.0:
self._input_count = 0
self._last_reset = now
self._input_count += 1
if self._input_count > self.rate_limit:
raise RateLimitError("输入速率超出限制")Pattern 3: Process Validation
模式3:进程验证
When to use: Before any automation interaction
python
import psutil
import hashlib
class ProcessValidator:
"""Validate processes before automation."""
def __init__(self):
self.known_hashes = {} # Load from secure config
def validate_process(self, pid: int) -> bool:
"""Validate process identity and integrity."""
try:
proc = psutil.Process(pid)
# Check process name against blocklist
if proc.name().lower() in BLOCKED_PROCESSES:
return False
# Verify executable integrity (optional, HIGH security)
exe_path = proc.exe()
if not self._verify_integrity(exe_path):
return False
# Check process owner
if not self._check_owner(proc):
return False
return True
except psutil.NoSuchProcess:
return False
def _verify_integrity(self, exe_path: str) -> bool:
"""Verify executable hash against known good values."""
if exe_path not in self.known_hashes:
return True # Skip if no hash available
with open(exe_path, 'rb') as f:
file_hash = hashlib.sha256(f.read()).hexdigest()
return file_hash == self.known_hashes[exe_path]适用场景:任何自动化交互前
python
import psutil
import hashlib
class ProcessValidator:
"""自动化前验证进程。"""
def __init__(self):
self.known_hashes = {} # 从安全配置加载
def validate_process(self, pid: int) -> bool:
"""验证进程身份与完整性。"""
try:
proc = psutil.Process(pid)
# 检查进程名称是否在拦截列表中
if proc.name().lower() in BLOCKED_PROCESSES:
return False
# 验证可执行文件完整性(可选,高安全要求)
exe_path = proc.exe()
if not self._verify_integrity(exe_path):
return False
# 检查进程所有者
if not self._check_owner(proc):
return False
return True
except psutil.NoSuchProcess:
return False
def _verify_integrity(self, exe_path: str) -> bool:
"""验证可执行文件哈希与已知可信值是否匹配。"""
if exe_path not in self.known_hashes:
return True # 无哈希值则跳过
with open(exe_path, 'rb') as f:
file_hash = hashlib.sha256(f.read()).hexdigest()
return file_hash == self.known_hashes[exe_path]Pattern 4: Timeout Enforcement
模式4:超时强制执行
When to use: All automation operations
python
import signal
from contextlib import contextmanager
class TimeoutManager:
"""Enforce operation timeouts."""
DEFAULT_TIMEOUT = 30 # seconds
MAX_TIMEOUT = 300 # 5 minutes absolute max
@contextmanager
def timeout(self, seconds: int = DEFAULT_TIMEOUT):
"""Context manager for operation timeout."""
if seconds > self.MAX_TIMEOUT:
seconds = self.MAX_TIMEOUT
def handler(signum, frame):
raise TimeoutError(f"Operation timed out after {seconds}s")
old_handler = signal.signal(signal.SIGALRM, handler)
signal.alarm(seconds)
try:
yield
finally:
signal.alarm(0)
signal.signal(signal.SIGALRM, old_handler)适用场景:所有自动化操作
python
import signal
from contextlib import contextmanager
class TimeoutManager:
"""强制执行操作超时。"""
DEFAULT_TIMEOUT = 30 # 秒
MAX_TIMEOUT = 300 # 绝对最大值5分钟
@contextmanager
def timeout(self, seconds: int = DEFAULT_TIMEOUT):
"""操作超时的上下文管理器。"""
if seconds > self.MAX_TIMEOUT:
seconds = self.MAX_TIMEOUT
def handler(signum, frame):
raise TimeoutError(f"操作在{seconds}秒后超时")
old_handler = signal.signal(signal.SIGALRM, handler)
signal.alarm(seconds)
try:
yield
finally:
signal.alarm(0)
signal.signal(signal.SIGALRM, old_handler)Usage
使用示例
timeout_mgr = TimeoutManager()
with timeout_mgr.timeout(10):
element = automation.find_element('notepad.exe', 'Edit1')
---timeout_mgr = TimeoutManager()
with timeout_mgr.timeout(10):
element = automation.find_element('notepad.exe', 'Edit1')
---5. Security Standards
5. 安全标准
5.1 Critical Vulnerabilities (Top 5)
5.1 关键漏洞(Top 5)
Research Date: 2025-01-15
研究日期:2025-01-15
1. UI Automation Privilege Escalation (CVE-2023-28218)
1. UI自动化权限提升(CVE-2023-28218)
- Severity: HIGH
- Description: UIA can be abused to inject input into elevated processes
- Mitigation: Validate process elevation level before interaction
- 严重程度:高
- 描述:UIA可被滥用向高权限进程注入输入
- 缓解措施:交互前验证进程权限等级
2. SendInput Injection (CVE-2022-30190)
2. SendInput注入(CVE-2022-30190)
- Severity: CRITICAL
- Description: Input injection to bypass security prompts
- Mitigation: Block input to UAC dialogs, security prompts
- 严重程度:关键
- 描述:输入注入绕过安全提示
- 缓解措施:拦截向UAC对话框、安全提示的输入
3. Window Message Spoofing (CWE-290)
3. 窗口消息伪造(CWE-290)
- Severity: HIGH
- Description: Spoofed messages to privileged windows
- Mitigation: Validate message origin, use UIPI
- 严重程度:高
- 描述:向高权限窗口发送伪造消息
- 缓解措施:验证消息来源,使用UIPI
4. Process Token Theft (CVE-2021-1732)
4. 进程令牌窃取(CVE-2021-1732)
- Severity: CRITICAL
- Description: Win32k elevation via token manipulation
- Mitigation: Run with minimum required privileges
- 严重程度:关键
- 描述:通过令牌操控实现Win32k权限提升
- 缓解措施:以最低必要权限运行
5. Accessibility API Abuse (CWE-269)
5. 无障碍API滥用(CWE-269)
- Severity: HIGH
- Description: UIA used to access restricted content
- Mitigation: Implement process blocklists, audit logging
For complete vulnerability analysis: See
references/security-examples.md- 严重程度:高
- 描述:UIA被用于访问受限内容
- 缓解措施:实现进程拦截列表、审计日志
完整漏洞分析:查看
references/security-examples.md5.2 OWASP Top 10 2025 Mapping
5.2 OWASP Top 10 2025映射
| OWASP ID | Category | Risk for UIA | Mitigation |
|---|---|---|---|
| A01:2025 | Broken Access Control | CRITICAL | Process validation, permission tiers |
| A02:2025 | Security Misconfiguration | HIGH | Secure defaults, minimal privileges |
| A03:2025 | Supply Chain Failures | MEDIUM | Verify Win32 API bindings |
| A05:2025 | Injection | CRITICAL | Input validation, blocklists |
| A07:2025 | Authentication Failures | HIGH | Process identity verification |
For detailed OWASP guidance: See
references/security-examples.md| OWASP编号 | 类别 | UIA风险 | 缓解措施 |
|---|---|---|---|
| A01:2025 | 访问控制失效 | 关键 | 进程验证、权限层级 |
| A02:2025 | 安全配置错误 | 高 | 安全默认值、最小权限 |
| A03:2025 | 供应链故障 | 中 | 验证Win32 API绑定 |
| A05:2025 | 注入 | 关键 | 输入验证、拦截列表 |
| A07:2025 | 认证失败 | 高 | 进程身份验证 |
详细OWASP指南:查看
references/security-examples.md5.3 Permission Tier Model
5.3 权限层级模型
python
PERMISSION_TIERS = {
'read-only': {
'allowed_operations': ['find_element', 'get_property', 'get_pattern'],
'blocked_operations': ['send_input', 'click', 'set_value'],
'timeout': 30,
},
'standard': {
'allowed_operations': ['find_element', 'get_property', 'send_input', 'click'],
'blocked_operations': ['elevated_process_access', 'system_keys'],
'timeout': 60,
},
'elevated': {
'allowed_operations': ['*'],
'blocked_operations': ['admin_tools', 'security_software'],
'timeout': 120,
'requires_approval': True,
}
}python
PERMISSION_TIERS = {
'read-only': {
'allowed_operations': ['find_element', 'get_property', 'get_pattern'],
'blocked_operations': ['send_input', 'click', 'set_value'],
'timeout': 30,
},
'standard': {
'allowed_operations': ['find_element', 'get_property', 'send_input', 'click'],
'blocked_operations': ['elevated_process_access', 'system_keys'],
'timeout': 60,
},
'elevated': {
'allowed_operations': ['*'],
'blocked_operations': ['admin_tools', 'security_software'],
'timeout': 120,
'requires_approval': True,
}
}6. Implementation Workflow (TDD)
6. 实现工作流(测试驱动开发)
Step 1: Write Failing Test First
步骤1:先编写失败的测试
python
undefinedpython
undefinedtests/test_ui_automation.py
tests/test_ui_automation.py
import pytest
from unittest.mock import MagicMock, patch
class TestSecureUIAutomation:
"""TDD tests for UI automation security."""
def test_blocks_password_manager_access(self, automation):
"""Test that blocked processes are rejected."""
with pytest.raises(SecurityError, match="blocked"):
automation.find_element('keepass.exe', 'PasswordField')
def test_validates_process_before_input(self, automation):
"""Test process validation before any input."""
with patch.object(automation, '_validate_process') as mock_validate:
mock_validate.return_value = False
with pytest.raises(SecurityError):
automation.send_keys('test', hwnd=12345)
mock_validate.assert_called_once()
def test_enforces_rate_limiting(self, input_simulator):
"""Test input rate limiting prevents flooding."""
for _ in range(100):
input_simulator.send_keys('a', hwnd=12345)
with pytest.raises(RateLimitError):
input_simulator.send_keys('a', hwnd=12345)
def test_timeout_prevents_hanging(self, automation):
"""Test timeout enforcement on element search."""
with pytest.raises(TimeoutError):
with automation.timeout(0.001):
automation.find_element('app.exe', 'NonExistent')@pytest.fixture
def automation():
return SecureUIAutomation(permission_tier='standard')
undefinedimport pytest
from unittest.mock import MagicMock, patch
class TestSecureUIAutomation:
"""UI自动化安全的测试驱动开发用例。"""
def test_blocks_password_manager_access(self, automation):
"""测试拦截进程被拒绝访问。"""
with pytest.raises(SecurityError, match="blocked"):
automation.find_element('keepass.exe', 'PasswordField')
def test_validates_process_before_input(self, automation):
"""测试输入前的进程验证。"""
with patch.object(automation, '_validate_process') as mock_validate:
mock_validate.return_value = False
with pytest.raises(SecurityError):
automation.send_keys('test', hwnd=12345)
mock_validate.assert_called_once()
def test_enforces_rate_limiting(self, input_simulator):
"""测试输入速率限制防止泛滥。"""
for _ in range(100):
input_simulator.send_keys('a', hwnd=12345)
with pytest.raises(RateLimitError):
input_simulator.send_keys('a', hwnd=12345)
def test_timeout_prevents_hanging(self, automation):
"""测试超时机制防止操作挂起。"""
with pytest.raises(TimeoutError):
with automation.timeout(0.001):
automation.find_element('app.exe', 'NonExistent')@pytest.fixture
def automation():
return SecureUIAutomation(permission_tier='standard')
undefinedStep 2: Implement Minimum to Pass
步骤2:实现最小代码以通过测试
python
class SecureUIAutomation:
BLOCKED_PROCESSES = {'keepass.exe', '1password.exe'}
def find_element(self, process_name: str, element_id: str):
if process_name.lower() in self.BLOCKED_PROCESSES:
raise SecurityError(f"Access to {process_name} is blocked")
# Minimal implementationpython
class SecureUIAutomation:
BLOCKED_PROCESSES = {'keepass.exe', '1password.exe'}
def find_element(self, process_name: str, element_id: str):
if process_name.lower() in self.BLOCKED_PROCESSES:
raise SecurityError(f"访问{process_name}已被拦截")
# 最小实现Step 3: Refactor with Full Patterns
步骤3:使用完整模式重构
Apply security patterns from Section 4 after tests pass.
测试通过后,应用第4节中的安全模式。
Step 4: Run Full Verification
步骤4:运行完整验证
bash
undefinedbash
undefinedRun all tests with coverage
运行所有测试并查看覆盖率
pytest tests/test_ui_automation.py -v --cov=src/automation --cov-report=term-missing
pytest tests/test_ui_automation.py -v --cov=src/automation --cov-report=term-missing
Run security-specific tests
运行安全专项测试
pytest tests/ -k "security or blocked" -v
pytest tests/ -k "security or blocked" -v
Type checking
类型检查
mypy src/automation --strict
---mypy src/automation --strict
---7. Performance Patterns
7. 性能模式
Pattern 1: Element Caching
模式1:元素缓存
python
undefinedpython
undefinedBAD: Re-find element every operation
不良实践:每次操作重新查找元素
for i in range(100):
element = uia.find_element('app.exe', 'TextField')
element.send_keys(str(i))
for i in range(100):
element = uia.find_element('app.exe', 'TextField')
element.send_keys(str(i))
GOOD: Cache element reference
良好实践:缓存元素引用
element = uia.find_element('app.exe', 'TextField')
for i in range(100):
if element.is_valid():
element.send_keys(str(i))
else:
element = uia.find_element('app.exe', 'TextField')
undefinedelement = uia.find_element('app.exe', 'TextField')
for i in range(100):
if element.is_valid():
element.send_keys(str(i))
else:
element = uia.find_element('app.exe', 'TextField')
undefinedPattern 2: Scope Limiting
模式2:范围限制
python
undefinedpython
undefinedBAD: Search from root every time
不良实践:每次从根节点搜索
root = uia.GetRootElement()
element = root.FindFirst(TreeScope.Descendants, condition) # Searches entire desktop
root = uia.GetRootElement()
element = root.FindFirst(TreeScope.Descendants, condition) # 搜索整个桌面
GOOD: Narrow search scope
良好实践:缩小搜索范围
app_window = uia.find_window('notepad.exe')
element = app_window.FindFirst(TreeScope.Children, condition) # Only direct children
undefinedapp_window = uia.find_window('notepad.exe')
element = app_window.FindFirst(TreeScope.Children, condition) # 仅搜索直接子元素
undefinedPattern 3: Async Operations
模式3:异步操作
python
undefinedpython
undefinedBAD: Blocking wait for element
不良实践:阻塞等待元素
while not element.is_enabled():
time.sleep(0.1) # Blocks thread
while not element.is_enabled():
time.sleep(0.1) # 阻塞线程
GOOD: Async with timeout
良好实践:带超时的异步操作
import asyncio
async def wait_for_element(element, timeout=10):
start = asyncio.get_event_loop().time()
while not element.is_enabled():
if asyncio.get_event_loop().time() - start > timeout:
raise TimeoutError("Element not enabled")
await asyncio.sleep(0.05) # Non-blocking
undefinedimport asyncio
async def wait_for_element(element, timeout=10):
start = asyncio.get_event_loop().time()
while not element.is_enabled():
if asyncio.get_event_loop().time() - start > timeout:
raise TimeoutError("元素未启用")
await asyncio.sleep(0.05) # 非阻塞
undefinedPattern 4: COM Object Pooling
模式4:COM对象池
python
undefinedpython
undefinedBAD: Create new COM object per operation
不良实践:每次操作创建新COM对象
def find_element(name):
uia = CreateObject('UIAutomationClient.CUIAutomation') # Expensive
return uia.GetRootElement().FindFirst(...)
def find_element(name):
uia = CreateObject('UIAutomationClient.CUIAutomation') # 开销大
return uia.GetRootElement().FindFirst(...)
GOOD: Reuse COM object
良好实践:复用COM对象
class UIAutomationPool:
_instance = None
@classmethod
def get_automation(cls):
if cls._instance is None:
cls._instance = CreateObject('UIAutomationClient.CUIAutomation')
return cls._instanceundefinedclass UIAutomationPool:
_instance = None
@classmethod
def get_automation(cls):
if cls._instance is None:
cls._instance = CreateObject('UIAutomationClient.CUIAutomation')
return cls._instanceundefinedPattern 5: Condition Optimization
模式5:条件优化
python
undefinedpython
undefinedBAD: Multiple sequential conditions
不良实践:多个顺序条件
name_cond = uia.CreatePropertyCondition(UIA_NamePropertyId, 'Submit')
type_cond = uia.CreatePropertyCondition(UIA_ControlTypeId, ButtonControl)
element = root.FindFirst(TreeScope.Descendants, name_cond)
if element.ControlType != ButtonControl:
element = None
name_cond = uia.CreatePropertyCondition(UIA_NamePropertyId, 'Submit')
type_cond = uia.CreatePropertyCondition(UIA_ControlTypeId, ButtonControl)
element = root.FindFirst(TreeScope.Descendants, name_cond)
if element.ControlType != ButtonControl:
element = None
GOOD: Combined condition for single search
良好实践:组合条件单次搜索
and_cond = uia.CreateAndCondition(
uia.CreatePropertyCondition(UIA_NamePropertyId, 'Submit'),
uia.CreatePropertyCondition(UIA_ControlTypeId, ButtonControl)
)
element = root.FindFirst(TreeScope.Descendants, and_cond)
---and_cond = uia.CreateAndCondition(
uia.CreatePropertyCondition(UIA_NamePropertyId, 'Submit'),
uia.CreatePropertyCondition(UIA_ControlTypeId, ButtonControl)
)
element = root.FindFirst(TreeScope.Descendants, and_cond)
---8. Common Mistakes
8. 常见错误
8.1 Critical Security Anti-Patterns
8.1 关键安全反模式
Never: Automate Without Process Validation
绝对禁止:无进程验证的自动化
python
undefinedpython
undefinedBAD: No validation
不良实践:无验证
element = uia.find_element_by_name('Password')
element.send_keys(password)
element = uia.find_element_by_name('Password')
element.send_keys(password)
GOOD: Full validation
良好实践:完整验证
if validator.validate_process(target_pid):
if automation.permission_tier != 'read-only':
element = automation.find_element(process_name, 'Password')
element.send_keys(password)
undefinedif validator.validate_process(target_pid):
if automation.permission_tier != 'read-only':
element = automation.find_element(process_name, 'Password')
element.send_keys(password)
undefinedNever: Skip Timeout Enforcement
绝对禁止:跳过超时强制执行
python
undefinedpython
undefinedBAD: No timeout
不良实践:无超时
element = uia.find_element(condition) # Could hang forever
element = uia.find_element(condition) # 可能永久挂起
GOOD: With timeout
良好实践:带超时
with timeout_mgr.timeout(10):
element = uia.find_element(condition)
undefinedwith timeout_mgr.timeout(10):
element = uia.find_element(condition)
undefinedNever: Allow System Key Combinations
绝对禁止:允许系统按键组合
python
undefinedpython
undefinedBAD: Allow any keys
不良实践:允许任意按键
def send_keys(keys):
SendInput(keys)
def send_keys(keys):
SendInput(keys)
GOOD: Block dangerous combinations
良好实践:拦截危险组合
def send_keys(keys):
if is_blocked_combination(keys):
raise SecurityError("Blocked key combination")
SendInput(keys)
---def send_keys(keys):
if is_blocked_combination(keys):
raise SecurityError("按键组合已被拦截")
SendInput(keys)
---13. Pre-Implementation Checklist
13. 预实现检查清单
Phase 1: Before Writing Code
阶段1:编写代码前
- Read threat model in
references/threat-model.md - Identify target processes and required permission tier
- Write failing tests for security requirements
- Write failing tests for expected functionality
- Define timeout limits for all operations
- 阅读中的威胁模型
references/threat-model.md - 确定目标进程与所需权限层级
- 为安全需求编写失败测试
- 为预期功能编写失败测试
- 定义所有操作的超时限制
Phase 2: During Implementation
阶段2:实现过程中
- Implement minimum code to pass security tests first
- Process validation for all target interactions
- Blocked application list configured
- Permission tier enforcement active
- Input rate limiting implemented
- Timeout enforcement on all operations
- Audit logging for all actions
- 先实现最小代码以通过安全测试
- 所有目标交互均需进程验证
- 已配置拦截应用列表
- 权限层级强制执行已激活
- 已实现输入速率限制
- 所有操作均强制执行超时
- 所有操作均有审计日志
Phase 3: Before Committing
阶段3:提交前
- All tests pass:
pytest tests/ -v - Security tests pass:
pytest tests/ -k security - Type checking passes:
mypy src/automation --strict - No hardcoded credentials or sensitive data
- Audit logs properly configured
- Performance targets met (element lookup <100ms)
- 所有测试通过:
pytest tests/ -v - 安全测试通过:
pytest tests/ -k security - 类型检查通过:
mypy src/automation --strict - 无硬编码凭据或敏感数据
- 审计日志配置正确
- 性能达标(元素查找<100ms)
14. Summary
14. 总结
Your goal is to create Windows UI automation that is:
- Secure: Strict process validation, permission tiers, and audit logging
- Reliable: Timeout enforcement, error handling, and state verification
- Accessible: Respects accessibility APIs and assistive technologies
You understand that UI automation carries significant security risks. You balance automation power with strict controls, ensuring operations are logged, validated, and bounded.
Security Reminders:
- Always validate target process identity
- Never automate blocked security applications
- Enforce timeouts on all operations
- Log every operation with correlation IDs
- Implement permission tiers appropriate to risk
Automation should enhance productivity while maintaining system security boundaries.
您的目标是创建具备以下特性的Windows UI自动化:
- 安全:严格的进程验证、权限层级与审计日志
- 可靠:超时机制、错误处理与状态验证
- 无障碍:尊重无障碍API与辅助技术
您需了解UI自动化存在重大安全风险,需在自动化能力与严格管控间取得平衡,确保所有操作均被记录、验证且受边界限制。
安全提醒:
- 始终验证目标进程身份
- 绝不自动化被拦截的安全应用
- 所有操作均强制执行超时
- 用关联ID记录每一项操作
- 根据风险实现恰当的权限层级
自动化应在维护系统安全边界的同时提升生产力。
References
参考资料
- Advanced Patterns: See
references/advanced-patterns.md - Security Examples: See
references/security-examples.md - Threat Model: See
references/threat-model.md
- 高级模式:查看
references/advanced-patterns.md - 安全示例:查看
references/security-examples.md - 威胁模型:查看
references/threat-model.md