simulation-validator
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseSimulation Validator
模拟验证器
Goal
目标
Provide a three-stage validation protocol: pre-flight checks, runtime monitoring, and post-flight validation for materials simulations.
为材料模拟提供三阶段验证流程:起飞前检查、运行时监控和起飞后验证。
Requirements
要求
- Python 3.10+
- No external dependencies (uses Python standard library only)
- Works on Linux, macOS, and Windows
- Python 3.10+
- 无外部依赖(仅使用Python标准库)
- 支持Linux、macOS和Windows
Inputs to Gather
需要收集的输入
Before running validation scripts, collect from the user:
| Input | Description | Example |
|---|---|---|
| Config file | Simulation configuration (JSON/YAML) | |
| Log file | Runtime output log | |
| Metrics file | Post-run metrics (JSON) | |
| Required params | Parameters that must exist | |
| Valid ranges | Parameter bounds | |
在运行验证脚本前,向用户收集以下信息:
| 输入 | 描述 | 示例 |
|---|---|---|
| 配置文件 | 模拟配置(JSON/YAML) | |
| 日志文件 | 运行时输出日志 | |
| 指标文件 | 运行后指标(JSON) | |
| 必填参数 | 必须存在的参数 | |
| 有效范围 | 参数边界 | |
Decision Guidance
决策指南
When to Run Each Stage
各阶段运行时机
Is simulation about to start?
├── YES → Run Stage 1: preflight_checker.py
│ └── BLOCK status? → Fix issues, do NOT run simulation
│ └── WARN status? → Review warnings, document if accepted
│ └── PASS status? → Proceed to run simulation
│
Is simulation running?
├── YES → Run Stage 2: runtime_monitor.py (periodically)
│ └── Alerts? → Consider stopping, check parameters
│
Has simulation finished?
├── YES → Run Stage 3: result_validator.py
│ └── Failed checks? → Do NOT use results
│ → Run failure_diagnoser.py
│ └── All passed? → Results are valid模拟即将启动?
├── 是 → 运行阶段1:preflight_checker.py
│ └── BLOCK状态? → 修复问题,**不要**启动模拟
│ └── WARN状态? → 查看警告,记录已接受的风险
│ └── PASS状态? → 继续启动模拟
│
模拟正在运行?
├── 是 → 定期运行阶段2:runtime_monitor.py
│ └── 出现警报? → 考虑停止模拟,检查参数
│
模拟已完成?
├── 是 → 运行阶段3:result_validator.py
│ └── 检查未通过? → **不要**使用结果
│ → 运行failure_diagnoser.py
│ └── 全部通过? → 结果有效Choosing Validation Thresholds
选择验证阈值
| Metric | Conservative | Standard | Relaxed |
|---|---|---|---|
| Mass tolerance | 1e-6 | 1e-3 | 1e-2 |
| Residual growth | 2x | 10x | 100x |
| dt reduction | 10x | 100x | 1000x |
| 指标 | 保守型 | 标准型 | 宽松型 |
|---|---|---|---|
| 质量容差 | 1e-6 | 1e-3 | 1e-2 |
| 残差增长 | 2x | 10x | 100x |
| dt 缩减 | 10x | 100x | 1000x |
Script Outputs (JSON Fields)
脚本输出(JSON字段)
| Script | Output Fields |
|---|---|
| |
| |
| |
| |
| 脚本 | 输出字段 |
|---|---|
| |
| |
| |
| |
Three-Stage Validation Protocol
三阶段验证流程
Stage 1: Pre-flight (Before Simulation)
阶段1:起飞前(模拟启动前)
- Run
scripts/preflight_checker.py --config simulation.json - BLOCK status: Stop immediately, fix all blocker issues
- WARN status: Review warnings, document accepted risks
- PASS status: Proceed to simulation
Note:validates required keys, numeric ranges, output-directory access, and disk space. It does not evaluate numerical stability (CFL / diffusion-Fourier). For explicit stability gating usepreflight_checker.py.skills/core-numerical/numerical-stability/scripts/cfl_checker.py
bash
python3 scripts/preflight_checker.py \
--config simulation.json \
--required dt,dx,kappa \
--ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \
--min-free-gb 1.0 \
--json- 运行
scripts/preflight_checker.py --config simulation.json - BLOCK状态:立即停止,修复所有阻塞问题
- WARN状态:查看警告,记录已接受的风险
- PASS状态:继续启动模拟
注意:会验证必填键、数值范围、输出目录权限和磁盘空间。它不评估数值稳定性(CFL / 扩散-傅里叶)。如需显式稳定性校验,请使用preflight_checker.py。skills/core-numerical/numerical-stability/scripts/cfl_checker.py
bash
python3 scripts/preflight_checker.py \
--config simulation.json \
--required dt,dx,kappa \
--ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \
--min-free-gb 1.0 \
--jsonStage 2: Runtime (During Simulation)
阶段2:运行时(模拟进行中)
- Run periodically
scripts/runtime_monitor.py --log simulation.log - Configure alert thresholds based on problem type
- Stop simulation if critical alerts appear
bash
python3 scripts/runtime_monitor.py \
--log simulation.log \
--residual-growth 10.0 \
--dt-drop 100.0 \
--json- 定期运行
scripts/runtime_monitor.py --log simulation.log - 根据问题类型配置警报阈值
- 出现严重警报时停止模拟
bash
python3 scripts/runtime_monitor.py \
--log simulation.log \
--residual-growth 10.0 \
--dt-drop 100.0 \
--jsonStage 3: Post-flight (After Simulation)
阶段3:起飞后(模拟完成后)
- Run
scripts/result_validator.py --metrics results.json - All checks PASS: Results are valid for analysis
- Any check FAIL: Do NOT use results, diagnose failure
bash
python3 scripts/result_validator.py \
--metrics results.json \
--bound-min 0.0 \
--bound-max 1.0 \
--mass-tol 1e-3 \
--jsonFor variational / gradient-flow models (Allen-Cahn, Cahn-Hilliard), add
to enforce a strict monotone non-increasing energy check.
--variational- 运行
scripts/result_validator.py --metrics results.json - 全部检查通过:结果可用于分析
- 任意检查失败:不要使用结果,诊断故障原因
bash
python3 scripts/result_validator.py \
--metrics results.json \
--bound-min 0.0 \
--bound-max 1.0 \
--mass-tol 1e-3 \
--json对于变分/梯度流模型(Allen-Cahn、Cahn-Hilliard),添加参数以强制严格的单调非递增能量检查。
--variationalFailure Diagnosis
故障诊断
When validation fails:
bash
python3 scripts/failure_diagnoser.py --log simulation.log --json验证失败时:
bash
python3 scripts/failure_diagnoser.py --log simulation.log --jsonConversational Workflow Example
对话工作流示例
User: My phase field simulation crashed after 1000 steps. Can you help me figure out why?
Agent workflow:
- First, check the log for obvious errors:
bash
python3 scripts/failure_diagnoser.py --log simulation.log --json - If diagnosis suggests numerical blow-up, check runtime stats:
bash
python3 scripts/runtime_monitor.py --log simulation.log --json - Recommend fixes based on findings:
- If residual grew rapidly → reduce time step
- If dt collapsed → check stability conditions
- If NaN detected → check initial conditions
用户:我的相场模拟在1000步后崩溃了,能帮我找出原因吗?
Agent工作流:
- 首先,检查日志中的明显错误:
bash
python3 scripts/failure_diagnoser.py --log simulation.log --json - 如果诊断显示数值爆炸,检查运行时统计数据:
bash
python3 scripts/runtime_monitor.py --log simulation.log --json - 根据结果推荐修复方案:
- 残差快速增长 → 减小时间步长
- dt崩溃 → 检查稳定性条件
- 检测到NaN → 检查初始条件
Error Handling
错误处理
| Error | Cause | Resolution |
|---|---|---|
| File path invalid | Verify config path exists |
| Parameter is not a number | Fix config file format |
| Parameter outside bounds | Adjust parameter or bounds |
| Permission issue | Check directory permissions |
| Disk nearly full on the output volume | Free up space or reduce output |
| | Use only letters, digits, |
| Inverted/degenerate | Ensure max > min |
| | Pass a finite positive value |
| Log exceeds the 500 MB parse cap | Truncate or pre-filter the log |
| 错误 | 原因 | 解决方法 |
|---|---|---|
| 文件路径无效 | 验证配置文件路径是否存在 |
| 参数不是数值类型 | 修复配置文件格式 |
| 参数超出边界 | 调整参数或边界范围 |
| 权限问题 | 检查目录权限 |
| 输出卷磁盘空间不足 | 释放空间或减少输出内容 |
| | 仅使用字母、数字、 |
| | 确保最大值大于最小值 |
| 提供了 | 传入有限的正数 |
| 日志文件超过500MB解析上限 | 截断或预过滤日志 |
Interpretation Guidance
解读指南
Status Meanings
状态含义
| Status | Meaning | Action |
|---|---|---|
| PASS | All checks passed | Proceed with confidence |
| WARN | Non-critical issues found | Review and document |
| BLOCK | Critical issues found | Must fix before proceeding |
| 状态 | 含义 | 操作 |
|---|---|---|
| PASS | 所有检查通过 | 放心继续 |
| WARN | 发现非关键问题 | 查看并记录 |
| BLOCK | 发现关键问题 | 必须修复后再继续 |
Confidence Score Interpretation
置信度分数解读
| Score | Meaning |
|---|---|
| 1.0 | All validation checks passed → proceed with confidence |
| 0.75+ | Most checks passed, minor issues |
| 0.5-0.75 | Significant issues, review carefully |
| < 0.5 | Major problems, do not trust results |
| No recognized metrics fields; no check ran — NOT a pass. Inspect the metrics file. |
A requested bound (/) with no matching /
in the metrics is reported as a failed check, never a vacuous pass.
For variational/gradient-flow runs, pass (or set
in the metrics) to enforce a strict monotone non-increasing energy check ();
otherwise a weaker check is used, which does not detect mid-run spikes.
--bound-min--bound-maxfield_minfield_maxbounds_unverifiable--variational"energy_variational": trueenergy_monotoneenergy_net_decrease| 分数 | 含义 |
|---|---|
| 1.0 | 所有验证检查通过 → 放心继续 |
| 0.75+ | 大部分检查通过,存在轻微问题 |
| 0.5-0.75 | 存在显著问题,需仔细检查 |
| < 0.5 | 存在重大问题,不要信任结果 |
| 未识别到指标字段;未运行任何检查 — 不代表通过。请检查指标文件。 |
若指标文件中没有匹配的/,则请求的边界(/)会被报告为检查失败,绝不会被视为默认通过。对于变分/梯度流运行,传入(或在指标中设置)以强制严格的单调非递增能量检查();否则会使用较弱的检查,无法检测运行中途的能量峰值。
field_minfield_max--bound-min--bound-maxbounds_unverifiable--variational"energy_variational": trueenergy_monotoneenergy_net_decreaseCommon Failure Patterns
常见失败模式
| Pattern in Log | Likely Cause | Recommended Fix |
|---|---|---|
| NaN, Inf, overflow | Numerical instability | Reduce dt, increase damping |
| max iterations, did not converge | Solver failure | Tune preconditioner, tolerances |
| out of memory | Memory exhaustion | Reduce mesh, enable out-of-core |
| dt reduced | Adaptive stepping triggered | May be okay if controlled |
| 日志中的模式 | 可能原因 | 推荐修复方案 |
|---|---|---|
| NaN、Inf、溢出 | 数值不稳定 | 减小dt,增加阻尼 |
| 达到最大迭代次数,未收敛 | 求解器失败 | 调整预处理器、容差 |
| 内存不足 | 内存耗尽 | 减小网格,启用核外计算 |
| dt 减小 | 触发自适应步长 | 若处于可控范围内则无问题 |
Verification checklist
验证清单
Do not trust a validation verdict until each applicable item below is satisfied
with the concrete artifact named. Record these in your summary to the user.
- Ran and confirmed
result_validator.py --jsonisresults.status(notPASS) ANDINSUFFICIENT_DATA; aresults.confidence_score == 1.0score ornullmeans no check ran — treat as unverified, not as a pass.INSUFFICIENT_DATA - Listed and confirmed every requested check actually appears (e.g.
results.checks,mass_conserved,bounds_satisfied, andno_nan/energy_monotone); confirmedenergy_net_decreaseis empty and contains noresults.failed_checksentry (which means a requested bound had nobounds_unverifiable/field_minto compare against).field_max - For variational/gradient-flow models (Allen-Cahn, Cahn-Hilliard), passed (or set
--variational) so"energy_variational": trueis enforced; recorded that the weakerenergy_monotonewas NOT relied on, since it cannot detect mid-run energy spikes.energy_net_decrease - Recorded the mass drift tolerance used (, default
--mass-tol) and confirmed it matches the Conservative/Standard/Relaxed column appropriate to the run; did not silently accept the default for a tight-conservation problem.1e-3 - Ran and recorded
runtime_monitor.py --json(min/max/last) andresidual_stats; confirmed there are nodt_statsfor NaN/Inf/overflow, residual growth abovealerts, or dt collapse below--residual-growth.--dt-drop - Confirmed numerical stability was gated separately via (CFL/Fourier limit) —
core-numerical/numerical-stability/scripts/cfl_checker.pydoes NOT evaluate CFL/Fourier and a PASS preflight says nothing about temporal/spatial stability.preflight_checker.py - On any or alert, ran
FAILand recorded thefailure_diagnoser.py --json/probable_causes, rather than reusing the results.recommended_fixes
在确认以下所有适用项均满足之前,不要信任验证结论。将这些内容记录到给用户的总结中。
- 运行并确认
result_validator.py --json为results.status(而非PASS)且INSUFFICIENT_DATA;分数为results.confidence_score == 1.0或状态为null意味着未运行任何检查 — 视为未验证,而非通过。INSUFFICIENT_DATA - 列出并确认所有请求的检查均已执行(如
results.checks、mass_conserved、bounds_satisfied以及no_nan/energy_monotone);确认energy_net_decrease为空且不包含results.failed_checks条目(该条目表示请求的边界无对应的bounds_unverifiable/field_min可对比)。field_max - 对于变分/梯度流模型(Allen-Cahn、Cahn-Hilliard),已传入(或设置
--variational)以强制"energy_variational": true检查;记录未依赖较弱的energy_monotone检查,因为它无法检测运行中途的能量峰值。energy_net_decrease - 记录使用的质量漂移容差(,默认值
--mass-tol)并确认其符合运行对应的保守型/标准型/宽松型列;对于严格守恒问题,不要默认接受默认值。1e-3 - 运行并记录
runtime_monitor.py --json(最小值/最大值/最后值)和residual_stats;确认没有关于NaN/Inf/溢出、残差增长超过dt_stats或dt崩溃低于--residual-growth的--dt-drop。alerts - 确认已通过单独校验数值稳定性(CFL/Fourier极限) —
core-numerical/numerical-stability/scripts/cfl_checker.py不评估CFL/Fourier,起飞前检查通过不代表时间/空间稳定性。preflight_checker.py - 出现任何或警报时,运行
FAIL并记录failure_diagnoser.py --json/probable_causes,不要复用旧结果。recommended_fixes
Common pitfalls & rationalizations
常见误区与合理化借口
| Tempting shortcut | Why it's wrong / what to do |
|---|---|
| "Preflight passed, so the run is numerically stable." | |
" | An empty or unrecognized metrics file returns |
| "Energy ends lower than it started, so the dissipative run is fine." | The default |
"I asked for bounds and didn't get a | If |
| "The simulation finished without crashing, so the results are trustworthy." | Run completion is not correctness. Verify mass conservation, energy behavior, physical bounds, and a clean |
| "dt got smaller during the run, so the solver is failing." | |
| "I'll just use the default thresholds." | Defaults ( |
| 诱人的捷径 | 为何错误 / 正确做法 |
|---|---|
| “起飞前检查通过了,所以运行过程数值稳定。” | |
“ | 空的或未识别的指标文件会返回 |
| “最终能量低于初始值,所以耗散型运行没问题。” | 默认的 |
“我要求检查边界但没有得到 | 若 |
| “模拟正常完成没有崩溃,所以结果可信。” | 运行完成不代表结果正确。在使用结果前,请验证质量守恒、能量行为、物理边界以及 |
| “运行过程中dt变小了,所以求解器失败了。” | |
| “我直接使用默认阈值就好。” | 默认值( |
Security
安全
Input Validation
输入验证
- Config file paths are validated for existence before parsing; non-existent paths produce clear errors (exit code 2)
- parameter names are validated against a safe-character allowlist (
--required); names with shell metacharacters are rejected^[A-Za-z0-9_.-]+$ - entries are parsed as
--rangeswith finite numeric bounds enforced andname:min:maxrequiredmax > min - is validated as a finite positive number (negatives, zero,
--min-free-gb,nanrejected)inf - and
--residual-growththresholds are validated as finite positive numbers--dt-drop - and
--bound-minare validated as finite numbers (--bound-max/nanrejected), andinfis enforced;--bound-max > --bound-minis validated as a finite positive number--mass-tol - Invalid input exits with code 2 and an explanatory message
- 解析前会验证配置文件路径是否存在;路径无效时会输出清晰错误(退出码2)
- 参数名称会根据安全字符白名单(
--required)验证;包含shell元字符的名称会被拒绝^[A-Za-z0-9_.-]+$ - 条目会被解析为
--ranges格式,强制要求有限数值边界且name:min:maxmax > min - 会被验证为有限正数(拒绝负数、零、
--min-free-gb、nan)inf - 和
--residual-growth阈值会被验证为有限正数--dt-drop - 和
--bound-min会被验证为有限数(拒绝--bound-max/nan),且强制要求inf;--bound-max > --bound-min会被验证为有限正数--mass-tol - 无效输入会以退出码2和解释性消息终止程序
File Access
文件访问
- reads a single user-specified config file (JSON/YAML) and checks disk space on the volume hosting the resolved output directory
preflight_checker.py - reads a single log file specified by
runtime_monitor.py; log files are size-limited (500 MB max) and rejected before parsing if larger--log - reads a single metrics file (JSON) specified by
result_validator.py--metrics - reads a single log file specified by
failure_diagnoser.py; log files are size-limited (500 MB max) before parsing--log - No scripts write to the filesystem; all output goes to stdout
- 读取用户指定的单个配置文件(JSON/YAML),并检查解析后的输出目录所在卷的磁盘空间
preflight_checker.py - 读取
runtime_monitor.py指定的单个日志文件;日志文件有大小限制(最大500MB),超过限制会在解析前被拒绝--log - 读取
result_validator.py指定的单个指标文件(JSON)--metrics - 读取
failure_diagnoser.py指定的单个日志文件;日志文件有大小限制(最大500MB),超过限制会在解析前被拒绝--log - 所有脚本均不写入文件系统;所有输出均输出到stdout
Tool Restrictions
工具限制
- Read: Used to inspect script source, references, config files, and simulation logs
- Bash: Used to execute the four Python validation scripts (,
preflight_checker.py,runtime_monitor.py,result_validator.py) with explicit argument listsfailure_diagnoser.py - Write: Used to save validation reports; writes are scoped to the user's working directory
- Grep/Glob: Used to locate log files, config files, and search references
- 读取:用于检查脚本源码、参考资料、配置文件和模拟日志
- Bash:用于执行四个Python验证脚本(、
preflight_checker.py、runtime_monitor.py、result_validator.py),使用显式参数列表failure_diagnoser.py - 写入:用于保存验证报告;写入范围仅限于用户工作目录
- Grep/Glob:用于定位日志文件、配置文件和搜索参考资料
Safety Measures
安全措施
- No ,
eval(), or dynamic code generationexec() - All subprocess calls use explicit argument lists (no )
shell=True - uses hardcoded, pre-compiled diagnostic regex patterns;
failure_diagnoser.pyaccepts optionalruntime_monitor.py/--residual-patternoverrides that are compiled with--dt-pattern(nore.compile) and applied only to the user's own logeval - Diagnostic strings emitted in output are drawn from the skill's fixed cause/fix table, not interpolated from raw log content
- 不使用、
eval()或动态代码生成exec() - 所有子进程调用均使用显式参数列表(不使用)
shell=True - 使用硬编码的预编译诊断正则表达式;
failure_diagnoser.py接受可选的runtime_monitor.py/--residual-pattern覆盖参数,这些参数会通过--dt-pattern编译(不使用re.compile),仅应用于用户自己的日志eval - 输出中的诊断信息来自技能内置的原因/修复表,不会从原始日志内容插值生成
Limitations
局限性
- Not a real-time monitor: Scripts analyze logs after-the-fact
- Regex-based: Log parsing depends on pattern matching; may miss unusual formats
- No automatic fixes: Scripts diagnose but don't modify simulations
- 非实时监控:脚本仅在事后分析日志
- 基于正则表达式:日志解析依赖模式匹配;可能无法识别非常规格式
- 无自动修复:脚本仅诊断问题,不修改模拟配置
References
参考资料
- - Detailed checklist and criteria
references/validation_protocol.md - - Common failure signatures and regex patterns
references/log_patterns.md
- - 详细清单和标准
references/validation_protocol.md - - 常见失败特征和正则表达式
references/log_patterns.md
Version History
版本历史
- v1.2.2 (2026-06-24): Added a Verification checklist (evidence-based, tied to the four scripts' JSON outputs) and a Common pitfalls & rationalizations table to harden agent interpretation of validation verdicts.
- v1.2.0 (2026-06-23): Corrected diagnostic regexes (no false convergence/blow-up on healthy logs), direction-aware dt-collapse detection, NaN/Inf scan in runtime monitor, strict variational energy check, non-vacuous bounds/confidence, config-relative output-dir + correct-volume disk check, and implemented the documented input-validation/file-size safeguards
- v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, Windows compatibility
- v1.0.0: Initial release with 4 validation scripts
- v1.2.2(2026-06-24):添加验证清单(基于证据,与四个脚本的JSON输出绑定)和常见误区与合理化借口表,强化Agent对验证结论的解读能力。
- v1.2.0(2026-06-23):修正诊断正则表达式(不会在正常日志中误判收敛/爆炸)、方向感知的dt崩溃检测、运行时监控中的NaN/Inf扫描、严格变分能量检查、非默认边界/置信度、配置相对输出目录+正确卷磁盘检查,并实现文档中记录的输入验证/文件大小防护措施
- v1.1.0(2024-12-24):增强文档、决策指南,支持Windows系统
- v1.0.0:初始版本,包含4个验证脚本