eve-verification-plans

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Eve Verification Plans

Eve验证计划

Author agentic verification plans — markdown documents that fully specify the steps to verify an Eve-compatible app works correctly AND conforms to Eve platform conventions. Plans are actionable by humans or agents.

编写智能代理式验证计划——即完整指定验证兼容Eve的应用是否正常运行且符合Eve平台规范步骤的Markdown文档。计划可由人工或Agent执行。

When to Use

适用场景

Building verification for a new Eve-compatible app
Auditing an existing app for Eve platform conformance
After significant feature work that needs structured validation
Before handoff — proving the app works the Eve way, end to end
Onboarding a new team to understand what "correctly built on Eve" looks like

为新的兼容Eve的应用构建验证体系
审计现有应用是否符合Eve平台规范
完成重大功能开发后需要结构化验证时
交付前——证明应用完全按照Eve的方式运行
帮助新团队理解“基于Eve正确构建”的标准

Quick Start

快速开始

bash

undefined

bash

undefined

1. Create the verification directory in your project

1. 在项目中创建验证目录

mkdir -p e2e-verification/00-smoke

2. Copy the smoke template

2. 复制冒烟测试模板

cp templates/00-smoke-test-plan.md e2e-verification/00-smoke/

3. Customize for your app (endpoints, services, agents)

3. 根据你的应用自定义（端点、服务、Agent）

4. Run it

4. 运行计划

undefined

undefined

The Six Verification Dimensions

六大验证维度

Every Eve app has up to six dimensions to verify. Cover all that apply.

Dimension	Tool	When	Conformance Check
Platform conformance	Eve CLI + manifest inspection	Always	CLI parity, manifest conventions, secrets model
Service layer	Eve CLI + REST API	Always	Every endpoint reachable via CLI; no kubectl needed
Input / ingestion	Repo fixtures + upload commands	When app accepts files	Deterministic fixtures, real parsing flows
Data layer	Eve CLI + DB migrations	When app has DB	Migrations via Eve pipeline, not manual SQL
UI / visual	agent-browser or Playwright	When frontend exists	SSO login, dark/light mode, agent-testable
Agent behavior	`eve job follow` + optimization	When app has agents	Efficient completion, no blind alleys

每个Eve应用最多有六个需要验证的维度，覆盖所有适用项即可。

维度	工具	适用时机	合规性检查内容
平台合规性	Eve CLI + 清单检查	始终需要	CLI一致性、清单规范、密钥模型
服务层	Eve CLI + REST API	始终需要	所有端点可通过CLI访问；无需kubectl
输入/导入	仓库fixtures + 上传命令	应用支持文件上传时	确定性fixtures、真实解析流程
数据层	Eve CLI + 数据库迁移	应用包含数据库时	通过Eve流水线执行迁移，而非手动SQL
UI/视觉	agent-browser 或 Playwright	存在前端时	SSO登录、明暗模式、可Agent测试
Agent行为	`eve job follow` + 优化工具	应用包含Agent时	高效完成任务、无无效操作

Verification Plan Format

验证计划格式

Each scenario is a self-contained markdown document:

markdown

undefined

每个场景都是独立的Markdown文档：

markdown

undefined

Scenario NN: <Name>

场景NN: <名称>

Time: ~Nm Environment: staging | local | both Parallel Safe: Yes/No Requires: LLM | Browser | None

<one-paragraph description>

耗时: ~Nm 环境: staging | local | both 支持并行: 是/否 依赖: LLM | Browser | 无

<一段描述>

Prerequisites

前置条件

What must be true before running
Required secrets, auth, prior scenarios

运行前必须满足的条件
所需密钥、认证、前置场景

Fixtures

File paths used by this scenario
Provenance or generation command for each
Why these files are representative

本场景使用的文件路径
每个文件的来源或生成命令
这些文件具有代表性的原因

Setup

准备步骤

```bash

Environment detection + auth

环境检测 + 认证

Project/org setup

项目/组织配置

Fixture validation or generation

Fixture验证或生成

```

Phases

执行阶段

Phase 1: <Name>

阶段1: <名称>

```bash

Commands to execute

执行命令

```

Expected:

Bullet list of assertions
Each assertion is pass/fail verifiable

```

预期结果:

断言列表
每个断言均可验证通过/失败

Phase 2: ...

阶段2: ...

Success Criteria

成功标准

Checkboxes for every pass/fail assertion
Grouped by phase

每个可验证断言的复选框
按阶段分组

Debugging

调试指南

Symptom	Diagnostic	Fix
...	...	...

症状	诊断方法	修复方案
...	...	...

Cleanup

清理步骤

```bash

Teardown commands

销毁资源命令

```

undefined

```

undefined

Format Rules

格式规则

Environment-aware: Every plan starts with environment detection —
```
EVE_API_URL
```
determines cloud vs local
Self-contained: No assumed state beyond documented prerequisites
Fixture-explicit: Every uploaded/imported artifact is checked in or generated from documented commands
Phased: Break into phases that can run independently (parallel where safe)
Assertion-driven: Every step has explicit
```
Expected:
```
with pass/fail criteria
Debuggable: Troubleshooting section with symptom → diagnostic → fix

环境感知: 每个计划都从环境检测开始——
```
EVE_API_URL
```
决定云端还是本地环境
独立完整: 除文档记录的前置条件外，无其他假设状态
Fixture明确: 所有上传/导入的工件已入库或通过文档命令生成
分阶段: 拆分为可独立运行的阶段（支持并行的场景可并行执行）
断言驱动: 每个步骤都有明确的
```
预期结果:
```
及通过/失败标准
可调试: 包含从症状→诊断→修复的故障排除部分

Platform Conformance Verification

平台合规性验证

Before testing functionality, verify the app is built the Eve way. Every verification suite starts with this checklist:

.eve/manifest.yaml

exists and passes

eve manifest validate --project <project_id> --path .eve/manifest.yaml

Manifest follows current conventions (
```
name
```
preferred over legacy
```
project
```
)
All services have health endpoints reachable via Eve ingress
CLI can interact with every API endpoint (no "UI-only" functionality)
Secrets managed via
```
eve secrets
```
, not hardcoded or env-file-based
DB migrations run as pipeline steps, not manual scripts
Agents (if any) defined in
```
agents.yaml
```
with harness profiles
Pipelines (if any) defined in manifest and runnable via
```
eve pipeline run
```
Frontend (if any) authenticates via Eve SSO, not custom auth
Upload/import flows (if any) have deterministic fixtures checked in or generated locally

See

references/eve-conformance-checks.md

for the full checklist with rationale.

测试功能前，需验证应用是否按照Eve的方式构建。每个验证套件都应从以下清单开始：

.eve/manifest.yaml

存在且通过

eve manifest validate --project <project_id> --path .eve/manifest.yaml

验证

清单遵循当前规范（优先使用
```
name
```
而非旧版
```
project
```
）
所有服务的健康端点可通过Eve入口访问
CLI可与所有API端点交互（无“仅UI”功能）
密钥通过
```
eve secrets
```
管理，而非硬编码或基于环境文件
数据库迁移作为流水线步骤运行，而非手动脚本
Agent（如有）在
```
agents.yaml
```
中定义并配置测试配置文件
流水线（如有）在清单中定义且可通过
```
eve pipeline run
```
运行
前端（如有）通过Eve SSO认证，而非自定义认证
上传/导入流程（如有）具有确定性fixtures，已入库或可本地生成

完整清单及理由请参考

references/eve-conformance-checks.md

。

Service Layer Verification

服务层验证

Test every API surface CLI-first:

bash

TOKEN=$(eve auth token --raw)

优先通过CLI测试所有API接口：

bash

TOKEN=$(eve auth token --raw)

Health check (always first)

健康检查（始终优先执行）

curl -sf "${APP_SCHEME}://api.${APP_DOMAIN}/health" | jq '.'

App API via auth token

使用认证令牌访问应用API

curl -sf -H "Authorization: Bearer $TOKEN"
"${APP_SCHEME}://api.${APP_DOMAIN}/endpoint" | jq '.field'


**CLI parity assertion**: For every `curl` call in a test plan, ask: "Can this also be done via a CLI command?" If not, file an issue — don't accept it.

Pattern:
1. **Eve CLI** commands where they exist (deploy, env, job, secrets)
2. **App CLI** if the app follows `eve-app-cli` patterns
3. **REST API** via `curl` with auth tokens for custom endpoints
4. **Auth**: Mint tokens via `eve auth mint` or `eve auth token --raw`

curl -sf -H "Authorization: Bearer $TOKEN"
"${APP_SCHEME}://api.${APP_DOMAIN}/endpoint" | jq '.field'


**CLI一致性断言**: 测试计划中的每个`curl`调用都需确认：“是否也可通过CLI命令完成？”若不可行，需提交问题——不得接受这种情况。

优先级:
1. **Eve CLI** 命令（如有：deploy、env、job、secrets）
2. 遵循`eve-app-cli`模式的**应用CLI**
3. 通过带认证令牌的`curl`访问**REST API**自定义端点
4. **认证**: 通过`eve auth mint`或`eve auth token --raw`生成令牌

Input / Ingestion Verification

输入/导入验证

When the app accepts uploads, imports, or document bundles, verification must include a fixture plan.

当应用支持上传、导入或文档包时，验证必须包含fixture计划。

Fixture Selection Order

Fixture选择顺序

Reuse existing repo fixtures if they match accepted file types and are deterministic
Manufacture synthetic fixtures locally with committed scripts
Source small public-domain fixtures only when local manufacture would reduce realism

若现有仓库fixtures匹配支持的文件类型且具有确定性，优先复用
通过已提交的脚本在本地生成合成fixtures
仅当本地生成会降低真实性时，才使用小型公共领域fixtures

Fixture Matrix

Fixture矩阵

Minimal valid — smallest acceptable file that exercises the happy path
Typical real-world — representative document/media/import file
Boundary / invalid — wrong type, malformed structure, or size edge
Cross-format — if the app accepts multiple types (PDF + Markdown + CSV), verify each

最小有效文件 —— 可执行正常流程的最小可接受文件
典型真实文件 —— 具有代表性的文档/媒体/导入文件
边界/无效文件 —— 错误类型、格式错误或大小边缘情况
跨格式文件 —— 若应用支持多种类型（PDF + Markdown + CSV），需逐一验证

What to Check

检查内容

File accepted through the real app surface (CLI, REST endpoint, or browser upload)
MIME/type detection and metadata are correct
Storage/persistence path is correct
Downstream processing produces expected results
Error handling is explicit for rejected or malformed fixtures

Rule: If a plan says "upload a sample PDF", it must include an actual file path or generator step. "Find a PDF online" is not acceptable.

See

references/fixture-patterns.md

for detailed guidance.

文件可通过真实应用界面（CLI、REST端点或浏览器上传）接受
MIME/类型检测及元数据正确
存储/持久化路径正确
下游处理产生预期结果
对被拒绝或格式错误的fixtures有明确的错误处理

规则: 若计划中提到“上传示例PDF”，必须包含实际文件路径或生成步骤。“在线找一个PDF”是不被接受的。

详细指南请参考

references/fixture-patterns.md

。

UI Verification

UI验证

When the app has a frontend, verify visual quality and interaction flows.

当应用包含前端时，需验证视觉质量及交互流程。

SSO Token Injection

SSO令牌注入

bash

undefined

bash

undefined

Mint an SSO token via CLI

通过CLI生成SSO令牌

SSO_TOKEN=$(eve auth mint --email user@example.com --org $ORG_ID --format sso-jwt)

Use agent-browser with the token

使用agent-browser携带令牌访问

agent-browser --session verify open "${APP_URL}/auth/callback?token=${SSO_TOKEN}" agent-browser --session verify wait --url "**/dashboard" agent-browser --session verify screenshot ./e2e-verification/artifacts/dashboard.png

undefined

undefined

What to Check

检查内容

Pages render without console errors
Dark mode and light mode both work (screenshot both)
Key user flows complete (login → dashboard → action → result)
Responsive layout at standard breakpoints
Forms submit correctly and validation fires

See

references/ui-verification-patterns.md

for tool choice guidance and patterns.

页面渲染无控制台错误
明暗模式均可正常工作（需分别截图）
关键用户流程可完成（登录→仪表盘→操作→结果）
在标准断点下响应式布局正常
表单提交正确且验证规则生效

工具选择指南及模式请参考

references/ui-verification-patterns.md

。

Agent Verification

Agent验证

When the app includes Eve agents, verification extends to behavior quality.

Create a job that exercises the agent's primary workflow
Follow execution:
```
eve job follow <job-id>
```
Check receipt:
```
eve job receipt <job-id>
```
(tokens, cost, duration)
Apply
```
eve-agent-optimisation
```
diagnostic workflow
Record baseline metrics in the test plan for regression detection

当应用包含Eve Agent时，验证需扩展到行为质量。

创建一个可执行Agent主要工作流的任务
跟踪执行过程:
```
eve job follow <job-id>
```
查看执行报告:
```
eve job receipt <job-id>
```
（令牌使用量、成本、时长）
应用
```
eve-agent-optimisation
```
诊断工作流
在测试计划中记录基线指标以检测回归

What to Check

检查内容

Agent completes its task (correct output)
Token usage within acceptable bounds
No unnecessary tool calls or blind alleys
Error cases handled gracefully (bad input, missing secrets)
Multi-agent coordination works (jobs complete in dependency order)

See

references/agent-verification-patterns.md

for integration with optimization.

Agent完成任务（输出正确）
令牌使用量在可接受范围内
无不必要的工具调用或无效操作
错误情况处理优雅（输入错误、密钥缺失）
多Agent协调正常（任务按依赖顺序完成）

与优化工具的集成请参考

references/agent-verification-patterns.md

。

Deploy Cycle Patterns

部署周期模式

Verification often reveals issues. The fix/deploy loop:

验证通常会发现问题，修复/部署循环如下：

Cloud (Staging) — Default

云端（Staging）——默认

discover bug → fix code → commit → tag release-v* → push tag →
  wait for CI to publish all seven service images →
  instance owner bumps the pinned platform version and deploys from the instance repo →
  re-run failed scenario

A source

release-v*

tag publishes artifacts only. It must not dispatch to or deploy a hosted instance.

发现bug → 修复代码 → 提交 → 打release-v*标签 → 推送标签 →
  等待CI发布所有七个服务镜像 →
  实例所有者更新固定平台版本并从实例仓库部署 →
  重新运行失败场景

release-v*

源标签仅发布工件，不得触发或部署托管实例。

Local (k3d)

本地（k3d）

discover bug → fix code → pnpm build →
  ./bin/eh k8s-image push → ./bin/eh k8s deploy →
  re-run failed scenario

See

references/deploy-cycle-patterns.md

for environment detection and wait patterns.

发现bug → 修复代码 → pnpm build →
  ./bin/eh k8s-image push → ./bin/eh k8s deploy →
  重新运行失败场景

环境检测及等待模式请参考

references/deploy-cycle-patterns.md

。

Scenario Discovery

场景发现

How to identify what scenarios to create for a given app:

Read the manifest — every service is a verification target
Read agents config — every agent needs a behavioral test
Read the API spec — every endpoint group is a potential scenario
Check upload/import surfaces — every accepted file class needs fixtures
Check pipelines — build/deploy/workflow pipelines need end-to-end verification
Check the UI — every page/route needs visual verification
Check integrations — webhooks, chat gateways, external APIs

如何为特定应用确定需要创建的场景：

阅读清单 —— 每个服务都是验证目标
阅读Agent配置 —— 每个Agent都需要行为测试
阅读API规范 —— 每个端点组都是潜在场景
检查上传/导入界面 —— 每个支持的文件类型都需要fixtures
检查流水线 —— 构建/部署/工作流流水线需要端到端验证
检查UI —— 每个页面/路由都需要视觉验证
检查集成 —— Webhook、聊天网关、外部API

Minimum Scenario Set

最小场景集

Scenario	Required	What It Covers
`00-smoke`	Always	Health, auth, connectivity + Eve conformance checklist
`01-deploy`	Always	Build, release, deploy via pipeline, verify endpoints
`02-core-flow`	Always	Primary user journey end-to-end (CLI + API)
`03-ui-visual`	If frontend	Screenshot verification, SSO login, dark/light mode
`04-input-ingestion`	If uploads/imports	Fixture upload, parsing, storage, error handling
`05-data-layer`	If database	Migrations via pipeline, schema correct, data integrity
`06-agent-execution`	If agents	Agents complete primary tasks correctly
`07-agent-optimization`	If agents	Baseline metrics + optimization pass
`08-pipeline-flows`	If pipelines	Each pipeline runs end-to-end, steps succeed in order

场景	是否必需	覆盖内容
`00-smoke`	始终必需	健康检查、认证、连通性 + Eve合规性清单
`01-deploy`	始终必需	构建、发布、流水线部署、端点验证
`02-core-flow`	始终必需	主要用户旅程端到端验证（CLI + API）
`03-ui-visual`	存在前端时必需	截图验证、SSO登录、明暗模式
`04-input-ingestion`	支持上传/导入时必需	Fixture上传、解析、存储、错误处理
`05-data-layer`	存在数据库时必需	流水线迁移、 schema正确、数据完整性
`06-agent-execution`	存在Agent时必需	Agent正确完成主要任务
`07-agent-optimization`	存在Agent时必需	基线指标 + 优化验证
`08-pipeline-flows`	存在流水线时必需	每个流水线端到端运行，步骤按顺序成功

Running Verification Plans

运行验证计划

Sequential Execution

顺序执行

bash

undefined

bash

undefined

Run scenarios in order

按顺序运行场景

for plan in e2e-verification/*/; do echo "=== Running: $plan ==="

Agent reads and executes the plan document

done

undefined

for plan in e2e-verification/*/; do echo "=== 运行中: $plan ==="

Agent读取并执行计划文档

done

undefined

Parallel Execution

并行执行

Scenarios marked

Parallel Safe: Yes

can run concurrently. Typically:

```
00-smoke
```
runs first (validates prerequisites)
```
01-deploy
```
through
```
03-ui-visual
```
can parallelize
Agent scenarios (
```
06-07
```
) depend on deploy completing

标记为

支持并行: 是

的场景可同时运行。通常：

```
00-smoke
```
先运行（验证前置条件）
```
01-deploy
```
至
```
03-ui-visual
```
可并行执行
Agent场景（
```
06-07
```
）依赖部署完成

CI Integration

CI集成

bash

undefined

bash

undefined

In CI, set environment and run

在CI中设置环境并运行

export EVE_API_URL=https://api.eve.example.com eve auth login --email $CI_EMAIL --ssh-key $CI_SSH_KEY

Run all scenarios, collect artifacts

运行所有场景，收集产物

mkdir -p e2e-verification/artifacts

Agent executes each plan, screenshots/logs go to artifacts/

Agent执行每个计划，截图/日志存入artifacts/

undefined

undefined

File Structure

文件结构

<project-root>/
  e2e-verification/
    README.md                    # Index of all scenarios
    00-smoke/
      00-smoke-test-plan.md
    01-deploy/
      01-deploy-test-plan.md
    02-core-flow/
      02-core-flow-test-plan.md
      fixtures/
        README.md
        test-data.json
    03-ui-visual/
      03-ui-visual-test-plan.md
    04-input-ingestion/
      04-input-ingestion-test-plan.md
      fixtures/
        README.md
        sample-document.pdf
        sample-import.csv
        scripts/
          make-fixtures.sh
    artifacts/                   # Generated during runs (gitignored)

Numbering:

NN-kebab-name/

directories with matching

NN-kebab-name-test-plan.md

. Numbers imply execution order. Gaps are fine — they allow insertion without renaming.

Fixtures: Every scenario that depends on uploaded or imported inputs gets a sibling

fixtures/

directory.

fixtures/README.md

records provenance, generation steps, and sensitivity notes.

<项目根目录>/
  e2e-verification/
    README.md                    # 所有场景的索引
    00-smoke/
      00-smoke-test-plan.md
    01-deploy/
      01-deploy-test-plan.md
    02-core-flow/
      02-core-flow-test-plan.md
      fixtures/
        README.md
        test-data.json
    03-ui-visual/
      03-ui-visual-test-plan.md
    04-input-ingestion/
      04-input-ingestion-test-plan.md
      fixtures/
        README.md
        sample-document.pdf
        sample-import.csv
        scripts/
          make-fixtures.sh
    artifacts/                   # 运行时生成（已加入git忽略）

编号规则:

NN-kebab-name/

目录下对应

NN-kebab-name-test-plan.md

文件。编号表示执行顺序，允许存在间隔——便于插入新场景而无需重命名。

Fixtures: 所有依赖上传或导入输入的场景都应有同级

fixtures/

目录。

fixtures/README.md

记录来源、生成步骤及敏感说明。

References

参考文档

```
references/test-plan-format.md
```
— Full format specification with annotated examples
```
references/eve-conformance-checks.md
```
— The Eve way: what to verify and why
```
references/fixture-patterns.md
```
— Fixture sourcing, manufacture, and documentation
```
references/deploy-cycle-patterns.md
```
— Fix/deploy loop for cloud and local
```
references/ui-verification-patterns.md
```
— Browser automation and SSO auth patterns
```
references/agent-verification-patterns.md
```
— Agent testing and optimization integration

```
references/test-plan-format.md
```
—— 完整格式规范及带注释示例
```
references/eve-conformance-checks.md
```
—— Eve标准：验证内容及理由
```
references/fixture-patterns.md
```
—— Fixture获取、生成及文档规范
```
references/deploy-cycle-patterns.md
```
—— 云端及本地环境的修复/部署循环
```
references/ui-verification-patterns.md
```
—— 浏览器自动化及SSO认证模式

references/agent-verification-patterns.md

—— Agent测试及优化集成

Templates

模板

```
templates/00-smoke-test-plan.md
```
— Starter smoke + conformance template
```
templates/scenario-test-plan.md
```
— General scenario skeleton
```
templates/upload-ingest-test-plan.md
```
— Input-heavy scenario with fixture matrix

```
templates/00-smoke-test-plan.md
```
—— 冒烟测试+合规性验证起始模板
```
templates/scenario-test-plan.md
```
—— 通用场景框架
```
templates/upload-ingest-test-plan.md
```
—— 含fixture矩阵的输入型场景模板

Skill	Relationship
`eve-agent-optimisation`	Called from agent verification scenarios
`eve-web-ui-testing-agent-browser`	UI verification tool
`eve-deploy-debugging`	Deploy cycle troubleshooting
`eve-cli-primitives`	CLI commands used in service-layer tests
`eve-manifest-authoring`	Manifest conventions that conformance checks validate
`eve-app-cli`	App CLI patterns — verification asserts CLI parity
`eve-pipelines-workflows`	Pipeline conventions tested by pipeline scenarios
`eve-auth-and-secrets`	Auth + secrets model validated by conformance checks
`eve-troubleshooting`	General debugging when verification fails
`eve-read-eve-docs`	Reference source for current CLI/manifest/auth behavior
`eve-agent-native-design`	Design principles encoded in conformance checks

技能	关系
`eve-agent-optimisation`	从Agent验证场景调用
`eve-web-ui-testing-agent-browser`	UI验证工具
`eve-deploy-debugging`	部署周期故障排查
`eve-cli-primitives`	服务层测试使用的CLI命令
`eve-manifest-authoring`	合规性检查验证的清单规范
`eve-app-cli`	应用CLI模式——验证断言CLI一致性
`eve-pipelines-workflows`	流水线场景测试的流水线规范
`eve-auth-and-secrets`	合规性检查验证的认证+密钥模型
`eve-troubleshooting`	验证失败时的通用调试技能
`eve-read-eve-docs`	当前CLI/清单/认证行为的参考来源
`eve-agent-native-design`	合规性检查中编码的设计原则