Huawei Cloud Skill Tester — Three-Track Eight-Phase E2E Testing Pipeline

Independent, repeatable Huawei Cloud Skill testing framework. Does not depend on skill-creator; can test any existing Huawei Cloud Skill. Focuses on real-environment functional testing and multi-skill orchestration scenarios.

Overview

Core Design Principles

1. Chain Verification — Before each Phase, check that the previous phase's JSON exists; if missing, refuse to execute
2. Agent-proof — Write operations require user confirmation for each item; automatic gate bypassing is not allowed
3. Three-Track Layering — Clear gates between Tiers; Tier 1 must be completed before entering Tier 2
4. Batch Repeatable — Supports

   --skills "skill-a,skill-b"

   or

   --all-installed

5. Fallback Strategy — When only 1 skill, Phase 5/6 automatically downgrade to single-skill lifecycle testing. Sibling auto-scan is ON by default (Phase 5/6 自动从被测 skill 同目录找其他 huawei-cloud-* skill 做编排组合测试). Use

   --no-siblings

   to opt-out.
6. Standardized JSON Output — All phases output in a unified schema; Phase 7 merges into a single report
7. Real-Environment First — All Tier 2 orchestrations execute against real Huawei Cloud; no mocks or simulations

Data Flow Diagram

User Input (--skills or --all-installed)
    │
    ├── Tier 1 ──── Iterate over each skill ────
    │   Phase 0 → 1 → 2 → 3 → 4
    │   (phase-N-summary.json chain validation)
    │
    ├── Tier 2 ──── Integration ────
    │   Phase 5 (orchestration scenario derivation + real-environment execution)
    │   Phase 6 (e2e full-flow: create→query→update→delete lifecycle)
    │   Only 1 skill → Downgrade single-skill closed loop
    │
    └── Tier 3 ──── Final ────
        Phase 7 (merge phase-0~6 JSON into consolidated report)

Prerequisites

1. hcloud CLI installed and authenticated (for Tier 2 CLI mode testing) — Reference: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html
2. Python 3.8+ +

   huaweicloudsdk

   packages (for SDK mode testing) — SDK Reference: https://console.huaweicloud.com/apiexplorer/#/sdkcenter
3. Huawei Cloud AK/SK — 自动扫描所有以

   HUAWEI

   /

   HW

   /

   HWC

   开头的环境变量（键名含 AK/SK 标记的键值对）。If missing, the framework emits the env-var setup template to stderr and exits 77. NEVER ask the user to type AK/SK in chat; user must set env vars in their shell profile out-of-band and re-run.（完整协议见

   references/agent-protocol.md

   ）
4. Target Skill must be under

   $SKILL_INSTALL_DIR/

   (auto-detected:

   ~/.agents/skills/

   →

   ~/.hermes/skills/

   → default) or a user-specified path
5. jq command (all JSON processing depends on it)
6. API Reference: https://console.huaweicloud.com/apiexplorer/#/openapi

Workflow — Three-Track Eight-Phase

Tier 1: Single-Skill Unit Testing
   Phase 0: Installation Verification (install/uninstall/reinstall)
   Phase 1: Feature Extraction (metadata + commands + resource types + doc checks)
   Phase 2: Technical Research (CLI→SDK→API three-level availability)
   Phase 3: Test Case Generation (functional TC-F + boundary + negative + API TC-A)
   Phase 4: Real-Environment Execution (read-only automatic + write ops require confirmation)

Tier 2: Integration Testing — Real-Environment Orchestration
   Phase 5: Multi-Skill Orchestration (conflict scan / scenario derivation / self-check)
   Phase 6: End-to-End Flow (resource lifecycle: create→query→update→delete)

Tier 3: Final Report
   Phase 7: Consolidated Report (merge phase-0~6 JSON, verdict + issues + reasons)

各 Phase 的完整实现规范（步骤、判定标准、JSON 字段）见

references/phase-details.md

。

Phases at a Glance

详细步骤、链式验证规则、JSON schema 见

references/phase-details.md

与

references/output-schema-spec.md

。

KooCLI Command Format Standard

bash

hcloud

hcloud

hcloud <Service> <Operation> --cli-region={region} [--param1=value1 ...]

--cli-region={region}

--param=value

--limit=1

hcloud obs

hcloud obs <command> [args...] [options...]

Core Commands

Full Pipeline Run

# Specify skills
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage"

# Specify multiple skills (comma-separated)
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage, huawei-cloud-ecs-manage"

# Scan all installed
bash scripts/run-test-pipeline.sh --all-installed

# Start from a specific phase (recovery scenarios only)
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --phase 4

# Fresh mode
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --fresh

Single Phase Run (Debug)

bash scripts/tier1/phase-0-install-check.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier1/phase-1-skill-analysis.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier2/phase-5-orchestration.sh --skills "skill-a, skill-b"
bash scripts/tier3/phase-7-final-report.sh --skills "skill-a, skill-b"

Run Multi-Skill Orchestration

# Derive and execute orchestration scenarios for 3 skills
bash scripts/tier2/phase-5-orchestration.sh --skills "ecs-manage, vpc-manage, eip-manage"

# Run E2E lifecycle test for a single skill
bash scripts/tier2/phase-6-full-flow.sh --skill "huawei-cloud-rds-intelligent-service"

Parameters

--skills

"bss-voucher-manage, ecs-manage"

--all-installed

huawei-cloud-*

$SKILL_INSTALL_DIR/huawei-cloud/

--phase

--phase 0

--fresh

phase-*.json

phases/archive/<timestamp>/

--output

--output ./test-reports

--skill-path

find_skill_path

--skill-path ./skills

--no-siblings

--no-siblings

--sibling-limit <N>

0

--no-siblings

--sibling-limit 3

Environment Variables (Advanced)

SKILL_INSTALL_DIR

~/.agents/skills

~/.hermes/skills

~/.agents/skills

SKILL_PATH_HERMES

SKILL_INSTALL_DIR

SKILL_INSTALL_CMD

hermes skills

""

ALLOW_WRITES

0

1

HUAWEI_REGION

cn-north-4

HUAWEI_ACCESS_KEY

HUAWEI_SECRET_KEY

HUAWEI*

HW*

HWC*

References

核心文档（必读）

• references/architecture.md

  — Three-track eight-phase architecture diagram (Mermaid)

• references/output-schema-spec.md

  — Complete JSON field specification for each phase (including Phase 7 final report schema)

• references/phase-transition-rules.md

  — Phase transition / fallback / skip rules

• references/acceptance-criteria.md

  — Quality gates + 17-item report acceptance checklist

• references/verification-method.md

  — How to manually verify each phase (PowerShell + Git Bash)

• references/phase-details.md

  — 各 Phase 完整实现规范（步骤、判定标准、JSON 字段）

• references/agent-protocol.md

  — 凭证请求协议（AK/SK 缺失时的完整处理流程）

配套参考

• references/cli-installation-guide.md

  — How to install and configure hcloud CLI (prerequisite for Phase 2/4)

• references/iam-policies.md

  — Minimum IAM permissions required to run the tester

模板（JSON Schema）

• templates/phase-report-schema.json

  — JSON Schema for

  phase-N-summary.json

  (N=0..6)

• templates/test-case-schema.json

  — JSON Schema for individual test cases (

  TC-F-*

  /

  TC-A-*

  /

  OF-*

  /

  FF-*

  )

• templates/scenario-template.json

  — Reference example for Phase 6 multi-skill scenario derivation

Output Format

<skill-name>-test-files/

skills/
├── huawei-cloud-rds-query/                  ← skill source (untouched)
└── huawei-cloud-rds-query-test-files/       ← test artifacts (created on first run, kept across runs)
    ├── phases/
    │   ├── phase-0-summary.json
    │   ├── phase-1-summary.json
    │   ├── ...
    │   └── phase-7-summary.json
    └── reports/
        ├── report-20260724-152309/          ← one subdir per run (timestamped)
        │   ├── test-report.json
        │   └── test-report.md
        ├── report-20260725-090000/
        │   ├── test-report.json
        │   └── test-report.md
        └── ...

• Phase 0~6 output

  phase-N-summary.json

  to

  <skill-name>-test-files/phases/

• Phase 7 merges them into

  <skill-name>-test-files/reports/report-<timestamp>/test-report.{json,md}

• Test artifacts are preserved across runs (no auto-cleanup, ever).

  --fresh

  does NOT delete — it archives old

  phases/*.json

  to

  phases/archive/<timestamp>/

  so the chain check resets while history stays intact.

  reports/

  is always kept.
• The test-installed copy of the skill in

  $SKILL_INSTALL_DIR/

  is still uninstalled on exit, so the next run sees a clean install state.

references/output-schema-spec.md

Best Practices

• Complete Tier 1 before entering Tier 2 to ensure skills are individually functional before orchestration
• Confirm write operations one by one in Phase 4 and Phase 5/6; do not batch-confirm to avoid misoperations
• With only 1 skill, Phase 5/6 automatically downgrade to single-skill closed loop; no need to manually skip
• When using

  --fresh

  to reset and rerun, confirm there are no uncleaned test resources
• Review orchestration scenarios before execution to ensure resource dependency order is correct

Notes

• Three-track eight-phase strictly follows sequential order; chain verification prevents skipping
• API endpoints are strictly prohibited from being inferred; only obtain from SDK

  _http_info

  or API Explorer
• Credentials are read from environment variables; hardcoding is prohibited
• If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user verbatim and instruct them to set env vars out-of-band (in their shell / PowerShell $PROFILE). The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.（完整协议见

  references/agent-protocol.md

  ）
• Resources created during testing must be tracked; if any are left behind, output manual cleanup instructions
• Orchestration scenarios are auto-derived; user should review and confirm before execution
• Write operations in orchestration scenarios require per-step user confirmation

Edge Cases

export HUAWEI_ACCESS_KEY=<your-access-key>

$env:HUAWEI_ACCESS_KEY=<your-access-key>

--fresh

--phase

--resume

downgraded_self_check

downgraded_single_skill_flow

ALLOW_WRITES=1

status=skip

status=warn

manual_test_items

data_flow_tests

phase-6-summary.json

cleanup.manual_required

Agent Protocol — Credential Request

__HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__

77

1. 检测到 exit 77 或 sentinel → 暂停流水线,不得跳过 Phase 5/7
2. 将 stderr 中的 env-var 设置模板原样输出给用户,引导用户在 shell profile / PowerShell $PROFILE 中带外配置(占位符

   <your-access-key-id>

   /

   <your-secret-access-key>

   )
3. 禁止在对话中索要 AK/SK 明文(ask_user / read -p / 剪贴板回传 / 读取 ~/.hcloud/config.json 等路径)
4. 用户带外设置后,重跑失败 phase;用户拒绝则明确标记"live phases skipped — no credentials",不得标记 pass

完整协议(6 步响应流程、直接终端模式、示例行为)见

references/agent-protocol.md

。

Design Principles

• Chain Verification — Each Phase checks the previous phase's JSON to prevent skipping
• Agent-proof — Write operations must be confirmed by the user; fake confirmations are not allowed
• Data-Driven — All phases output in JSON format; Phase 7 merges
• Batch Repeatable — The same set of skills can be tested repeatedly; --fresh resets
• Real-Environment First — All orchestrations and E2E flows execute against real Huawei Cloud; no mocks
• Degrade Without Losing Value — Single skill does not run empty orchestration phases; degrades to meaningful single-skill lifecycle tests
• Resource Safety — Resources created during testing must be tracked; if any remain, output clear manual cleanup instructions
• Credentials Mandatory — If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user and instruct them to set env vars out-of-band. The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.