Back to Details

omni-gateway-benchmark

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Run Gateway Benchmark

运行网关基准测试

Overview

概述

This skill drives the benchmark harness in 
scripts/
to measure Flex Gateway throughput, latency, and resource use under configurable API counts and policies. It provisions a dedicated EKS cluster, deploys Flex + an upstream echo service, runs a k6 
TestRun
, exports Grafana dashboards as PNGs, and emits a self-contained Markdown report per run.
Detailed architecture, dashboards, idempotency design, and design rationale are documented in 
references/ARCHITECTURE.md
. Read it before extending the harness; this skill assumes it as background.
mermaid
flowchart TD
    Start{What do you want to do?}

    Start -->|First-time setup| FT[Prerequisites + Registration + Push Upstream]
    Start -->|Provision infra| UP["make up — EKS cluster"]
    Start -->|Deploy + run a single benchmark| RUN["make benchmark or make run"]
    Start -->|Re-run with new scenario knobs| KNOB[Edit env + make run]
    Start -->|Retrieve results| REP[reports/RUN_ID/]
    Start -->|Reset between sessions| CLEAN["make clean-deployment"]
    Start -->|Stop AWS billing| DOWN["make down — destructive"]
此 skill 驱动 
scripts/
中的基准测试工具，在可配置的 API 数量和策略下测量 Flex Gateway 的吞吐量、延迟和资源使用情况。它会置备专用 EKS 集群，部署 Flex 和上游回显服务，运行 k6 
TestRun
，将 Grafana 仪表板导出为 PNG，并为每次运行生成一份自包含的 Markdown 报告。
详细架构、仪表板、幂等性设计和设计理由记录在 
references/ARCHITECTURE.md
中。在扩展此工具之前请先阅读该文档；本 skill 将其作为背景知识。
mermaid
flowchart TD
    Start{What do you want to do?}

    Start -->|First-time setup| FT[Prerequisites + Registration + Push Upstream]
    Start -->|Provision infra| UP["make up — EKS cluster"]
    Start -->|Deploy + run a single benchmark| RUN["make benchmark or make run"]
    Start -->|Re-run with new scenario knobs| KNOB[Edit env + make run]
    Start -->|Retrieve results| REP[reports/RUN_ID/]
    Start -->|Reset between sessions| CLEAN["make clean-deployment"]
    Start -->|Stop AWS billing| DOWN["make down — destructive"]

Safety and Cost Guardrails

安全与成本护栏

The harness provisions billable AWS infrastructure (EKS control plane, NAT gateway, EBS volumes, c6i.2xlarge nodes, ECR). Two operations are destructive and must be confirmed with the user before executing — never run them autonomously based on inferred intent:
OperationEffectConfirm before running
make preflight
Read-only checks on CLIs, docker daemon, AWS auth, 
.env
, registration, and connectivity; reports remediation per gap		No — safe; never mutates state
make prepare-registration
Generates 
.run/registration/registration.yaml
and, when 
POLICIES
includes 
client-id-enforcement
, prompts for 
CLIENT_ID
/ 
CLIENT_SECRET
and writes them to 
.env
(keeps existing values unless 
FORCE=1
)		Confirm before re-running with 
FORCE=1
(rotates gateway identity, forces a Flex redeploy)
make up
Creates VPC + EKS + ECR; starts AWS billingYes — confirm region, cluster name, and that costs are expected
make down
Destroys all infra including the cluster, ECR repository contents, and Terraform state referencesYes — confirm the user has retrieved any reports they want to keep
make clean-deployment
Removes Flex + upstream + k6 TestRuns, keeps clusterSafe to run between runs; preserves billing-relevant infra
If 
TEARDOWN=1
is set in 
.env
, 
make benchmark
runs 
make down
automatically at the end of the pipeline. Verify this is the user's intent before invoking 
make benchmark
with that flag set.
此工具会置备需要付费的 AWS 基础设施（EKS control plane、NAT gateway、EBS volumes、c6i.2xlarge nodes、ECR）。其中两项操作具有破坏性，执行前必须向用户确认——绝不可根据推测出的意图自主执行：
操作效果执行前是否确认
make preflight
对 CLI、docker daemon、AWS auth、
.env
、registration 和连接性进行只读检查；针对每项缺失项报告修复方式		否——安全操作；绝不改变状态
make prepare-registration
生成 
.run/registration/registration.yaml
；当 
POLICIES
包含 
client-id-enforcement
时，提示输入 
CLIENT_ID
/ 
CLIENT_SECRET
并将其写入 
.env
（除非使用 
FORCE=1
，否则保留现有值）		使用 
FORCE=1
重新运行前需要确认（轮换网关身份并强制重新部署 Flex）
make up
创建 VPC + EKS + ECR；开始产生 AWS 费用是——确认区域、集群名称以及费用符合预期
make down
销毁所有基础设施，包括集群、ECR repository 内容和 Terraform state 引用是——确认用户已获取需要保留的所有报告
make clean-deployment
移除 Flex + upstream + k6 TestRuns，保留集群可在运行之间安全执行；保留会产生费用的基础设施
如果 
.env
中设置了 
TEARDOWN=1
，
make benchmark
会在流水线结束时自动运行 
make down
。在带此标志调用 
make benchmark
前，务必确认这是用户的意图。

Step 1 — Verify prerequisites

第 1 步——验证前置条件

All commands run from 
skills/omni-gateway/omni-gateway-benchmark/
(where the 
Makefile
lives — the shell scripts the Makefile calls are under 
scripts/
).
Run the bundled preflight; it covers every check below in one pass and prints a remediation command for each gap:
bash
make preflight
What 
make preflight
verifies:
  • Required CLIs present: 
    terraform
    , 
    kubectl
    , 
    helm
    , 
    aws
    , 
    docker
    (with 
    buildx
    ), 
    flexctl
    , 
    curl
    , 
    jq
    , 
    python3
    , 
    envsubst
    , 
    sha256sum
    , 
    shellcheck
    .
  • Docker engine — client + server versions reported, and 
    docker info
    confirms the daemon is reachable on the current context.
  • AWS identity (
    aws sts get-caller-identity
    ), the resolved 
    AWS_PROFILE
    , and 
    AWS_REGION
    loaded from 
    .env
    .
  • .env
    exists and 
    .run/registration/registration.yaml
    is present.
  • When 
    POLICIES
    includes 
    client-id-enforcement
    , both 
    CLIENT_ID
    and 
    CLIENT_SECRET
    are non-empty.
  • Egress to 
    flex-packages.anypoint.mulesoft.com
    (the Helm chart source).
Exits non-zero if any check fails — wire it into your shell pipeline (
make preflight && make up
) so a missing prerequisite stops the run before AWS billing starts.
The AWS principal must have permissions to create EC2, EKS, VPC, ECR, IAM, and EBS resources in the target region.
所有命令都在 
skills/omni-gateway/omni-gateway-benchmark/
下运行（Makefile 位于该目录——Makefile 调用的 shell 脚本位于 
scripts/
中）。
运行内置的 preflight；它一次完成以下所有检查，并为每项缺失内容打印修复命令：
bash
make preflight
make preflight
验证以下内容：
  • 必需的 CLI 是否存在：
    terraform
    、
    kubectl
    、
    helm
    、
    aws
    、
    docker
    （包含 
    buildx
    ）、
    flexctl
    、
    curl
    、
    jq
    、
    python3
    、
    envsubst
    、
    sha256sum
    、
    shellcheck
    。
  • Docker engine——报告 client 和 server 版本，并通过 
    docker info
    确认当前上下文中的 daemon 可以访问。
  • AWS identity（
    aws sts get-caller-identity
    ）、解析出的 
    AWS_PROFILE
    ，以及从 
    .env
    加载的 
    AWS_REGION
    。
  • .env
    是否存在，以及 
    .run/registration/registration.yaml
    是否存在。
  • 当 
    POLICIES
    包含 
    client-id-enforcement
    时，
    CLIENT_ID
    和 
    CLIENT_SECRET
    是否均非空。
  • 是否能够访问 
    flex-packages.anypoint.mulesoft.com
    （Helm chart source）。
如果任何检查失败，命令会以非零状态退出——将其接入 shell 流水线（
make preflight && make up
），以便在开始产生 AWS 费用前因缺少前置条件而停止运行。
AWS principal 必须拥有在目标区域中创建 EC2、EKS、VPC、ECR、IAM 和 EBS 资源的权限。

Step 2 — Configure the environment

第 2 步——配置环境

Copy the example 
.env
and fill in scenario knobs:
bash
cp .env.example .env
Edit 
.env
based on the scenario the user wants. Reference table — the harness reads every variable below:
VariableDefaultDescription
AWS_REGION
us-east-2
AWS region for the EKS cluster
CLUSTER_NAME
flex-bench
EKS cluster + VPC name prefix
FLEX_VERSION
1.13.0
Flex Gateway Helm chart and image version
FLEX_IMAGE_REPOSITORY
mulesoft/flex-gateway
Override to use a private mirror
FLEX_IMAGE_TAG
(empty → 
FLEX_VERSION
)		Override to decouple tag from version
N_APIS
10
Number of 
ApiInstance
resources to create
POLICIES
rate-limit,client-id-enforcement
Comma-list of policies to attach per API
RPS
1000
Target requests per second (constant-arrival-rate)
VUS
200
Pre-allocated k6 virtual users
DURATION
2m
k6 test duration (e.g., 
2m
, 
5m
)
CLIENT_ID
(empty)Required when 
client-id-enforcement
is in 
POLICIES
CLIENT_SECRET
(empty)Required when 
client-id-enforcement
is in 
POLICIES
GRAFANA_PORT
33000
Local port used by 
export-grafana-snapshot.sh
TEARDOWN
0
Set to 
1
to auto-run 
make down
after 
make benchmark
REGISTRATION_FILE
.run/registration/registration.yaml
Path to Flex local-mode registration
Validation checklist before continuing:
  • If 
    POLICIES
    contains 
    client-id-enforcement
    , both 
    CLIENT_ID
    and 
    CLIENT_SECRET
    are set.
  • If 
    POLICIES=
    (empty), no auth headers are required.
  • FLEX_VERSION
    matches a chart version published at 
    flex-packages.anypoint.mulesoft.com/helm
    .
复制示例 
.env
并填写场景参数：
bash
cp .env.example .env
根据用户想要的场景编辑 
.env
。参考表如下——此工具会读取以下每个变量：
变量默认值说明
AWS_REGION
us-east-2
EKS 集群所在的 AWS 区域
CLUSTER_NAME
flex-bench
EKS 集群和 VPC 名称前缀
FLEX_VERSION
1.13.0
Flex Gateway Helm chart 和镜像版本
FLEX_IMAGE_REPOSITORY
mulesoft/flex-gateway
覆盖此值可使用私有镜像仓库
FLEX_IMAGE_TAG
(为空 → 
FLEX_VERSION
)		覆盖此值可解除 tag 与版本之间的关联
N_APIS
10
要创建的 
ApiInstance
资源数量
POLICIES
rate-limit,client-id-enforcement
每个 API 要附加的策略，以逗号分隔的列表表示
RPS
1000
目标每秒请求数（constant-arrival-rate）
VUS
200
预分配的 k6 virtual users 数量
DURATION
2m
k6 测试时长（例如 
2m
、
5m
）
CLIENT_ID
(为空)当 
POLICIES
中包含 
client-id-enforcement
时必填
CLIENT_SECRET
(为空)当 
POLICIES
中包含 
client-id-enforcement
时必填
GRAFANA_PORT
33000
export-grafana-snapshot.sh
使用的本地端口
TEARDOWN
0
设置为 
1
后，
make benchmark
完成后会自动运行 
make down
REGISTRATION_FILE
.run/registration/registration.yaml
Flex local-mode registration 的路径
继续执行前的验证清单：
  • 如果 
    POLICIES
    包含 
    client-id-enforcement
    ，则必须同时设置 
    CLIENT_ID
    和 
    CLIENT_SECRET
    。
  • 如果 
    POLICIES=
    （为空），则不需要 auth headers。
  • FLEX_VERSION
    必须与 
    flex-packages.anypoint.mulesoft.com/helm
    上已发布的 chart 版本一致。

Step 3 — Generate the Flex registration artifact (one-time)

第 3 步——生成 Flex registration 工件（一次性）

The benchmark runs Flex in local mode. Use the bundled target — it generates the artifact once and, when 
POLICIES
includes 
client-id-enforcement
, prompts for 
CLIENT_ID
/ 
CLIENT_SECRET
and writes them into 
.env
:
bash
make prepare-registration
Behavior:
  • Skips registration generation if 
    .run/registration/registration.yaml
    already exists. To rotate the gateway identity (forces a Flex redeploy), re-run with 
    FORCE=1 make prepare-registration
    — the existing file is backed up to 
    *.bak
    first.
  • Skips the credential prompt unless 
    POLICIES
    contains 
    client-id-enforcement
    .
  • Skips overwriting existing non-empty 
    CLIENT_ID
    / 
    CLIENT_SECRET
    unless 
    FORCE=1
    .
  • Supports non-interactive use: 
    CLIENT_ID=... CLIENT_SECRET=... make prepare-registration
    .
Manual equivalent (use only if 
flexctl
is being driven outside the harness):
bash
flexctl registration create \
  --connected=false \
  --output-directory=.run/registration
Do not commit 
.run/registration/registration.yaml
— it is gateway identity material and 
.run/
is gitignored.
基准测试以 local mode 运行 Flex。使用内置 target——它只生成一次该工件；当 
POLICIES
包含 
client-id-enforcement
时，提示输入 
CLIENT_ID
/ 
CLIENT_SECRET
并将其写入 
.env
：
bash
make prepare-registration
行为：
  • 如果 
    .run/registration/registration.yaml
    已存在，则跳过 registration 生成。要轮换网关身份（会强制重新部署 Flex），请使用 
    FORCE=1 make prepare-registration
    重新运行——现有文件会先备份为 
    *.bak
    。
  • 除非 
    POLICIES
    包含 
    client-id-enforcement
    ，否则跳过凭据提示。
  • 除非使用 
    FORCE=1
    ，否则跳过对现有非空 
    CLIENT_ID
    / 
    CLIENT_SECRET
    的覆盖。
  • 支持非交互式使用：
    CLIENT_ID=... CLIENT_SECRET=... make prepare-registration
    。
等效的手动操作（仅在工具之外驱动 
flexctl
时使用）：
bash
flexctl registration create \
  --connected=false \
  --output-directory=.run/registration
不要提交 
.run/registration/registration.yaml
——它属于网关身份材料，并且 
.run/
已加入 gitignore。

Step 4 — Build and push the upstream image (one-time)

第 4 步——构建并推送 upstream 镜像（一次性）

The k6 driver hits an in-cluster echo server. The image must exist in ECR before 
deploy-upstream
runs:
bash
make push-upstream
This builds for 
linux/amd64
via 
docker buildx
(cross-compiling on Apple Silicon) and pushes to the ECR repository created by Terraform. If 
make up
has not yet run, this fails because the ECR repo does not exist — provision infra first (Step 5).
Skip condition: If 
make up
has already been run and the upstream image is unchanged, this step is a no-op. The deploy-upstream hash check will skip redeployment if the rendered manifest matches the live state.
k6 driver 会向集群内的 echo server 发起请求。必须在 
deploy-upstream
运行之前，镜像已存在于 ECR 中：
bash
make push-upstream
此操作通过 
docker buildx
为 
linux/amd64
构建镜像（在 Apple Silicon 上进行 cross-compiling），并推送到 Terraform 创建的 ECR repository。如果尚未运行 
make up
，此操作会失败，因为 ECR repository 尚不存在——请先置备基础设施（第 5 步）。
**跳过条件：**如果已经运行过 
make up
且 upstream 镜像未发生变化，则此步骤不会执行任何操作。如果渲染后的 manifest 与当前状态一致，
deploy-upstream
的 hash 检查会跳过重新部署。

Step 5 — Provision infrastructure (destructive on AWS billing)

第 5 步——置备基础设施（会在 AWS 上产生费用且具有破坏性）

Confirm with the user before running. This creates a VPC, EKS cluster, ECR, and installs the observability + k6-operator Helm releases:
bash
make up
Expected duration: 15–25 minutes for a fresh cluster. Re-running on an existing cluster is idempotent and finishes in seconds.
After completion, verify the cluster is reachable:
bash
aws eks update-kubeconfig --region "$AWS_REGION" --name "$CLUSTER_NAME"
kubectl get nodes
You should see two 
system
nodes (
t3.large
, tainted) and three 
workload
nodes (
c6i.2xlarge
, 
node-role=workload
).
运行前请向用户确认。这会创建 VPC、EKS 集群和 ECR，并安装 observability + k6-operator Helm releases：
bash
make up
预计耗时：新建集群需要 15–25 分钟。在现有集群上重新运行具有幂等性，几秒内即可完成。
完成后，验证集群可以访问：
bash
aws eks update-kubeconfig --region "$AWS_REGION" --name "$CLUSTER_NAME"
kubectl get nodes
应该看到两个 
system
nodes（
t3.large
，带 taint）和三个 
workload
nodes（
c6i.2xlarge
，
node-role=workload
）。

Step 6 — Deploy Flex + upstream

第 6 步——部署 Flex + upstream

These targets are idempotent and content-addressed via SHA-256 hashes stored as pod template annotations / Deployment labels. Re-running with unchanged config is a no-op:
bash
make deploy-upstream    # bench-upstream Service + Deployment in 'default' namespace
make deploy-flex        # Flex Gateway Helm release + CRD resources in 'flex' namespace
deploy-flex
renders 
flex-config.yaml
from 
assets/config/flex-config-header.yaml
+ 
assets/config/snippets/api-instance.yaml
+ 
assets/config/policies/<policy>.yaml
, hashes it together with the image reference, and skips 
helm upgrade
if the hash matches the live deployment. Config changes apply via 
kubectl apply
to CRDs without restarting Flex pods.
这些 target 具备幂等性，并通过存储在 pod template annotations / Deployment labels 中的 SHA-256 hashes 实现基于内容寻址。配置未变时重新运行不会执行任何操作：
bash
make deploy-upstream    # bench-upstream Service + Deployment in 'default' namespace
make deploy-flex        # Flex Gateway Helm release + CRD resources in 'flex' namespace
deploy-flex
根据 
assets/config/flex-config-header.yaml
+ 
assets/config/snippets/api-instance.yaml
+ 
assets/config/policies/<policy>.yaml
渲染 
flex-config.yaml
，将其与镜像引用一同计算 hash；如果 hash 与当前部署匹配，则跳过 
helm upgrade
。配置变更通过 
kubectl apply
应用到 CRD，无需重启 Flex pods。

Step 7 — Run the benchmark

第 7 步——运行基准测试

A single load test, including report generation:
bash
make run
This sequences: render 
scenario.js
→ create the per-run ConfigMap → apply the k6 
TestRun
CRD → poll until 
finished
(default timeout 1800s) → port-forward Grafana and export PNGs → emit the Markdown report.
RUN_ID
defaults to a UTC timestamp (
20260612T140530Z
). Override it for a stable identifier:
bash
RUN_ID=baseline-1.13.0-rl make run
For the full pipeline (
up
→ deploys → 
run
, optionally 
down
if 
TEARDOWN=1
):
bash
make benchmark
Re-running with different scenario knobs does not require redeploying:
bash
RPS=2000 VUS=400 DURATION=5m make run
N_APIS=50 POLICIES= make run            # 50 APIs, no policies
FLEX_VERSION=1.14.0 make deploy-flex && make run
If a 
TestRun
is already in flight in the 
k6-operator-system
namespace, the new run will fail with a name collision — wait for the previous run to finish or run 
make clean-deployment
to clear stale TestRuns.
执行单次负载测试，包括报告生成：
bash
make run
此过程会依次执行：渲染 
scenario.js
→ 创建每次运行的 ConfigMap → 应用 k6 
TestRun
CRD → 轮询直至 
finished
（默认超时 1800s）→ 对 Grafana 进行端口转发并导出 PNG → 生成 Markdown 报告。
RUN_ID
默认为 UTC 时间戳（
20260612T140530Z
）。如需使用稳定标识符，可覆盖此值：
bash
RUN_ID=baseline-1.13.0-rl make run
如需运行完整流水线（
up
→ 各次部署 → 
run
，如果 
TEARDOWN=1
则最后执行 
down
）：
bash
make benchmark
使用不同场景参数重新运行时，无需重新部署：
bash
RPS=2000 VUS=400 DURATION=5m make run
N_APIS=50 POLICIES= make run            # 50 APIs, no policies
FLEX_VERSION=1.14.0 make deploy-flex && make run
如果 
k6-operator-system
namespace 中已有正在执行的 
TestRun
，新运行会因名称冲突而失败——请等待上一次运行结束，或运行 
make clean-deployment
清除过期的 TestRuns。

Step 8 — Retrieve results

第 8 步——获取结果

Each run persists artifacts under 
reports/<RUN_ID>/
:
ArtifactPathNotes
Markdown report
reports/<RUN_ID>/flex<VERSION>_n<N_APIS>_<POLICIES>_rps<RPS>_<RUN_ID>.md
Scenario params, raw k6 metrics (
http_reqs
, 
http_req_failed
, 
http_req_duration
, 
checks_succeeded
), Grafana deep-links
Dashboard PNGs
reports/<RUN_ID>/<DashboardTitle>.png
1600×900, one per dashboard (Flex / Envoy, Flex / Pods, k6 / Driver, plus default kube-prometheus-stack boards)
Ephemeral render output lives under 
.run/<RUN_ID>/
(
flex-config.yaml
, 
scenario.js
, 
flex-values.yaml
, 
upstream-deployment.yaml
) and is safe to delete.
To re-export Grafana for the most recent run without re-running k6:
bash
make report
To open Grafana interactively while a run is in flight (anonymous Viewer is enabled; no login required):
bash
make watch        # opens port-forward + prints live URLs + opens k6 dashboard
每次运行都会在 
reports/<RUN_ID>/
下持久保存以下工件：
工件路径说明
Markdown report
reports/<RUN_ID>/flex<VERSION>_n<N_APIS>_<POLICIES>_rps<RPS>_<RUN_ID>.md
场景参数、原始 k6 metrics（
http_reqs
、
http_req_failed
、
http_req_duration
、
checks_succeeded
）、Grafana 深度链接
Dashboard PNGs
reports/<RUN_ID>/<DashboardTitle>.png
1600×900，每个仪表板一张（Flex / Envoy、Flex / Pods、k6 / Driver，以及默认的 kube-prometheus-stack 仪表板）
临时渲染输出保存在 
.run/<RUN_ID>/
下（
flex-config.yaml
、
scenario.js
、
flex-values.yaml
、
upstream-deployment.yaml
），可以安全删除。
如需在不重新运行 k6 的情况下重新导出最近一次运行的 Grafana 内容：
bash
make report
如需在运行期间交互式打开 Grafana（已启用匿名 Viewer；无需登录）：
bash
make watch        # opens port-forward + prints live URLs + opens k6 dashboard

or, manually:

or, manually:

kubectl -n monitoring port-forward svc/kps-grafana 3000:80
kubectl -n monitoring port-forward svc/kps-grafana 3000:80

→ http://localhost:3000

→ http://localhost:3000


`make watch` accepts `RUN_ID=<id>` to filter the k6 / Driver dashboard to a
specific run, and `OPEN_BROWSER=0` to skip the auto-open.

---

`make watch` 接受 `RUN_ID=<id>`，用于将 k6 / Driver dashboard 筛选到特定运行；也接受 `OPEN_BROWSER=0` 以跳过自动打开浏览器。

---

Step 9 — Clean up

第 9 步——清理

Two cleanup paths, ordered by aggressiveness:
bash
make clean-deployment    # Remove Flex + upstream + k6 TestRuns; keep cluster + observability
make down                # terraform destroy — destroys everything (confirm with user first)
clean-deployment
is safe between sessions and preserves the cluster cost (~$0.10/hr for the EKS control plane plus node-group on-demand pricing). 
make down
stops all AWS billing for this environment but is irreversible — any reports not copied out of 
reports/
will remain locally, but ECR images and cluster state are gone.
提供两条清理路径，按破坏性从低到高排列：
bash
make clean-deployment    # Remove Flex + upstream + k6 TestRuns; keep cluster + observability
make down                # terraform destroy — destroys everything (confirm with user first)
clean-deployment
可在会话之间安全执行，并保留集群相关费用（EKS control plane 约为 ~$0.10/hr，另加 node-group on-demand pricing）。
make down
会停止此环境的全部 AWS 计费，但不可逆——未从 
reports/
中复制出来的报告仍会保留在本地，但 ECR 镜像和集群状态将被删除。

Scenario Cookbook

场景示例

Common parameter combinations:
GoalConfiguration
Baseline, no policies
N_APIS=10 POLICIES= RPS=1000 VUS=200 DURATION=2m
Rate-limit only
POLICIES=rate-limit RATE_LIMIT_RPS=500
Client-ID enforcement only
POLICIES=client-id-enforcement CLIENT_ID=... CLIENT_SECRET=...
Both policies, high fan-out
N_APIS=50 POLICIES=rate-limit,client-id-enforcement RPS=2000 VUS=400 DURATION=5m
Version comparisonRun twice with 
FLEX_VERSION=1.13.0
and 
FLEX_VERSION=1.14.0
; both 
make deploy-flex && make run
; diff the two 
reports/<RUN_ID>/*.md
files
The k6 thresholds in the rendered scenario are hard failures: 
http_req_failed < 1%
, 
p(95) < 500ms
, 
p(99) < 1000ms
. A run that exceeds them will surface in the report's 
checks_succeeded
line — the harness still completes and reports.
常见参数组合：
目标配置
无策略的基线测试
N_APIS=10 POLICIES= RPS=1000 VUS=200 DURATION=2m
仅启用 rate-limit
POLICIES=rate-limit RATE_LIMIT_RPS=500
仅启用 client-ID enforcement
POLICIES=client-id-enforcement CLIENT_ID=... CLIENT_SECRET=...
同时启用两种策略和高 fan-out
N_APIS=50 POLICIES=rate-limit,client-id-enforcement RPS=2000 VUS=400 DURATION=5m
版本比较分别使用 
FLEX_VERSION=1.13.0
和 
FLEX_VERSION=1.14.0
运行两次；两次都执行 
make deploy-flex && make run
；比较两个 
reports/<RUN_ID>/*.md
文件
渲染后的场景中，k6 thresholds 属于 hard failures：
http_req_failed < 1%
、
p(95) < 500ms
、
p(99) < 1000ms
。超过这些阈值的运行会显示在报告的 
checks_succeeded
行中——此工具仍会完成运行并生成报告。

Troubleshooting

故障排查

ProblemLikely causeFix
make up
fails with 
AccessDenied
AWS principal lacks EKS/VPC/ECR/IAM permissionsAttach a policy granting the missing actions; verify with 
aws sts get-caller-identity
make push-upstream
fails with 
repository does not exist
make up
not yet run; ECR repo missing		Run 
make up
first, then retry 
make push-upstream
deploy-flex
fails pulling the chart
FLEX_VERSION
not published at the public chart repo		Check 
helm search repo flex-gateway/flex-gateway --versions
; pick a published version
make run
blocks past the 1800s timeout		k6 TestRun stuck in 
error
or 
creating
; missing image; namespace not ready
kubectl -n k6-operator-system get testruns,pods
; inspect 
kubectl describe testrun <name>
Report missing PNGsGrafana port-forward race or image-renderer plugin unavailableRe-run 
make report
; if persistent, check 
kubectl -n monitoring logs deploy/kps-grafana
for the renderer container
client-id-enforcement
returns 401 on every request
CLIENT_ID
/ 
CLIENT_SECRET
empty or do not match a Contract resource		Set both in 
.env
; verify the Contract resource exists for the API instance
Flex pod in 
CrashLoopBackOff
after 
deploy-flex
Missing or stale 
flex-registration
secret		Regenerate via 
flexctl registration create --connected=false
; rerun 
make deploy-flex
bench/spec-hash
annotation never updates		Hash collision (rare) or annotation patch failedForce redeploy: 
kubectl -n flex annotate deploy flex-gateway bench/spec-hash- --overwrite
then 
make deploy-flex
RUN_ID
collision causes TestRun apply to fail		Previous TestRun with same 
RUN_SLUG
still in the namespace
make clean-deployment
or 
kubectl -n k6-operator-system delete testrun <name>
For deeper triage of the gateway itself (not the harness), invoke 
omni-gateway-logs
or 
omni-gateway-diagnose
.
问题可能原因修复方法
make up
因 
AccessDenied
失败		AWS principal 缺少 EKS/VPC/ECR/IAM 权限附加授予缺失操作的 policy；通过 
aws sts get-caller-identity
验证
make push-upstream
因 
repository does not exist
失败		尚未运行 
make up
；ECR repo 不存在		先运行 
make up
，然后重试 
make push-upstream
deploy-flex
拉取 chart 时失败		公共 chart repo 中未发布 
FLEX_VERSION
检查 
helm search repo flex-gateway/flex-gateway --versions
；选择已发布的版本
make run
超过 1800s 超时时间仍阻塞		k6 TestRun 卡在 
error
或 
creating
状态；镜像缺失；namespace 未就绪
kubectl -n k6-operator-system get testruns,pods
；检查 
kubectl describe testrun <name>
报告缺少 PNGsGrafana port-forward 竞态或 image-renderer plugin 不可用重新运行 
make report
；如果问题持续，检查 
kubectl -n monitoring logs deploy/kps-grafana
中的 renderer container
client-id-enforcement
对每个请求都返回 401
CLIENT_ID
/ 
CLIENT_SECRET
为空，或与 Contract resource 不匹配		在 
.env
中设置两者；验证 API instance 存在对应的 Contract resource
部署 
deploy-flex
后 Flex pod 出现 
CrashLoopBackOff
flex-registration
secret 缺失或过期		通过 
flexctl registration create --connected=false
重新生成；然后重新运行 
make deploy-flex
bench/spec-hash
annotation 始终未更新		Hash 冲突（很少见）或 annotation patch 失败强制重新部署：
kubectl -n flex annotate deploy flex-gateway bench/spec-hash- --overwrite
，然后运行 
make deploy-flex
RUN_ID
冲突导致 TestRun apply 失败		namespace 中仍存在具有相同 
RUN_SLUG
的上一个 TestRun		运行 
make clean-deployment
或 
kubectl -n k6-operator-system delete testrun <name>
如需更深入地排查网关本身的问题（而非此工具的问题），请调用 
omni-gateway-logs
或 
omni-gateway-diagnose
。

Available Scripts

可用脚本

The 
make
targets above are the supported entry points. They delegate to the scripts in 
scripts/
, listed here for reference when debugging the harness or composing steps manually. Every script reads its configuration from 
.env
and accepts 
--help
for usage:
ScriptDriven byPurpose
preflight.sh
make preflight
Read-only check of every CLI, daemon, credential, and one-time artifact before any run
prepare-registration.sh
make prepare-registration
Generate the local-mode Flex registration artifact; collect 
CLIENT_ID
/
CLIENT_SECRET
when needed
push-upstream.sh
make push-upstream
Build the upstream echo image for 
linux/amd64
and push to ECR
deploy-upstream.sh
make deploy-upstream
Deploy (or hash-skip) the 
bench-upstream
Service + Deployment
deploy-flex.sh
make deploy-flex
Render config, then deploy (or hash-skip) the Flex Gateway Helm release + CRDs
render-flex-config.sh
deploy-flex.sh
Assemble 
flex-config.yaml
from header + API-instance + policy snippets
wait-for-flex.sh
deploy-flex.sh
Block until the Flex Gateway deployment finishes rolling out
render-k6-script.sh
run-bench.sh
Render the k6 
scenario.js
from scenario knobs
run-bench.sh
make run
Create the per-run k6 
TestRun
, wait for it, and export the Grafana snapshot
wait-for-testrun.sh
run-bench.sh
Poll a k6 
TestRun
until it reaches 
finished
(or times out)
export-grafana-snapshot.sh
run-bench.sh
Port-forward Grafana and export dashboard PNGs for a time window
generate-report.sh
make run
Parse k6 summary from runner logs and emit the per-run Markdown report
watch-grafana.sh
make watch
Open a Grafana port-forward and print live dashboard URLs
clean-deployment.sh
make clean-deployment
Tear down per-run workloads while keeping the cluster, observability, and k6-operator
上述 
make
targets 是受支持的入口点。它们会将工作委托给 
scripts/
中的脚本；下表列出这些脚本，供调试工具或手动编排步骤时参考。 每个脚本均从 
.env
读取配置，并接受 
--help
查看用法：
脚本驱动方式用途
preflight.sh
make preflight
在任何运行开始前，只读检查所有 CLI、daemon、凭据和一次性工件
prepare-registration.sh
make prepare-registration
生成 local-mode Flex registration 工件；按需收集 
CLIENT_ID
/
CLIENT_SECRET
push-upstream.sh
make push-upstream
为 
linux/amd64
构建 upstream echo 镜像并推送到 ECR
deploy-upstream.sh
make deploy-upstream
部署（或通过 hash 跳过）
bench-upstream
Service + Deployment
deploy-flex.sh
make deploy-flex
渲染配置，然后部署（或通过 hash 跳过）Flex Gateway Helm release + CRDs
render-flex-config.sh
deploy-flex.sh
根据 header + API-instance + policy snippets 组装 
flex-config.yaml
wait-for-flex.sh
deploy-flex.sh
阻塞等待 Flex Gateway deployment 完成 rollout
render-k6-script.sh
run-bench.sh
根据场景参数渲染 k6 
scenario.js
run-bench.sh
make run
创建本次运行的 k6 
TestRun
，等待其完成并导出 Grafana snapshot
wait-for-testrun.sh
run-bench.sh
轮询 k6 
TestRun
，直至其达到 
finished
状态或超时
export-grafana-snapshot.sh
run-bench.sh
对 Grafana 进行端口转发，并导出指定时间窗口内的 dashboard PNGs
generate-report.sh
make run
从 runner logs 中解析 k6 summary，并为本次运行生成 Markdown report
watch-grafana.sh
make watch
打开 Grafana port-forward 并打印实时 dashboard URLs
clean-deployment.sh
make clean-deployment
拆除每次运行的工作负载，同时保留集群、observability 和 k6-operator

Related Jobs

相关 Jobs

  • omni-gateway-install
    — install Flex on Linux/Docker/Kubernetes (this skill provisions its own EKS install for benchmarking; use 
    omni-gateway-install
    for production deployments)
  • omni-gateway-config
    — validate the rendered 
    flex-config.yaml
    before 
    make deploy-flex
  • omni-gateway-logs
    — parse Flex pod logs when a benchmark surfaces unexpected failures
  • omni-gateway-diagnose
    — triage non-harness gateway issues observed during a run
  • omni-gateway-install
    ——在 Linux/Docker/Kubernetes 上安装 Flex（此 skill 会为基准测试置备自己的 EKS 安装；生产部署应使用 
    omni-gateway-install
    ）
  • omni-gateway-config
    ——在执行 
    make deploy-flex
    前验证渲染后的 
    flex-config.yaml
  • omni-gateway-logs
    ——当基准测试出现意外失败时解析 Flex pod logs
  • omni-gateway-diagnose
    ——排查运行期间发现且并非由工具导致的网关问题