omni-gateway-logs
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseInspect Gateway Logs
检查网关日志
Log Format Reference
日志格式参考
Omni Gateway produces logs in two formats depending on the deployment platform.
Omni Gateway 会根据部署平台生成两种格式的日志。
Docker format
Docker 格式
[component][level] messageExamples:
[flex-gateway-agent][info] Running Envoy XDS Service...
[flex-gateway-agent][error] Error authorizing https://anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
[flex-gateway-envoy][info] all dependencies initialized. starting workers
[flex-external-processor][info] Ext Proc Server successfully started.
[flex-gateway-agent][warn] Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.[component][level] message示例:
[flex-gateway-agent][info] Running Envoy XDS Service...
[flex-gateway-agent][error] Error authorizing https://anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
[flex-gateway-envoy][info] all dependencies initialized. starting workers
[flex-external-processor][info] Ext Proc Server successfully started.
[flex-gateway-agent][warn] Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.Kubernetes format (structured)
Kubernetes 格式(结构化)
<ISO-timestamp> LEVEL [pod-name] component - messageExamples:
2026-02-26T22:56:56.935Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Running Envoy XDS Service...
2026-02-26T23:01:13.53Z ERROR [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Error authorizing https://stgx.anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
2026-02-26T22:56:57.447Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Creating gatewayIn Kubernetes, multiple pods appear in the same log stream; each line includes the pod name in brackets.
<ISO-timestamp> LEVEL [pod-name] component - message示例:
2026-02-26T22:56:56.935Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Running Envoy XDS Service...
2026-02-26T23:01:13.53Z ERROR [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Error authorizing https://stgx.anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
2026-02-26T22:56:57.447Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Creating gateway在 Kubernetes 中,多个 Pod 会出现在同一个日志流中;每行日志的方括号中都会包含 Pod 名称。
Component glossary
组件术语表
| Component | What it does |
|---|---|
| Main agent: gateway lifecycle, registration with Anypoint, config management, token refresh, control-plane sync |
| Envoy proxy: HTTP/HTTPS/gRPC routing, TLS termination, upstream connections, xDS config consumer |
| Fluent Bit: log collection and forwarding |
| External processor (Java): handles external-processing policies |
| OpenTelemetry collector: metrics and tracing export |
| 组件 | 作用 |
|---|---|
| 主 Agent:负责网关生命周期、Anypoint 注册、配置管理、令牌刷新以及控制平面同步 |
| Envoy 代理:负责 HTTP/HTTPS/gRPC 路由、TLS 终止、上游连接以及 xDS 配置消费 |
| Fluent Bit:负责日志收集和转发 |
| 外部处理器(Java):负责处理外部处理策略 |
| OpenTelemetry Collector:负责指标和追踪数据导出 |
Normal Startup Sequence
正常启动顺序
A healthy gateway startup produces these INFO messages in order. Use this as a reference to identify where startup stalled:
Running Envoy XDS Service...
RTMDataSource: Starting
KubernetesDataSource: Starting ← Kubernetes only
Creating gateway
Processing commands
Validating gateway
Generating config
Gateway: Platform=... Mode=connected ReplicaName=...
Writing envoy bootstrap configuration to /var/tmp/mulesoft/flex-gateway/envoy.json
RTMDataSource: Change configuration
flex-gateway-envoy: Starting
flex-gateway-fluent: Starting
flex-external-processor: Starting
all dependencies initialized. starting workers ← from flex-gateway-envoy
Anypoint websocket: connected ← connected mode only
open delta watch ID:1 for ...Cluster... ← xDS subscriptions active
open delta watch ID:2 for ...Listener...If the log cuts off before , the gateway did not finish starting.
all dependencies initialized. starting workers健康的网关启动过程会按顺序产生以下 INFO 消息。可将其作为参考,以确定启动流程停滞的位置:
Running Envoy XDS Service...
RTMDataSource: Starting
KubernetesDataSource: Starting ← Kubernetes only
Creating gateway
Processing commands
Validating gateway
Generating config
Gateway: Platform=... Mode=connected ReplicaName=...
Writing envoy bootstrap configuration to /var/tmp/mulesoft/flex-gateway/envoy.json
RTMDataSource: Change configuration
flex-gateway-envoy: Starting
flex-gateway-fluent: Starting
flex-external-processor: Starting
all dependencies initialized. starting workers ← from flex-gateway-envoy
Anypoint websocket: connected ← connected mode only
open delta watch ID:1 for ...Cluster... ← xDS subscriptions active
open delta watch ID:2 for ...Listener...如果日志在 之前中断,则网关未完成启动。
all dependencies initialized. starting workersHow to Obtain Logs
如何获取日志
Linux (systemd)
Linux(systemd)
bash
undefinedbash
undefinedView last 200 log lines
View last 200 log lines
sudo journalctl -u flex-gateway -n 200 --no-pager
sudo journalctl -u flex-gateway -n 200 --no-pager
Stream logs in real-time
Stream logs in real-time
sudo journalctl -u flex-gateway -f
sudo journalctl -u flex-gateway -f
View logs within a time window
View logs within a time window
sudo journalctl -u flex-gateway --since "2026-06-08 14:00:00" --until "2026-06-08 15:00:00"
undefinedsudo journalctl -u flex-gateway --since "2026-06-08 14:00:00" --until "2026-06-08 15:00:00"
undefinedDocker
Docker
bash
undefinedbash
undefinedView last 200 log lines
View last 200 log lines
docker logs --tail=200 flex-gateway
docker logs --tail=200 flex-gateway
Stream logs in real-time
Stream logs in real-time
docker logs -f flex-gateway
docker logs -f flex-gateway
View logs with timestamps
View logs with timestamps
docker logs --timestamps flex-gateway
undefineddocker logs --timestamps flex-gateway
undefinedKubernetes
Kubernetes
Replace and with the values from your cluster. If you don't know the pod name, list pods first.
<namespace><pod-name>bash
undefined将 和 替换为集群中的实际值。如果不知道 Pod 名称,请先列出 Pod。
<namespace><pod-name>bash
undefinedFind gateway pods
Find gateway pods
kubectl get pods -n <namespace> -l app.kubernetes.io/name=flex-gateway
kubectl get pods -n <namespace> -l app.kubernetes.io/name=flex-gateway
View logs from a specific pod
View logs from a specific pod
kubectl logs -n <namespace> <pod-name> --tail=200
kubectl logs -n <namespace> <pod-name> --tail=200
View logs from all gateway pods
View logs from all gateway pods
kubectl logs -n <namespace> -l app.kubernetes.io/name=flex-gateway --all-containers=true --tail=100
kubectl logs -n <namespace> -l app.kubernetes.io/name=flex-gateway --all-containers=true --tail=100
View logs from a previous (crashed) pod instance
View logs from a previous (crashed) pod instance
kubectl logs -n <namespace> <pod-name> --previous
kubectl logs -n <namespace> <pod-name> --previous
Stream logs in real-time
Stream logs in real-time
kubectl logs -n <namespace> <pod-name> -f
undefinedkubectl logs -n <namespace> <pod-name> -f
undefinedHow to Enable Debug Logging
如何启用调试日志
To enable debug-level logs, create a Configuration resource in the user conf.d directory.
Create or edit :
/usr/local/share/mulesoft/flex-gateway/conf.d/logging.yamlyaml
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
name: logging
spec:
logging:
runtimeLogs:
logLevel: debugOn Kubernetes, add this as a ConfigMap entry mounted into the conf.d volume. The gateway picks up the change without a restart.
To revert to INFO logging, change to or delete the Configuration resource.
logLevelinfoNote: There is no command to enable debug logging. The Configuration YAML is the only supported method and works identically in connected and local modes.
flexctl要启用 debug 级别日志,请在用户的 conf.d 目录中创建一个 Configuration 资源。
创建或编辑 :
/usr/local/share/mulesoft/flex-gateway/conf.d/logging.yamlyaml
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
name: logging
spec:
logging:
runtimeLogs:
logLevel: debug在 Kubernetes 上,将其添加为挂载到 conf.d 卷中的 ConfigMap 条目。网关无需重启即可识别此更改。
要恢复为 INFO 日志级别,请将 改为 ,或删除 Configuration 资源。
logLevelinfo注意: 没有可用于启用调试日志的 命令。Configuration YAML 是唯一受支持的方法,并且在 connected 模式和 local 模式下的行为完全相同。
flexctlAnalysis Checklist
分析检查清单
Step 1 — Parse by severity
第 1 步——按严重级别解析
Count occurrences of each severity level. Use these keywords:
- Docker: ,
][error],][warn],][info]][debug] - Kubernetes: ,
ERROR,WARN,INFODEBUG
Decision logic:
- Zero lines and few
errorlines → gateway is likely healthy. Report the baseline and stop unless the user is investigating a specific problem.warn - Any lines present, or the user reports an issue → continue to Steps 2–4.
error
统计每个严重级别出现的次数。使用以下关键词:
- Docker: 、
][error]、][warn]、][info]][debug] - Kubernetes: 、
ERROR、WARN、INFODEBUG
判断逻辑:
- 没有 行且只有少量
error行 → 网关很可能运行正常。报告基线情况并停止分析,除非用户正在调查特定问题。warn - 存在任何 行,或用户报告了问题 → 继续执行第 2–4 步。
error
Step 2 — Identify components
第 2 步——识别组件
Group error and warning lines by their component prefix:
- errors** → registration failures, token refresh failures, Anypoint connectivity, lifecycle issues, config processing errors
**flex-gateway-agent - errors** → upstream connectivity, TLS, routing, or xDS config issues
**flex-gateway-envoy - errors** → policy execution failures (DataWeave, LDAP, ext-proc policies)
**flex-external-processor - errors** → log forwarding failures (non-critical for gateway traffic)
**flex-gateway-fluent - errors** → metrics/tracing export failures (non-critical for gateway traffic)
**open-telemetry-collector
Note which components are generating errors and at what point in the lifecycle.
按组件前缀对错误行和警告行进行分组:
- 错误 → 注册失败、令牌刷新失败、Anypoint 连接问题、生命周期问题或配置处理错误
**flex-gateway-agent - 错误 → 上游连接、TLS、路由或 xDS 配置问题
**flex-gateway-envoy - 错误 → 策略执行失败(DataWeave、LDAP、ext-proc 策略)
**flex-external-processor - 错误 → 日志转发失败(对网关流量不重要)
**flex-gateway-fluent - 错误 → 指标或追踪数据导出失败(对网关流量不重要)
**open-telemetry-collector
记录哪些组件正在产生错误,以及错误发生在生命周期的哪个阶段。
Step 3 — Deduplicate repeated errors
第 3 步——对重复错误去重
Identical or structurally similar error messages often repeat many times. Collapse these into a single entry with a count:
- Instead of: "Error authorizing https://anypoint.mulesoft.com" repeated 14 times
- Report: "14× :
flex-gateway-agent"Error authorizing https://anypoint.mulesoft.com: ... connection timed out
This separates signal from noise and makes the analysis readable.
相同或结构相似的错误消息往往会重复出现多次。将这些消息合并为一条记录,并附上出现次数:
- 不要报告:“Error authorizing https://anypoint.mulesoft.com”重复了 14 次
- 应报告:“14× :
flex-gateway-agent”Error authorizing https://anypoint.mulesoft.com: ... connection timed out
这样可以区分信号与噪声,使分析结果更易于阅读。
Step 4 — Check for known failure patterns
第 4 步——检查已知故障模式
Match errors and warnings against this table:
| Log pattern | Severity | Likely cause | Next step |
|---|---|---|---|
| ERROR | Gateway cannot reach Anypoint OAuth endpoint to refresh its token | Check network/firewall rules: the gateway must be able to reach |
| ERROR | Sustained failure to refresh the Anypoint OAuth token; gateway is now disconnected from the control plane | Immediate: restore connectivity to |
| ERROR | A Kubernetes | Remove the |
| WARN | Registration was created with an older version of | Re-register the gateway using the current |
| WARN | Cannot report startup metrics to Anypoint (non-critical: gateway traffic is unaffected) | Check Anypoint connectivity; analytics events will be missing for this startup |
| WARN | Cannot report periodic metrics to Anypoint (non-critical: gateway traffic is unaffected) | Check Anypoint connectivity; analytics data will have gaps |
| ERROR (Envoy access log) | Upstream service not reachable or closed connection | Verify the |
| ERROR | TLS misconfiguration, expired certificate, or incomplete certificate chain | Check certificate validity and expiration; review TLS config in conf.d |
| WARNING (Envoy) | Envoy using a deprecated internal config field | Expected and harmless — known issue in current Omni Gateway versions; ignore unless other problems are present |
| INFO | Normal Anypoint OAuth token refresh cycle | Expected and normal — gateway is proactively refreshing its token; not an error |
将错误和警告与下表进行匹配:
| 日志模式 | 严重级别 | 可能原因 | 后续步骤 |
|---|---|---|---|
| ERROR | 网关无法访问 Anypoint OAuth 端点以刷新令牌 | 检查网络/防火墙规则:网关必须能够通过端口 443 访问 |
| ERROR | 持续无法刷新 Anypoint OAuth 令牌;网关现已与控制平面断开连接 | 立即恢复与 |
| ERROR | 存在 Kubernetes | 删除 |
| WARN | 注册时使用的是旧版本的 | 使用当前的 |
| WARN | 无法向 Anypoint 报告启动指标(非关键问题,不影响网关流量) | 检查 Anypoint 连接;此次启动的分析事件将会缺失 |
| WARN | 无法向 Anypoint 报告周期性指标(非关键问题,不影响网关流量) | 检查 Anypoint 连接;分析数据将出现缺口 |
| ERROR(Envoy 访问日志) | 上游服务不可达或关闭了连接 | 检查 ApiInstance 中的 |
| ERROR | TLS 配置错误、证书过期或证书链不完整 | 检查证书有效性和过期时间;审查 conf.d 中的 TLS 配置 |
| WARNING(Envoy) | Envoy 使用了已弃用的内部配置字段 | 预期且无害——这是当前 Omni Gateway 版本中的已知问题;除非存在其他问题,否则可以忽略 |
| INFO | 正常的 Anypoint OAuth 令牌刷新周期 | 预期且正常——网关正在主动刷新令牌;这不是错误 |
Step 5 — If logs are truncated or incomplete
第 5 步——日志被截断或不完整时
If the log excerpt is cut off or covers too short a window:
- Ask the user: when did the issue start? Request a longer window (e.g., last 1000 lines or last hour).
- Provide the appropriate streaming or time-scoped command for their platform (see "How to Obtain Logs" above).
- If there was a triggering event (deployment, config change, restart, traffic spike), ask the user to anchor it with a timestamp so you can focus on that window.
如果日志摘录被截断,或覆盖的时间窗口过短:
- 询问用户:问题何时开始?请求提供更长的时间窗口(例如最近 1000 行或最近一小时)。
- 提供适合其平台的实时流式输出或限定时间范围命令(参见上面的“如何获取日志”)。
- 如果存在触发事件(部署、配置更改、重启、流量峰值),请用户提供一个时间戳作为锚点,以便聚焦对应时间窗口。
Step 6 — If errors are sparse or the cause is unclear
第 6 步——错误很少或原因不明确时
If the logs contain few errors but the user reports a problem:
- Enable debug logging using the Configuration YAML method above.
- Ask the user to reproduce the failing request or wait for the issue to recur.
- Re-collect logs using the appropriate platform command.
- Return to Step 1 with the verbose logs.
Debug logs reveal timing issues, intermediate state transitions, token refresh details, and policy evaluation steps that INFO logs omit.
如果日志中只有少量错误,但用户报告了问题:
- 使用上述 Configuration YAML 方法启用调试日志。
- 请用户重现失败的请求,或等待问题再次出现。
- 使用适合其平台的命令重新收集日志。
- 使用详细日志返回第 1 步。
调试日志能够揭示时序问题、中间状态转换、令牌刷新详情以及 INFO 日志中省略的策略评估步骤。
Step 7 — Produce structured summary
第 7 步——生成结构化摘要
Output a structured analysis block:
undefined输出结构化分析块:
undefinedLog Analysis Summary
Log Analysis Summary
Period: <start timestamp> – <end timestamp> (or "unknown" if timestamps absent)
Total lines analyzed: <N>
Log level: <INFO / DEBUG / TRACE / mixed>
Platform: <Docker / Kubernetes / Linux>
Mode: <connected / local / unknown>
Period: <start timestamp> – <end timestamp> (or "unknown" if timestamps absent)
Total lines analyzed: <N>
Log level: <INFO / DEBUG / TRACE / mixed>
Platform: <Docker / Kubernetes / Linux>
Mode: <connected / local / unknown>
Severity Counts
Severity Counts
- ERROR: N
- WARN: N
- INFO: N
- DEBUG/TRACE: N
- ERROR: N
- WARN: N
- INFO: N
- DEBUG/TRACE: N
Component Breakdown
Component Breakdown
- : N lines, X errors
flex-gateway-agent - : N lines, X errors
flex-gateway-envoy - : N lines, X errors
flex-external-processor - : N lines, X errors
flex-gateway-fluent - : N lines, X errors
open-telemetry-collector
- : N lines, X errors
flex-gateway-agent - : N lines, X errors
flex-gateway-envoy - : N lines, X errors
flex-external-processor - : N lines, X errors
flex-gateway-fluent - : N lines, X errors
open-telemetry-collector
Top Issues (deduplicated)
Top Issues (deduplicated)
- [Nx flex-gateway-agent] Error authorizing ... connection timed out — hypothesis
- [Nx flex-gateway-agent] Processing commands errors: Ingress ... — hypothesis
- ...
- [Nx flex-gateway-agent] Error authorizing ... connection timed out — hypothesis
- [Nx flex-gateway-agent] Processing commands errors: Ingress ... — hypothesis
- ...
Startup Completion
Startup Completion
Gateway reached "all dependencies initialized" / "Anypoint websocket: connected": YES / NO / UNKNOWN
Gateway reached "all dependencies initialized" / "Anypoint websocket: connected": YES / NO / UNKNOWN
Root Cause Hypothesis
Root Cause Hypothesis
<One or two sentences on the likely root cause based on patterns and timeline.>
<One or two sentences on the likely root cause based on patterns and timeline.>
Recommended Next Steps
Recommended Next Steps
- <Specific action, e.g., "Check egress firewall rules: gateway needs port 443 access to anypoint.mulesoft.com">
- <Specific action, e.g., "Run — error suggests an Ingress resource is present">
omni-gateway-config - <Specific action, e.g., "Re-register the gateway with current flexctl to upgrade from v1 registration">
undefined- <Specific action, e.g., "Check egress firewall rules: gateway needs port 443 access to anypoint.mulesoft.com">
- <Specific action, e.g., "Run — error suggests an Ingress resource is present">
omni-gateway-config - <Specific action, e.g., "Re-register the gateway with current flexctl to upgrade from v1 registration">
undefinedRelated Jobs
相关任务
- — End-to-end triage router when you need to decide which investigation path to take.
**omni-gateway-diagnose** - — When analysis points to a configuration issue in conf.d.
**omni-gateway-config** - — When logs show the gateway failed to register, lost control-plane connection, or has a v1 registration warning (re-run the registration steps).
**omni-gateway-install**
- — 需要判断应采用哪条调查路径时使用的端到端分诊路由。
**omni-gateway-diagnose** - — 当分析指向 conf.d 中的配置问题时使用。
**omni-gateway-config** - — 当日志显示网关注册失败、失去控制平面连接或出现 v1 注册警告时使用(重新执行注册步骤)。
**omni-gateway-install**