omni-gateway-logs

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Inspect Gateway Logs

检查网关日志

Log Format Reference

日志格式参考

Omni Gateway produces logs in two formats depending on the deployment platform.
Omni Gateway 会根据部署平台生成两种格式的日志。

Docker format

Docker 格式

[component][level] message
Examples:
[flex-gateway-agent][info] Running Envoy XDS Service...
[flex-gateway-agent][error] Error authorizing https://anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
[flex-gateway-envoy][info] all dependencies initialized. starting workers
[flex-external-processor][info] Ext Proc Server successfully started.
[flex-gateway-agent][warn] Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.
[component][level] message
示例:
[flex-gateway-agent][info] Running Envoy XDS Service...
[flex-gateway-agent][error] Error authorizing https://anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
[flex-gateway-envoy][info] all dependencies initialized. starting workers
[flex-external-processor][info] Ext Proc Server successfully started.
[flex-gateway-agent][warn] Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.

Kubernetes format (structured)

Kubernetes 格式(结构化)

<ISO-timestamp> LEVEL [pod-name] component - message
Examples:
2026-02-26T22:56:56.935Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Running Envoy XDS Service...
2026-02-26T23:01:13.53Z ERROR [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Error authorizing https://stgx.anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
2026-02-26T22:56:57.447Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Creating gateway
In Kubernetes, multiple pods appear in the same log stream; each line includes the pod name in brackets.
<ISO-timestamp> LEVEL [pod-name] component - message
示例:
2026-02-26T22:56:56.935Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Running Envoy XDS Service...
2026-02-26T23:01:13.53Z ERROR [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Error authorizing https://stgx.anypoint.mulesoft.com: Post "https://anypoint.mulesoft.com/accounts/oauth2/token": dial tcp 54.173.213.129:443: connect: connection timed out
2026-02-26T22:56:57.447Z INFO [pbtest1-766cbd68f9-g2r7n] flex-gateway-agent - Creating gateway
在 Kubernetes 中,多个 Pod 会出现在同一个日志流中;每行日志的方括号中都会包含 Pod 名称。

Component glossary

组件术语表

ComponentWhat it does
flex-gateway-agent
Main agent: gateway lifecycle, registration with Anypoint, config management, token refresh, control-plane sync
flex-gateway-envoy
Envoy proxy: HTTP/HTTPS/gRPC routing, TLS termination, upstream connections, xDS config consumer
flex-gateway-fluent
Fluent Bit: log collection and forwarding
flex-external-processor
External processor (Java): handles external-processing policies
open-telemetry-collector
OpenTelemetry collector: metrics and tracing export
组件作用
flex-gateway-agent
主 Agent:负责网关生命周期、Anypoint 注册、配置管理、令牌刷新以及控制平面同步
flex-gateway-envoy
Envoy 代理:负责 HTTP/HTTPS/gRPC 路由、TLS 终止、上游连接以及 xDS 配置消费
flex-gateway-fluent
Fluent Bit:负责日志收集和转发
flex-external-processor
外部处理器(Java):负责处理外部处理策略
open-telemetry-collector
OpenTelemetry Collector:负责指标和追踪数据导出

Normal Startup Sequence

正常启动顺序

A healthy gateway startup produces these INFO messages in order. Use this as a reference to identify where startup stalled:
Running Envoy XDS Service...
RTMDataSource: Starting
KubernetesDataSource: Starting          ← Kubernetes only
Creating gateway
Processing commands
Validating gateway
Generating config
Gateway: Platform=... Mode=connected ReplicaName=...
Writing envoy bootstrap configuration to /var/tmp/mulesoft/flex-gateway/envoy.json
RTMDataSource: Change configuration
flex-gateway-envoy: Starting
flex-gateway-fluent: Starting
flex-external-processor: Starting
all dependencies initialized. starting workers     ← from flex-gateway-envoy
Anypoint websocket: connected                       ← connected mode only
open delta watch ID:1 for ...Cluster...             ← xDS subscriptions active
open delta watch ID:2 for ...Listener...
If the log cuts off before
all dependencies initialized. starting workers
, the gateway did not finish starting.
健康的网关启动过程会按顺序产生以下 INFO 消息。可将其作为参考,以确定启动流程停滞的位置:
Running Envoy XDS Service...
RTMDataSource: Starting
KubernetesDataSource: Starting          ← Kubernetes only
Creating gateway
Processing commands
Validating gateway
Generating config
Gateway: Platform=... Mode=connected ReplicaName=...
Writing envoy bootstrap configuration to /var/tmp/mulesoft/flex-gateway/envoy.json
RTMDataSource: Change configuration
flex-gateway-envoy: Starting
flex-gateway-fluent: Starting
flex-external-processor: Starting
all dependencies initialized. starting workers     ← from flex-gateway-envoy
Anypoint websocket: connected                       ← connected mode only
open delta watch ID:1 for ...Cluster...             ← xDS subscriptions active
open delta watch ID:2 for ...Listener...
如果日志在
all dependencies initialized. starting workers
之前中断,则网关未完成启动。

How to Obtain Logs

如何获取日志

Linux (systemd)

Linux(systemd)

bash
undefined
bash
undefined

View last 200 log lines

View last 200 log lines

sudo journalctl -u flex-gateway -n 200 --no-pager
sudo journalctl -u flex-gateway -n 200 --no-pager

Stream logs in real-time

Stream logs in real-time

sudo journalctl -u flex-gateway -f
sudo journalctl -u flex-gateway -f

View logs within a time window

View logs within a time window

sudo journalctl -u flex-gateway --since "2026-06-08 14:00:00" --until "2026-06-08 15:00:00"
undefined
sudo journalctl -u flex-gateway --since "2026-06-08 14:00:00" --until "2026-06-08 15:00:00"
undefined

Docker

Docker

bash
undefined
bash
undefined

View last 200 log lines

View last 200 log lines

docker logs --tail=200 flex-gateway
docker logs --tail=200 flex-gateway

Stream logs in real-time

Stream logs in real-time

docker logs -f flex-gateway
docker logs -f flex-gateway

View logs with timestamps

View logs with timestamps

docker logs --timestamps flex-gateway
undefined
docker logs --timestamps flex-gateway
undefined

Kubernetes

Kubernetes

Replace
<namespace>
and
<pod-name>
with the values from your cluster. If you don't know the pod name, list pods first.
bash
undefined
将
<namespace>
和
<pod-name>
替换为集群中的实际值。如果不知道 Pod 名称,请先列出 Pod。
bash
undefined

Find gateway pods

Find gateway pods

kubectl get pods -n <namespace> -l app.kubernetes.io/name=flex-gateway
kubectl get pods -n <namespace> -l app.kubernetes.io/name=flex-gateway

View logs from a specific pod

View logs from a specific pod

kubectl logs -n <namespace> <pod-name> --tail=200
kubectl logs -n <namespace> <pod-name> --tail=200

View logs from all gateway pods

View logs from all gateway pods

kubectl logs -n <namespace> -l app.kubernetes.io/name=flex-gateway --all-containers=true --tail=100
kubectl logs -n <namespace> -l app.kubernetes.io/name=flex-gateway --all-containers=true --tail=100

View logs from a previous (crashed) pod instance

View logs from a previous (crashed) pod instance

kubectl logs -n <namespace> <pod-name> --previous
kubectl logs -n <namespace> <pod-name> --previous

Stream logs in real-time

Stream logs in real-time

kubectl logs -n <namespace> <pod-name> -f
undefined
kubectl logs -n <namespace> <pod-name> -f
undefined

How to Enable Debug Logging

如何启用调试日志

To enable debug-level logs, create a Configuration resource in the user conf.d directory.
Create or edit
/usr/local/share/mulesoft/flex-gateway/conf.d/logging.yaml
:
yaml
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: logging
spec:
  logging:
    runtimeLogs:
      logLevel: debug
On Kubernetes, add this as a ConfigMap entry mounted into the conf.d volume. The gateway picks up the change without a restart.
To revert to INFO logging, change
logLevel
to
info
or delete the Configuration resource.
Note: There is no
flexctl
command to enable debug logging. The Configuration YAML is the only supported method and works identically in connected and local modes.
要启用 debug 级别日志,请在用户的 conf.d 目录中创建一个 Configuration 资源。
创建或编辑
/usr/local/share/mulesoft/flex-gateway/conf.d/logging.yaml
:
yaml
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: logging
spec:
  logging:
    runtimeLogs:
      logLevel: debug
在 Kubernetes 上,将其添加为挂载到 conf.d 卷中的 ConfigMap 条目。网关无需重启即可识别此更改。
要恢复为 INFO 日志级别,请将
logLevel
改为
info
,或删除 Configuration 资源。
注意: 没有可用于启用调试日志的
flexctl
命令。Configuration YAML 是唯一受支持的方法,并且在 connected 模式和 local 模式下的行为完全相同。

Analysis Checklist

分析检查清单

Step 1 — Parse by severity

第 1 步——按严重级别解析

Count occurrences of each severity level. Use these keywords:
  • Docker:
    ][error]
    ,
    ][warn]
    ,
    ][info]
    ,
    ][debug]
  • Kubernetes:
    ERROR
    ,
    WARN
    ,
    INFO
    ,
    DEBUG
Decision logic:
  • Zero
    error
    lines and few
    warn
    lines → gateway is likely healthy. Report the baseline and stop unless the user is investigating a specific problem.
  • Any
    error
    lines present, or the user reports an issue → continue to Steps 2–4.
统计每个严重级别出现的次数。使用以下关键词:
  • Docker:
    ][error]
    、
    ][warn]
    、
    ][info]
    、
    ][debug]
  • Kubernetes:
    ERROR
    、
    WARN
    、
    INFO
    、
    DEBUG
判断逻辑:
  • 没有
    error
    行且只有少量
    warn
    行 → 网关很可能运行正常。报告基线情况并停止分析,除非用户正在调查特定问题。
  • 存在任何
    error
    行,或用户报告了问题 → 继续执行第 2–4 步。

Step 2 — Identify components

第 2 步——识别组件

Group error and warning lines by their component prefix:
  • **flex-gateway-agent
    errors** → registration failures, token refresh failures, Anypoint connectivity, lifecycle issues, config processing errors
  • **flex-gateway-envoy
    errors** → upstream connectivity, TLS, routing, or xDS config issues
  • **flex-external-processor
    errors** → policy execution failures (DataWeave, LDAP, ext-proc policies)
  • **flex-gateway-fluent
    errors** → log forwarding failures (non-critical for gateway traffic)
  • **open-telemetry-collector
    errors** → metrics/tracing export failures (non-critical for gateway traffic)
Note which components are generating errors and at what point in the lifecycle.
按组件前缀对错误行和警告行进行分组:
  • **flex-gateway-agent
    错误 → 注册失败、令牌刷新失败、Anypoint 连接问题、生命周期问题或配置处理错误
  • **flex-gateway-envoy
    错误 → 上游连接、TLS、路由或 xDS 配置问题
  • **flex-external-processor
    错误 → 策略执行失败(DataWeave、LDAP、ext-proc 策略)
  • **flex-gateway-fluent
    错误 → 日志转发失败(对网关流量不重要)
  • **open-telemetry-collector
    错误 → 指标或追踪数据导出失败(对网关流量不重要)
记录哪些组件正在产生错误,以及错误发生在生命周期的哪个阶段。

Step 3 — Deduplicate repeated errors

第 3 步——对重复错误去重

Identical or structurally similar error messages often repeat many times. Collapse these into a single entry with a count:
  • Instead of: "Error authorizing https://anypoint.mulesoft.com" repeated 14 times
  • Report: "14×
    flex-gateway-agent
    :
    Error authorizing https://anypoint.mulesoft.com: ... connection timed out
    "
This separates signal from noise and makes the analysis readable.
相同或结构相似的错误消息往往会重复出现多次。将这些消息合并为一条记录,并附上出现次数:
  • 不要报告:“Error authorizing https://anypoint.mulesoft.com”重复了 14 次
  • 应报告:“14×
    flex-gateway-agent
    :
    Error authorizing https://anypoint.mulesoft.com: ... connection timed out
    ”
这样可以区分信号与噪声,使分析结果更易于阅读。

Step 4 — Check for known failure patterns

第 4 步——检查已知故障模式

Match errors and warnings against this table:
Log patternSeverityLikely causeNext step
Error authorizing https://*.anypoint.mulesoft.com: ... connect: connection timed out
ERRORGateway cannot reach Anypoint OAuth endpoint to refresh its tokenCheck network/firewall rules: the gateway must be able to reach
anypoint.mulesoft.com
(or the appropriate region endpoint) on port 443; verify egress rules from the pod/host
Max retries reached for token authorization: ...
ERRORSustained failure to refresh the Anypoint OAuth token; gateway is now disconnected from the control planeImmediate: restore connectivity to
anypoint.mulesoft.com
; gateway continues serving cached config but won't receive updates
Processing commands errors: (Ingress .../...: the resource is not allowed in connected mode)
ERRORA Kubernetes
Ingress
resource is present but the gateway is running in connected mode — Ingress is only supported in local mode
Remove the
Ingress
resource; in connected mode, use
ApiInstance
resources managed via Anypoint Runtime Manager instead
Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.
WARNRegistration was created with an older version of
flexctl
Re-register the gateway using the current
flexctl register
command; invoke
omni-gateway-install
skill (registration steps)
MetricsScheduler: flex_startup event error: ... connect: connection timed out
WARNCannot report startup metrics to Anypoint (non-critical: gateway traffic is unaffected)Check Anypoint connectivity; analytics events will be missing for this startup
MetricsScheduler: flex_snapshot event error: ... connect: connection timed out
WARNCannot report periodic metrics to Anypoint (non-critical: gateway traffic is unaffected)Check Anypoint connectivity; analytics data will have gaps
upstream connect error or disconnect/reset before headers
ERROR (Envoy access log)Upstream service not reachable or closed connectionVerify the
spec.services[*].address
in the ApiInstance; confirm upstream is running and accepting connections
TLS handshake failed
/
certificate verify failed
/
CERTIFICATE_VERIFY_FAILED
ERRORTLS misconfiguration, expired certificate, or incomplete certificate chainCheck certificate validity and expiration; review TLS config in conf.d
Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option ... http2_protocol_options
WARNING (Envoy)Envoy using a deprecated internal config fieldExpected and harmless — known issue in current Omni Gateway versions; ignore unless other problems are present
Token expiring soon (in ...)
INFONormal Anypoint OAuth token refresh cycleExpected and normal — gateway is proactively refreshing its token; not an error
将错误和警告与下表进行匹配:
日志模式严重级别可能原因后续步骤
Error authorizing https://*.anypoint.mulesoft.com: ... connect: connection timed out
ERROR网关无法访问 Anypoint OAuth 端点以刷新令牌检查网络/防火墙规则:网关必须能够通过端口 443 访问
anypoint.mulesoft.com
(或相应区域的端点);验证 Pod/主机的出站规则
Max retries reached for token authorization: ...
ERROR持续无法刷新 Anypoint OAuth 令牌;网关现已与控制平面断开连接立即恢复与
anypoint.mulesoft.com
的连接;网关会继续使用缓存配置提供服务,但将无法接收更新
Processing commands errors: (Ingress .../...: the resource is not allowed in connected mode)
ERROR存在 Kubernetes
Ingress
资源,但网关正在以 connected 模式运行——Ingress 仅支持 local 模式
删除
Ingress
资源;在 connected 模式下,改用通过 Anypoint Runtime Manager 管理的
ApiInstance
资源
Registration is using v1 version, but some metrics are only supported for v2. Please renew your registration to avoid errors.
WARN注册时使用的是旧版本的
flexctl
使用当前的
flexctl register
命令重新注册网关;调用
omni-gateway-install
skill(注册步骤)
MetricsScheduler: flex_startup event error: ... connect: connection timed out
WARN无法向 Anypoint 报告启动指标(非关键问题,不影响网关流量)检查 Anypoint 连接;此次启动的分析事件将会缺失
MetricsScheduler: flex_snapshot event error: ... connect: connection timed out
WARN无法向 Anypoint 报告周期性指标(非关键问题,不影响网关流量)检查 Anypoint 连接;分析数据将出现缺口
upstream connect error or disconnect/reset before headers
ERROR(Envoy 访问日志)上游服务不可达或关闭了连接检查 ApiInstance 中的
spec.services[*].address
;确认上游服务正在运行并接受连接
TLS handshake failed
/
certificate verify failed
/
CERTIFICATE_VERIFY_FAILED
ERRORTLS 配置错误、证书过期或证书链不完整检查证书有效性和过期时间;审查 conf.d 中的 TLS 配置
Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option ... http2_protocol_options
WARNING(Envoy)Envoy 使用了已弃用的内部配置字段预期且无害——这是当前 Omni Gateway 版本中的已知问题;除非存在其他问题,否则可以忽略
Token expiring soon (in ...)
INFO正常的 Anypoint OAuth 令牌刷新周期预期且正常——网关正在主动刷新令牌;这不是错误

Step 5 — If logs are truncated or incomplete

第 5 步——日志被截断或不完整时

If the log excerpt is cut off or covers too short a window:
  1. Ask the user: when did the issue start? Request a longer window (e.g., last 1000 lines or last hour).
  2. Provide the appropriate streaming or time-scoped command for their platform (see "How to Obtain Logs" above).
  3. If there was a triggering event (deployment, config change, restart, traffic spike), ask the user to anchor it with a timestamp so you can focus on that window.
如果日志摘录被截断,或覆盖的时间窗口过短:
  1. 询问用户:问题何时开始?请求提供更长的时间窗口(例如最近 1000 行或最近一小时)。
  2. 提供适合其平台的实时流式输出或限定时间范围命令(参见上面的“如何获取日志”)。
  3. 如果存在触发事件(部署、配置更改、重启、流量峰值),请用户提供一个时间戳作为锚点,以便聚焦对应时间窗口。

Step 6 — If errors are sparse or the cause is unclear

第 6 步——错误很少或原因不明确时

If the logs contain few errors but the user reports a problem:
  1. Enable debug logging using the Configuration YAML method above.
  2. Ask the user to reproduce the failing request or wait for the issue to recur.
  3. Re-collect logs using the appropriate platform command.
  4. Return to Step 1 with the verbose logs.
Debug logs reveal timing issues, intermediate state transitions, token refresh details, and policy evaluation steps that INFO logs omit.
如果日志中只有少量错误,但用户报告了问题:
  1. 使用上述 Configuration YAML 方法启用调试日志。
  2. 请用户重现失败的请求,或等待问题再次出现。
  3. 使用适合其平台的命令重新收集日志。
  4. 使用详细日志返回第 1 步。
调试日志能够揭示时序问题、中间状态转换、令牌刷新详情以及 INFO 日志中省略的策略评估步骤。

Step 7 — Produce structured summary

第 7 步——生成结构化摘要

Output a structured analysis block:
undefined
输出结构化分析块:
undefined

Log Analysis Summary

Log Analysis Summary

Period: <start timestamp> – <end timestamp> (or "unknown" if timestamps absent) Total lines analyzed: <N> Log level: <INFO / DEBUG / TRACE / mixed> Platform: <Docker / Kubernetes / Linux> Mode: <connected / local / unknown>
Period: <start timestamp> – <end timestamp> (or "unknown" if timestamps absent) Total lines analyzed: <N> Log level: <INFO / DEBUG / TRACE / mixed> Platform: <Docker / Kubernetes / Linux> Mode: <connected / local / unknown>

Severity Counts

Severity Counts

  • ERROR: N
  • WARN: N
  • INFO: N
  • DEBUG/TRACE: N
  • ERROR: N
  • WARN: N
  • INFO: N
  • DEBUG/TRACE: N

Component Breakdown

Component Breakdown

  • flex-gateway-agent
    : N lines, X errors
  • flex-gateway-envoy
    : N lines, X errors
  • flex-external-processor
    : N lines, X errors
  • flex-gateway-fluent
    : N lines, X errors
  • open-telemetry-collector
    : N lines, X errors
  • flex-gateway-agent
    : N lines, X errors
  • flex-gateway-envoy
    : N lines, X errors
  • flex-external-processor
    : N lines, X errors
  • flex-gateway-fluent
    : N lines, X errors
  • open-telemetry-collector
    : N lines, X errors

Top Issues (deduplicated)

Top Issues (deduplicated)

  1. [Nx flex-gateway-agent] Error authorizing ... connection timed out — hypothesis
  2. [Nx flex-gateway-agent] Processing commands errors: Ingress ... — hypothesis
  3. ...
  1. [Nx flex-gateway-agent] Error authorizing ... connection timed out — hypothesis
  2. [Nx flex-gateway-agent] Processing commands errors: Ingress ... — hypothesis
  3. ...

Startup Completion

Startup Completion

Gateway reached "all dependencies initialized" / "Anypoint websocket: connected": YES / NO / UNKNOWN
Gateway reached "all dependencies initialized" / "Anypoint websocket: connected": YES / NO / UNKNOWN

Root Cause Hypothesis

Root Cause Hypothesis

<One or two sentences on the likely root cause based on patterns and timeline.>
<One or two sentences on the likely root cause based on patterns and timeline.>

Recommended Next Steps

Recommended Next Steps

  1. <Specific action, e.g., "Check egress firewall rules: gateway needs port 443 access to anypoint.mulesoft.com">
  2. <Specific action, e.g., "Run
    omni-gateway-config
    — error suggests an Ingress resource is present">
  3. <Specific action, e.g., "Re-register the gateway with current flexctl to upgrade from v1 registration">
undefined
  1. <Specific action, e.g., "Check egress firewall rules: gateway needs port 443 access to anypoint.mulesoft.com">
  2. <Specific action, e.g., "Run
    omni-gateway-config
    — error suggests an Ingress resource is present">
  3. <Specific action, e.g., "Re-register the gateway with current flexctl to upgrade from v1 registration">
undefined

Related Jobs

相关任务

  • **omni-gateway-diagnose**
    — End-to-end triage router when you need to decide which investigation path to take.
  • **omni-gateway-config**
    — When analysis points to a configuration issue in conf.d.
  • **omni-gateway-install**
    — When logs show the gateway failed to register, lost control-plane connection, or has a v1 registration warning (re-run the registration steps).
  • **omni-gateway-diagnose**
    — 需要判断应采用哪条调查路径时使用的端到端分诊路由。
  • **omni-gateway-config**
    — 当分析指向 conf.d 中的配置问题时使用。
  • **omni-gateway-install**
    — 当日志显示网关注册失败、失去控制平面连接或出现 v1 注册警告时使用(重新执行注册步骤)。