cx-profiling

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Profiling Skill

Profiling Skill

Use this skill to answer continuous-profiling questions — which services burn the most CPU or memory, and which functions inside a service are responsible — via the Coralogix profiling API. Requires the
profiles.cpu-profiles:Read
permission.
使用此 skill 回答 continuous profiling 相关问题——通过 Coralogix profiling API 找出哪些服务消耗最多 CPU 或内存,以及服务内部哪些函数应对此负责。需要
profiles.cpu-profiles:Read
权限。

CLI Commands

CLI 命令

CommandPurposeKey flags
cx profiling services
Profiled services ranked by CPU+wall seconds (
cpu
) or allocated+heap bytes (
memory
)
--category
(required),
--environment
,
--limit
(default 20, max 100),
--offset
,
--include-time-series
cx profiling metadata <service>
A service's language and which profile types have data-
cx profiling filters <service>
Label keys a service's profiles can be filtered by, with their values
--profile-type
(required),
--filter
,
--max-values-per-key
(default 50)
cx profiling metrics <service>
One profile type's usage over time, split by
service
,
pod
, or any label; highest totals first
--profile-type
(required),
--sum-by
(default
service
),
--max-buckets
(default 60),
--limit
(default 10),
--filter
cx profiling flame-graph <service>
Decoded flame graph plus the top functions by self value
--profile-type
(required),
--filter
,
--top-functions
(default 20),
--min-node-ratio
(default 0.01),
--max-depth
,
--show-kernel
  • All commands are read-only and support
    -o json
    /
    -o toon
    .
  • Every command takes
    --start
    /
    --end
    (default
    now-1h
    /
    now
    ):
    now
    ,
    now-1h
    -style relative expressions, or ISO-8601 timestamps.
  • --profile-type
    is one of
    cpu
    ,
    wall
    ,
    memory-allocation-bytes
    ,
    memory-allocation-objects
    ,
    memory-heap-bytes
    ,
    memory-heap-objects
    (case-insensitive;
    MEMORY_HEAP_BYTES
    also works). Check
    metadata
    first — only types with
    has_data: true
    return anything.
  • --filter
    is repeatable and filters AND together:
    key=v1,v2
    (any of the values),
    key!=v1,v2
    (none of them),
    key:exists
    ,
    key:not-exists
    . Discover keys and values with
    filters
    .
  • services --environment
    matches
    deployment.environment.name
    ; it is the only filter the services ranking honors.
  • services
    pages through the ranking: when a page is full the CLI prints the
    --offset
    to use next. Time series are omitted unless
    --include-time-series
    is passed.
  • flame-graph
    can take ~40 s on large services; don't set a short
    --http-timeout
    .
命令用途关键参数
cx profiling services
按 CPU+wall 秒数(
cpu
)或已分配+堆字节数(
memory
)对已分析的服务进行排名
--category
(必填)、
--environment
、
--limit
(默认 20,最大 100)、
--offset
、
--include-time-series
cx profiling metadata <service>
服务的语言以及哪些 profile 类型存在数据-
cx profiling filters <service>
服务的 profile 可按哪些标签键筛选,以及对应的标签值
--profile-type
(必填)、
--filter
、
--max-values-per-key
(默认 50)
cx profiling metrics <service>
某种 profile 类型随时间变化的使用量,可按
service
、
pod
或任意标签拆分;总量最高的排在前面
--profile-type
(必填)、
--sum-by
(默认
service
)、
--max-buckets
(默认 60)、
--limit
(默认 10)、
--filter
cx profiling flame-graph <service>
解码后的 flame graph,以及按自身值(self value)排名前列的函数
--profile-type
(必填)、
--filter
、
--top-functions
(默认 20)、
--min-node-ratio
(默认 0.01)、
--max-depth
、
--show-kernel
  • 所有命令均为只读操作,并支持
    -o json
    /
    -o toon
    。
  • 每个命令都接受
    --start
    /
    --end
    (默认
    now-1h
    /
    now
    ):
    now
    、
    now-1h
    风格的相对表达式,或 ISO-8601 时间戳。
  • --profile-type
    可取
    cpu
    、
    wall
    、
    memory-allocation-bytes
    、
    memory-allocation-objects
    、
    memory-heap-bytes
    、
    memory-heap-objects
    之一(不区分大小写;
    MEMORY_HEAP_BYTES
    也可用)。请先检查
    metadata
    ——只有
    has_data: true
    的类型才会返回数据。
  • --filter
    可以重复使用,并以 AND 方式组合筛选条件:
    key=v1,v2
    (匹配任一值)、
    key!=v1,v2
    (均不匹配)、
    key:exists
    、
    key:not-exists
    。使用
    filters
    发现可用的键和值。
  • services --environment
    匹配
    deployment.environment.name
    ;这是服务排名唯一会采用的筛选条件。
  • services
    会通过分页遍历排名:当某一页已满时,CLI 会打印下一页应使用的
    --offset
    。除非传入
    --include-time-series
    ,否则不会返回时间序列。
  • 对大型服务执行
    flame-graph
    可能需要约 40 秒;不要设置过短的
    --http-timeout
    。

Investigation Workflow

调查流程

  1. Find the heavy services (skip if the user named one):
    bash
    cx profiling services --category cpu --limit 10 -o json
    cx profiling services --category memory --environment <env> --limit 10 -o json
  2. Check which profile types exist for the service:
    bash
    cx profiling metadata <service> -o json
  3. See what you can filter or split by (pods, versions, environments, ...):
    bash
    cx profiling filters <service> --profile-type cpu -o json
  4. Find when usage changed and who caused it — split by
    pod
    or a label key from step 3:
    bash
    cx profiling metrics <service> --profile-type cpu --sum-by pod --limit 5 -o json
    cx profiling metrics <service> --profile-type cpu --sum-by service.version --start now-24h -o json
  5. Find the code responsible — start with the top functions, then the tree:
    bash
    cx profiling flame-graph <service> --profile-type cpu --top-functions 10 -o json | jq '.top_functions'
    cx profiling flame-graph <service> --profile-type cpu --filter k8s.pod.name=<pod> \
      --min-node-ratio 0.05 --max-depth 20 -o json
  1. 查找高负载服务(如果用户已指定服务,则跳过此步骤):
    bash
    cx profiling services --category cpu --limit 10 -o json
    cx profiling services --category memory --environment <env> --limit 10 -o json
  2. 检查该服务存在哪些 profile 类型:
    bash
    cx profiling metadata <service> -o json
  3. 查看可以按哪些维度筛选或拆分(例如 pod、版本、环境等):
    bash
    cx profiling filters <service> --profile-type cpu -o json
  4. 查找使用量何时发生变化以及由谁引起——按
    pod
    或第 3 步中的标签键进行拆分:
    bash
    cx profiling metrics <service> --profile-type cpu --sum-by pod --limit 5 -o json
    cx profiling metrics <service> --profile-type cpu --sum-by service.version --start now-24h -o json
  5. 查找相关代码——先从排名靠前的函数入手,再查看调用树:
    bash
    cx profiling flame-graph <service> --profile-type cpu --top-functions 10 -o json | jq '.top_functions'
    cx profiling flame-graph <service> --profile-type cpu --filter k8s.pod.name=<pod> \
      --min-node-ratio 0.05 --max-depth 20 -o json

Reading the Output

读取输出

  • Services:
    cpu
    rows carry
    total_cpu_seconds
    /
    total_wall_seconds
    ;
    memory
    rows carry
    total_memory_allocations_bytes
    /
    total_heap_bytes
    .
    last_active
    is epoch seconds.
  • Metrics:
    series
    is ranked by
    total
    (sum of the bucket totals) and trimmed to
    --limit
    ;
    total_series
    is how many the server returned.
    unit
    is
    seconds
    ,
    bytes
    , or
    objects
    depending on the profile type.
  • Flame graph:
    top_functions
    sums self value per function+file across the whole tree, so it is unaffected by tree trimming.
    tree
    nodes carry
    function
    ,
    file
    ,
    line
    ,
    self
    ,
    total
    ,
    self_percent
    ,
    total_percent
    , and
    children
    (sorted by total, highest first).
    omitted_subtrees
    counts nodes dropped by
    --min-node-ratio
    /
    --max-depth
    . The server already prunes nodes under 0.1% of the total. Frames with no symbol show as their address or
    unknown
    .
    warnings
    (e.g.
    maxBytes
    ) mean the server stopped reading profiles early, so the graph covers part of the range.
  • Services:
    cpu
    行包含
    total_cpu_seconds
    /
    total_wall_seconds
    ;
    memory
    行包含
    total_memory_allocations_bytes
    /
    total_heap_bytes
    。
    last_active
    是 Unix epoch 秒数。
  • Metrics:
    series
    按
    total
    (各时间桶总量之和)排名,并裁剪至
    --limit
    ;
    total_series
    表示服务器返回的序列总数。根据 profile 类型,
    unit
    为
    seconds
    、
    bytes
    或
    objects
    。
  • Flame graph:
    top_functions
    汇总整棵调用树中每个函数+文件的自身值, 因此不受调用树裁剪的影响。
    tree
    节点包含
    function
    、
    file
    、
    line
    、
    self
    、
    total
    、
    self_percent
    、
    total_percent
    和
    children
    (按
    total
    降序排列)。
    omitted_subtrees
    表示因
    --min-node-ratio
    /
    --max-depth
    而被丢弃的节点数量。 服务器已经会裁剪总量占比低于 0.1% 的节点。没有符号信息的调用帧会显示其地址或
    unknown
    。出现
    warnings
    (例如
    maxBytes
    )意味着服务器提前停止了 profile 读取, 因此该图仅覆盖部分时间范围。

Key Principles

关键原则

  • Rank first, drill down second —
    services
    and
    metrics
    are cheap;
    flame-graph
    is the expensive call, so narrow the service, profile type, time range, and filters before asking for one.
  • Large trees are big — a deep stack (Java, Scala) can produce hundreds of KB of JSON. Raise
    --min-node-ratio
    or set
    --max-depth
    for the tree, or read just
    .top_functions
    .
  • Self vs total — high
    self
    means the function itself is hot; high
    total
    with low
    self
    means the cost is in what it calls.
  • Empty results are not errors — a service with no data for the chosen profile type or range returns empty series or a zero-total graph; re-check
    metadata
    and widen
    --start
    .
  • No profiling data — if
    metadata
    shows no profile type with
    has_data: true
    , the service is not profiled. Do not look for other service names: answer from
    cx-service-catalog
    or
    cx-telemetry-querying
    (how much CPU or memory, when, which pod) and say that function-level detail needs profiling enabled for that service.
  • Multi-profile fan-out works on every subcommand: repeat
    -p <profile>
    ; results are tagged with
    profile
    when more than one is given.
  • 先排名,再下钻——
    services
    和
    metrics
    的开销较低;
    flame-graph
    的开销较高,因此应先缩小服务、profile 类型、时间范围和筛选条件,再发起请求。
  • 大型调用树可能非常庞大——深层调用栈(例如 Java、Scala)可能产生数百 KB 的 JSON。 请提高
    --min-node-ratio
    或为调用树设置
    --max-depth
    ,或者只读取
    .top_functions
    。
  • Self 与 total 的区别——较高的
    self
    表示函数自身是热点;较高的
    total
    搭配较低的
    self
    表示开销主要来自该函数调用的其他函数。
  • 空结果并非错误——如果服务在所选 profile 类型或时间范围内没有数据,则会返回空序列或总量为零的图; 请重新检查
    metadata
    并扩大
    --start
    范围。
  • 没有 profiling 数据——如果
    metadata
    显示没有任何 profile 类型的
    has_data: true
    , 则说明该服务未启用 profiling。不要尝试寻找其他服务名称:应改用
    cx-service-catalog
    或
    cx-telemetry-querying
    (CPU 或内存使用量、发生时间、对应 pod),并说明函数级明细需要为该服务启用 profiling。
  • 每个子命令都支持多 profile fan-out:重复传入
    -p <profile>
    ;提供多个 profile 时,结果会通过
    profile
    字段进行区分。

Related Skills

相关 Skills

  • cx-service-catalog
    — RED metrics (latency, errors) for the same services; correlate a latency regression with a CPU spike found here.
  • cx-telemetry-querying
    — pivot to logs, spans, or PromQL metrics for a service or pod surfaced by profiling (
    cx logs
    ,
    cx spans
    ,
    cx metrics
    ).
  • cx-infra
    — host and container health when a hot pod points to an infrastructure problem.
  • cx-service-catalog
    ——提供相同服务的 RED 指标(延迟、错误);可将延迟回归与此处发现的 CPU 峰值进行关联。
  • cx-telemetry-querying
    ——针对 profiling 发现的 service 或 pod,切换到 logs、spans 或 PromQL metrics(
    cx logs
    、
    cx spans
    、
    cx metrics
    )。
  • cx-infra
    ——当热点 pod 指向基础设施问题时,用于检查主机和容器健康状况。