observability-onboarding

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Observability Onboarding

可观测性接入

Instrument an application with the Elastic Distribution of OpenTelemetry (EDOT) and prove the telemetry arrived. The instrumentation change is only half the job: an application can be configured perfectly and still emit nothing, so this skill ends by querying Elasticsearch for the service's data rather than declaring success from a config diff.

The scope is application instrumentation with the EDOT SDKs. Deploying an EDOT Collector, collecting logs from files or infrastructure, and onboarding data through Elastic Agent, Fleet, or integration packages are separate ingest paths that this skill does not cover.

Once telemetry is flowing, use the observability-sre-triage skill to assess service health, observability-k8s-investigation for Kubernetes-layer failures, observability-service-reliability to define SLOs and alerts on the new signals, and observability-llm-obs for GenAI and agentic workloads.

使用Elastic Distribution of OpenTelemetry (EDOT)为应用埋点，并验证遥测数据已送达。埋点配置只是完成了一半工作：即使应用配置完美，也可能无法生成任何数据，因此本技能最终会通过查询Elasticsearch获取服务数据，而非仅通过配置差异就宣告成功。

本技能的范围是使用EDOT SDK进行应用埋点。部署EDOT Collector、从文件或基础设施收集日志，以及通过Elastic Agent、Fleet或集成包接入数据属于独立的 ingestion 路径，不在本技能覆盖范围内。

当遥测数据开始流动后，可使用observability-sre-triage技能评估服务健康状况，使用observability-k8s-investigation排查Kubernetes层故障，使用observability-service-reliability基于新信号定义SLO和告警，使用observability-llm-obs处理生成式AI和Agent类工作负载。

Environment Configuration

环境配置

This skill executes Elasticsearch operations through the

elastic

CLI. If the

elastic

CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g.,

GET /

GET /_cat/indices

GET /{index}/_mapping

GET /{index}/_settings/index.mode

POST /_query

). The Operations table at the end of this document maps each shorthand to the equivalent

elastic

CLI command — always use the CLI rather than calling the HTTP API directly.

本技能通过

elastic

CLI执行Elasticsearch操作。如果未安装

elastic

CLI，请告知用户其用途。请勿猜测凭证、直接调用HTTP API或尝试其他变通方法。

本技能以HTTP简写形式引用操作（例如

GET /

、

GET /_cat/indices

、

GET /{index}/_mapping

、

GET /{index}/_settings/index.mode

、

POST /_query

）。本文档末尾的操作表格将每个简写映射为等效的

elastic

CLI命令——请始终使用CLI而非直接调用HTTP API。

Analysis without cluster access

无集群权限时的分析

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Only the verification step in this skill talks to Elasticsearch; it runs through

POST /_query

. Everything else edits application code and configuration in the user's workspace.

上述CLI检查仅限制查询集群——不限制分析。当用户已在问题中提供证据（指标值、计数、状态原因、日志行、告警负载、配置）时，请基于该证据进行推理并给出结论。

当确实需要用户未提供的数据时，请说明你要检查的内容及方式——指明能解决问题的具体查询、索引和字段——然后再请求用户配置CLI。明确检查内容的回答即使没有集群权限也有价值；仅要求配置CLI的回答则无意义。

本技能中只有验证步骤会与Elasticsearch交互，通过

POST /_query

执行。其他所有操作均在用户工作区中编辑应用代码和配置。

Jobs to be done

待完成任务

Decide whether a service needs fresh instrumentation or a migration off the classic Elastic APM agent
Apply the EDOT configuration for the service's language and runtime
Point telemetry at the correct OTLP destination and reject APM Server URLs
Verify that traces, metrics, and logs land in Elasticsearch under the expected
```
service.name
```
Diagnose an instrumented service that is reporting nothing, or reporting from the wrong agent

判断服务是需要全新埋点还是从传统Elastic APM Agent迁移
根据服务的语言和运行时应用EDOT配置
将遥测数据指向正确的OTLP目标，拒绝APM Server URL
验证链路追踪、指标和日志是否以预期的
```
service.name
```
进入Elasticsearch
排查已埋点但未上报数据或上报来源错误的服务

Output discipline

输出规范

Applies to every response produced under this skill.

Make the edit. When the language, the current agent, and the destination are known, apply the change. Do not narrate a plan and ask for approval on each file; ask only when a decision genuinely cannot be inferred.
Do not dump documentation. Link the relevant page and state the two or three rules that apply to this service. Pasting a whole setup guide into the reply is a defect.
Verify before reporting success. "Configured correctly" is not the finding; "telemetry is arriving" is. If verification returns no rows, say the service is not reporting and give the most likely cause.
Do not speculate past the evidence. If the verification query is empty, that is one fact with several possible causes — name the likeliest and how to distinguish it, rather than asserting one.
End on the result. No trailing offers such as "want me to set up dashboards next?". Follow-up work belongs in a recommendations list, phrased as a recommendation.

适用于本技能生成的所有回复。

直接执行修改：当已知语言、当前Agent和目标地址时，直接应用更改。无需叙述计划并请求每个文件的批准；仅在确实无法推断决策时才询问。
不要粘贴完整文档：链接相关页面并说明适用于该服务的两三条规则。将整个设置指南粘贴到回复中属于缺陷。
验证后再报告成功：“配置正确”不是最终结论；“遥测数据已送达”才是。如果验证返回无结果，请说明服务未上报，并给出最可能的原因。
不要超出证据推测：如果验证查询为空，这是一个事实，可能有多种原因——请列出最可能的原因及区分方法，而非断言某一种。
以结果收尾：不要添加诸如“需要我接下来设置仪表盘吗？”之类的后续提议。后续工作应放在建议列表中，以建议的形式呈现。

Process: route the request

流程：请求路由

Identify the language and runtime. Read the project's dependency manifest —

pom.xml

build.gradle

requirements.txt

pyproject.toml

*.csproj

composer.json

. The decision: which language reference to open. Also note how the process starts (a Dockerfile

ENTRYPOINT

, a Kubernetes pod spec, a systemd unit), because that is where the agent attaches and where the environment variables must be set.

Runtime	Reference	How EDOT attaches
Java	references/java.md	`-javaagent:` flag or `JAVA_TOOL_OPTIONS`
Python	references/python.md	`opentelemetry-instrument` entrypoint wrapper
.NET	references/dotnet.md	`builder.AddElasticOpenTelemetry()` in startup
PHP	references/php.md	native extension via OS package, then full process restart

For a language not listed, follow the same shape — the required configuration and the following verification step are language-independent — and use the upstream EDOT SDK documentation for the attach mechanism.

Detect a classic Elastic APM agent. The decision: instrument path or migrate path. Each language reference lists the markers; the common ones are
```
ELASTIC_APM_*
```
environment variables and a language-specific Elastic APM package. If any marker is present, take the migrate path — do not layer EDOT on top.
Resolve the telemetry destination. The decision: which URL goes in
```
OTEL_EXPORTER_OTLP_ENDPOINT
```
. Two valid answers, and one common wrong one:
- The Elastic managed OTLP endpoint for the deployment or Serverless project. Correct default.
- An EDOT Collector the user already runs, when telemetry must be enriched, sampled, or fanned out before it reaches Elastic.
- Never an APM Server URL. If the candidate contains
```
apm-server
```
  , ends in port 8200, or has an
```
/intake/v2/events
```
  path, it is the classic ingest endpoint and EDOT cannot use it. On the migrate path this is the single most common mistake — the old
```
ELASTIC_APM_SERVER_URL
```
  value must not be carried over.
Apply the shared configuration below, then the language reference. The shared rules are the same for every runtime; the reference covers only what is language-specific.
Verify. Do not report success from the configuration alone — run the verification process.

识别语言和运行时：读取项目的依赖清单——

pom.xml

或

build.gradle

、

requirements.txt

或

pyproject.toml

、

*.csproj

、

composer.json

。决策：打开哪个语言参考文档。同时记录进程启动方式（Dockerfile的

ENTRYPOINT

、Kubernetes Pod规格、systemd单元），因为这是Agent挂载的位置，也是必须设置环境变量的地方。

运行时	参考文档	EDOT挂载方式
Java	references/java.md	`-javaagent:` 参数或 `JAVA_TOOL_OPTIONS`
Python	references/python.md	`opentelemetry-instrument` 入口点包装器
.NET	references/dotnet.md	启动时调用 `builder.AddElasticOpenTelemetry()`
PHP	references/php.md	通过OS包安装原生扩展，然后重启整个进程

对于未列出的语言，请遵循相同流程——所需配置和后续验证步骤与语言无关——并使用上游EDOT SDK文档获取挂载机制。

检测传统Elastic APM Agent：决策是选择全新埋点路径还是迁移路径。每个语言参考文档都列出了标记；常见标记包括
```
ELASTIC_APM_*
```
环境变量和特定语言的Elastic APM包。如果存在任何标记，请选择迁移路径——不要在传统Agent之上叠加EDOT。
解析遥测目标地址：决策
```
OTEL_EXPORTER_OTLP_ENDPOINT
```
应使用哪个URL。有两个有效答案，以及一个常见错误答案：
- Elastic托管OTLP端点：适用于部署或Serverless项目。默认正确选项。
- 用户已运行的EDOT Collector：当遥测数据在到达Elastic之前需要 enrichment、采样或扇出时使用。
- 绝不能使用APM Server URL：如果候选地址包含
```
apm-server
```
  、以端口8200结尾或包含
```
/intake/v2/events
```
  路径，则属于传统Ingest端点，EDOT无法使用。在迁移路径中，这是最常见的错误——不能沿用旧的
```
ELASTIC_APM_SERVER_URL
```
  值。
应用以下共享配置，然后应用语言参考文档中的配置：共享规则对所有运行时相同；参考文档仅涵盖语言特定内容。
验证：不要仅根据配置就报告成功——请执行验证流程。

Required configuration

必需配置

Identical across every EDOT SDK. Set exactly three environment variables:

Variable	Value
`OTEL_SERVICE_NAME`	The service's name. This becomes `service.name` in Elasticsearch and is how every other Observability skill finds it.
`OTEL_EXPORTER_OTLP_ENDPOINT`	The managed OTLP endpoint or EDOT Collector URL resolved in step 3.
`OTEL_EXPORTER_OTLP_HEADERS`	The credential, in `Authorization=ApiKey <key>` or `Authorization=Bearer <token>` form.

Two rules that hold for every language:

Do not set
OTEL_TRACES_EXPORTER
,
OTEL_METRICS_EXPORTER
, or
OTEL_LOGS_EXPORTER
. The defaults are already correct. Setting them is the usual reason a service reports traces but no metrics or logs.
Never run a classic Elastic APM agent and EDOT together on the same process. They double-instrument and produce inconsistent traces. Removal and replacement belong in the same change.

所有EDOT SDK通用。需设置恰好三个环境变量：

变量	值
`OTEL_SERVICE_NAME`	服务名称。该值将成为Elasticsearch中的 `service.name` ，也是其他所有可观测性技能查找该服务的依据。
`OTEL_EXPORTER_OTLP_ENDPOINT`	步骤3中解析的托管OTLP端点或EDOT Collector URL。
`OTEL_EXPORTER_OTLP_HEADERS`	凭证，格式为 `Authorization=ApiKey <key>` 或 `Authorization=Bearer <token>` 。

适用于所有语言的两条规则：

不要设置
OTEL_TRACES_EXPORTER
、
OTEL_METRICS_EXPORTER
或
OTEL_LOGS_EXPORTER
：默认值已正确。设置这些变量通常是服务仅上报链路追踪但未上报指标或日志的原因。
切勿在同一进程中同时运行传统Elastic APM Agent和EDOT：它们会重复埋点，生成不一致的链路追踪。移除和替换应在同一变更中完成。

Process: verify telemetry arrives

流程：验证遥测数据送达

Run this after every instrument or migrate change, and as the entry point when a user reports that an instrumented service is missing. Give the pipeline a minute or two after the deployment restarts before concluding anything.

Confirm documents are landing for the service. Query
```
POST /_query
```
, scoping to the service name that was set in
```
OTEL_SERVICE_NAME
```
. The decision: is anything arriving at all, and across which signals.
esql
```
FROM traces-*,metrics-*,logs-* METADATA _index
| WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
| STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY _index
| SORT docs DESC
| LIMIT 20
```
A healthy result has rows from a traces data stream, a metrics data stream, and a logs data stream, with
```
last_seen
```
inside the last minute or two. Missing signals are diagnostic, not cosmetic: traces only, with no metrics or logs, usually means an
```
OTEL_*_EXPORTER
```
variable was set.
If nothing came back, find out what name the service is reporting under. An empty result is ambiguous between "no telemetry at all" and "telemetry arriving under a different name" — a typo in
```
OTEL_SERVICE_NAME
```
, or the SDK default of
```
unknown_service
```
. Drop the service filter and look at what is actually arriving.
esql
```
FROM traces-*,metrics-*,logs-*
| WHERE @timestamp > NOW() - 15 minutes
| STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY service.name
| SORT docs DESC
| LIMIT 25
```
The decision: if the service appears under an unexpected name, fix
```
OTEL_SERVICE_NAME
```
. If it does not appear at all, the exporter is not reaching the destination — go back to step 3 of the routing process and re-check the endpoint and credential.
Confirm EDOT is the reporter, not something else. This is what distinguishes a correct migration from a half-finished one, and vanilla OpenTelemetry from EDOT.
esql
```
FROM traces-*
| WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
| STATS spans = COUNT(*) BY telemetry.sdk.language, telemetry.sdk.name, telemetry.distro.name, telemetry.distro.version
| SORT spans DESC
| LIMIT 10
```
```
telemetry.distro.name
```
is
```
elastic
```
when an EDOT SDK is reporting. A null distro with
```
telemetry.sdk.name == "opentelemetry"
```
means the application is on the upstream OpenTelemetry SDK rather than EDOT. Two rows for one service — one with the Elastic distro and one without, or two different
```
telemetry.sdk.language
```
values — means more than one agent is attached, or an old deployment is still running.

Diagnose from the shape of the result.

Verification result	Most likely cause
No rows for the service, and the name is absent everywhere	Exporter never reached the destination — wrong endpoint or bad credential
Service appears under a different name	`OTEL_SERVICE_NAME` unset or misspelled; SDK fell back to its default
Traces only, no metrics or logs	An `OTEL_*_EXPORTER` variable was set and overrode the default
Rows present but `last_seen` is stale	The process exited, or the deployment was rolled back
Two distro values for one service	Classic agent and EDOT both attached, or an old replica still running

If the field names above are absent from the mapping, confirm what is available with

GET /<index>/_mapping

before concluding the telemetry is wrong.

每次完成埋点或迁移变更后执行此流程，也可作为用户报告已埋点服务缺失时的入口。部署重启后，请等待一两分钟再得出结论。

确认服务的文档已送达：执行
```
POST /_query
```
查询，范围限定为
```
OTEL_SERVICE_NAME
```
中设置的服务名称。决策：是否有任何数据送达，以及覆盖哪些信号。
esql
```
FROM traces-*,metrics-*,logs-* METADATA _index
| WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
| STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY _index
| SORT docs DESC
| LIMIT 20
```
健康结果应包含链路追踪数据流、指标数据流和日志数据流的行，且
```
last_seen
```
在最近一两分钟内。缺失信号具有诊断意义，而非 cosmetic：仅存在链路追踪、无指标或日志，通常意味着设置了
```
OTEL_*_EXPORTER
```
变量。
如果无结果返回，查明服务实际上报的名称：空结果在“完全无遥测数据”和“遥测数据以其他名称送达”之间存在歧义——
```
OTEL_SERVICE_NAME
```
拼写错误，或SDK默认使用
```
unknown_service
```
。移除服务过滤器，查看实际送达的数据。
esql
```
FROM traces-*,metrics-*,logs-*
| WHERE @timestamp > NOW() - 15 minutes
| STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY service.name
| SORT docs DESC
| LIMIT 25
```
决策：如果服务以意外名称出现，请修复
```
OTEL_SERVICE_NAME
```
。如果服务未出现，则说明导出器未到达目标地址——返回路由流程的步骤3，重新检查端点和凭证。
确认上报方是EDOT而非其他工具：这是区分正确迁移与未完成迁移、原生OpenTelemetry与EDOT的关键。
esql
```
FROM traces-*
| WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
| STATS spans = COUNT(*) BY telemetry.sdk.language, telemetry.sdk.name, telemetry.distro.name, telemetry.distro.version
| SORT spans DESC
| LIMIT 10
```
当使用EDOT SDK上报时，
```
telemetry.distro.name
```
为
```
elastic
```
。
```
telemetry.distro.name
```
为空且
```
telemetry.sdk.name == "opentelemetry"
```
意味着应用使用的是上游OpenTelemetry SDK而非EDOT。同一服务存在两行结果——一行带有Elastic发行版，一行没有，或两行的
```
telemetry.sdk.language
```
值不同——意味着挂载了多个Agent，或旧部署仍在运行。

根据结果形态诊断问题

验证结果	最可能的原因
服务无结果行，且名称未出现在任何位置	导出器从未到达目标地址——端点错误或凭证无效
服务以其他名称出现	`OTEL_SERVICE_NAME` 未设置或拼写错误；SDK使用默认值
仅存在链路追踪，无指标或日志	设置了 `OTEL_*_EXPORTER` 变量，覆盖了默认值
存在结果行但 `last_seen` 已过期	进程已退出，或部署已回滚
同一服务存在两个发行版值	同时挂载了传统Agent和EDOT，或旧副本仍在运行

如果上述字段名称在映射中不存在，请先通过

GET /<index>/_mapping

确认可用字段，再判定遥测数据存在问题。

Examples

示例

"Add observability to my Spring Boot service" — read

pom.xml

to confirm Java, check for

co.elastic.apm

and

ELASTIC_APM_*

markers, and finding none, take the instrument path in

references/java.md

. Add the

-javaagent

flag to the container entrypoint, set the three environment variables against the managed OTLP endpoint, then run the verification query and report which signals arrived.

"We're moving off the Elastic APM agent to EDOT, we're on .NET" — this is the migrate path. Remove the

Elastic.Apm.*

packages, the

AddAllElasticApm()

call, and the

ElasticApm

block from

appsettings.json

in the same change that adds

Elastic.OpenTelemetry

and

builder.AddElasticOpenTelemetry()

. Translate the configuration rather than copying it — in particular do not reuse

ELASTIC_APM_SERVER_URLS

as the OTLP endpoint. Verify with step 3 and confirm exactly one distro is reporting.

"I instrumented my Python app but nothing shows up in Elastic" — start at the verification process, not at the configuration. Run step 1; if it is empty, run step 2 to see whether the service is reporting under a different name. If the service is absent entirely, the two candidates are an unwrapped entrypoint (

opentelemetry-instrument

missing) and an endpoint still pointing at APM Server. Check the entrypoint first — it is the more common of the two.

"Our service sends traces but we have no logs or metrics" — this is the

OTEL_*_EXPORTER

signature. Confirm it with step 1, which will show a traces data stream and nothing else, then remove the

OTEL_TRACES_EXPORTER

OTEL_METRICS_EXPORTER

, or

OTEL_LOGS_EXPORTER

variable from the deployment and re-verify.

"为我的Spring Boot服务添加可观测性"——读取

pom.xml

确认是Java语言，检查

co.elastic.apm

和

ELASTIC_APM_*

标记，若未找到，则采用

references/java.md

中的全新埋点路径。在容器入口点添加

-javaagent

参数，针对托管OTLP端点设置三个环境变量，然后执行验证查询并报告哪些信号已送达。

"我们要从Elastic APM Agent迁移到EDOT，使用.NET"——这属于迁移路径。在同一变更中移除

Elastic.Apm.*

包、

AddAllElasticApm()

调用以及

appsettings.json

中的

ElasticApm

块，同时添加

Elastic.OpenTelemetry

和

builder.AddElasticOpenTelemetry()

。转换配置而非直接复制——尤其不要将

ELASTIC_APM_SERVER_URLS

复用为OTLP端点。通过步骤3验证，确认仅存在一个发行版的上报数据。

"我为Python应用埋点了，但Elastic中没有任何数据"——从验证流程开始，而非配置。执行步骤1；如果无结果，执行步骤2查看服务是否以其他名称上报。如果服务完全未出现，两个可能原因是入口点未包装（缺少

opentelemetry-instrument

）或端点仍指向APM Server。先检查入口点——这是更常见的原因。

"我们的服务发送链路追踪，但没有日志或指标"——这是

OTEL_*_EXPORTER

变量导致的典型问题。通过步骤1确认，结果将显示链路追踪数据流但无其他数据，然后从部署中移除

OTEL_TRACES_EXPORTER

、

OTEL_METRICS_EXPORTER

或

OTEL_LOGS_EXPORTER

变量并重新验证。

Guidelines

指南

Read the setup or migration guide for the language before editing. The per-language references link the authoritative page; the attach mechanism differs enough between runtimes that pattern-matching from another language produces broken configurations.
Removal and replacement go in one change. A window where both the classic agent and EDOT are attached produces double-instrumented traces that are worse than either alone.
Never carry the classic server URL forward.
```
ELASTIC_APM_SERVER_URL
```
points at APM Server;
```
OTEL_EXPORTER_OTLP_ENDPOINT
```
must point at the managed OTLP endpoint or an EDOT Collector.
Set the three variables and no more. Additional
```
OTEL_*
```
tuning is occasionally justified, but the exporter variables specifically should be left at their defaults.
Treat an empty verification result as unknown, not as failure of a specific component. Distinguish the causes with step 2 before recommending a fix.
Never place a credential in the skill's output or in a committed file.
```
OTEL_EXPORTER_OTLP_HEADERS
```
belongs in the deployment's secret mechanism.
Attribution: the per-language EDOT guidance consolidated here originates with the
```
apm-agent-devs
```
team.

编辑前阅读语言对应的设置或迁移指南：各语言参考文档链接了权威页面；不同运行时的挂载机制差异较大，照搬其他语言的模式会生成无效配置。
移除和替换在同一变更中完成：同时挂载传统Agent和EDOT的窗口期会生成重复埋点的链路追踪，效果比单独使用任何一种Agent都差。
切勿沿用传统服务器URL：
```
ELASTIC_APM_SERVER_URL
```
指向APM Server；
```
OTEL_EXPORTER_OTLP_ENDPOINT
```
必须指向托管OTLP端点或EDOT Collector。
仅设置三个变量：偶尔需要额外的
```
OTEL_*
```
调优，但导出器变量应保留默认值。
将空验证结果视为未知，而非特定组件故障：通过步骤2区分原因后再推荐修复方案。
切勿在技能输出或已提交文件中放置凭证：
```
OTEL_EXPORTER_OTLP_HEADERS
```
应放在部署的机密机制中。
致谢：此处整合的各语言EDOT指南源自
```
apm-agent-devs
```
团队。

Operations

操作

HTTP API (shorthand)	`elastic` CLI command
`GET /`	`elastic es info`
`POST /_query`	`elastic es esql query --format tsv --query '<esql>'`
`GET /<index>/_mapping`	`elastic es indices get-mapping --index '<index>'`

HTTP API (简写)	`elastic` CLI命令
`GET /`	`elastic es info`
`POST /_query`	`elastic es esql query --format tsv --query '<esql>'`
`GET /<index>/_mapping`	`elastic es indices get-mapping --index '<index>'`