elasticsearch-cluster-health

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Diagnose Cluster Health

诊断集群健康状态

Triage a non-green Elasticsearch cluster read-only: localize the problem, classify the allocation decider, and report the single most likely cause with remediation. Never mutate cluster state — surface findings and let the operator act.

对非绿色状态的Elasticsearch集群进行只读分类排查：定位问题、分类分配决策器，并报告最可能的问题原因及修复建议。切勿修改集群状态——仅呈现排查结果，由操作员执行操作。

Environment Configuration

环境配置

This skill executes Elasticsearch operations through the

elastic

CLI. If the

elastic

CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g.,

GET /

GET /_cat/indices

GET /{index}/_mapping

GET /{index}/_settings/index.mode

POST /_query

). The Operations table at the end of this document maps each shorthand to the equivalent

elastic

CLI command — always use the CLI rather than calling the HTTP API directly.

本技能通过

elastic

CLI执行Elasticsearch操作。如果未安装

elastic

CLI，请告知用户其用途。请勿猜测凭证、直接调用HTTP API或尝试其他变通方法。

本技能使用HTTP简写形式引用操作（例如：

GET /

、

GET /_cat/indices

、

GET /{index}/_mapping

、

GET /{index}/_settings/index.mode

、

POST /_query

）。本文档末尾的操作表格将每个简写映射为对应的

elastic

CLI命令——请始终使用CLI而非直接调用HTTP API。

Process

流程

Read the overall status. Call
```
GET /_cluster/health
```
. The
```
status
```
field is the verdict:
- ```
green
```
  — every primary and replica is assigned. Report healthy and stop.
- ```
yellow
```
  — every primary is assigned but at least one replica is not. Data remains readable; redundancy is degraded. This is not data loss.
- ```
red
```
  — at least one primary is unassigned. Data for that shard is unavailable; treat as urgent.
Also read
```
unassigned_shards
```
,
```
initializing_shards
```
, and
```
relocating_shards
```
. The decision: continue only when status is yellow or red. If
```
initializing_shards > 0
```
and
```
unassigned_shards == 0
```
, the cluster is recovering on its own — call
```
GET /_cat/recovery
```
to confirm progress, wait, and re-check
```
GET /_cluster/health
```
before escalating.
Data needed: cluster-wide
```
status
```
and shard counters.
Localize the problem to one index. Call
```
GET /_cluster/health?level=indices
```
and pick the index that drives the cluster-wide status:
- Any red index outranks every yellow index.
- Among reds or yellows, prefer the index with the most
```
unassigned_shards
```
  .
- A red system index (
```
.security
```
  ,
```
.kibana*
```
  ,
```
.fleet-*
```
  ) outranks application indices because the rest of the stack depends on it.
Optionally call
```
GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason
```
to list every unassigned shard on that index and see whether failures are primaries (
```
prirep=p
```
) or replicas (
```
prirep=r
```
).
The decision: focus the next steps on exactly one index — the one whose recovery unblocks the cluster.
Data needed: per-index
```
status
```
and
```
unassigned_shards
```
; shard role (primary vs replica) when available.
Separate trigger from root cause. Call
```
POST /_cluster/allocation/explain
```
with no body so Elasticsearch selects an unassigned shard, or target the worst shard explicitly:
json
```
{ "index": "<index>", "shard": <id>, "primary": <true|false> }
```
Read these fields in order:
- ```
primary
```
  —
```
false
```
  means a replica is unassigned (typical yellow);
```
true
```
  means a primary is unassigned (typical red).
- ```
can_allocate
```
  — top-level allocation verdict (
```
no
```
  ,
```
yes
```
  ,
```
throttled
```
  ,
```
no_valid_shard_copy
```
  , …).
- ```
unassigned_info.reason
```
  — what triggered reassignment (e.g.
```
NODE_LEFT
```
  ,
```
INDEX_CREATED
```
  ). This is not the root cause when
```
can_allocate
```
  is
```
no
```
  ; it only explains why the shard became unassigned.
- ```
allocate_explanation
```
  — human-readable summary; quote it verbatim in the report.
- ```
node_allocation_decisions[].deciders[]
```
  — per-node decider results. Find deciders with
```
decision: "NO"
```
  ; the decider name (e.g.
```
disk_threshold
```
  ,
```
filter
```
  ,
```
awareness
```
  ) is the root cause class.
The decision:
- Yellow +
  primary: false
  — impact is limited to replica redundancy; no data loss. Continue to step 4 to name the blocking decider (do not stop at
```
NODE_LEFT
```
  ).
- Red +
  primary: true
  — data for that shard is missing. Continue to step 4; if
```
can_allocate
```
  is
```
no_valid_shard_copy
```
  , treat as potential data loss immediately.
Data needed: allocation-explain response for one representative unassigned shard on the chosen index.

Classify the decider. Map the blocking signal to a cause class. Prefer the decider with

decision: "NO"

over the

unassigned_info.reason

trigger.

Signal	Cause class	Typical remediation (operator applies)
`decider: disk_threshold` , `decision: NO`	Disk high/low watermark exceeded	Free disk on the named node, add data-node capacity, or adjust `cluster.routing.allocation.disk.watermark.*` after confirming usage via `GET /_cat/allocation`
`decider: filter` or `decider: awareness` , `decision: NO`	Allocation filtering or zone awareness	Add a node that satisfies `index.routing.allocation.*` / awareness attributes, or adjust index/cluster allocation settings
`decider: throttling` or recovery in progress	Transient recovery	Wait; monitor `GET /_cat/recovery` and re-check `GET /_cluster/health`
`can_allocate: no_valid_shard_copy` (often with empty `node_allocation_decisions` )	No surviving shard copy	See step 5 — data loss scenario
`can_allocate: yes` but shard still unassigned	Delayed allocation or cluster state catch-up	Check `unassigned_info.at` delay; wait and re-check

For disk pressure (common yellow scenario after

NODE_LEFT

): replicas relocate to remaining nodes; if a survivor is above the high watermark (

cluster.routing.allocation.disk.watermark.high

, default 90%), the

disk_threshold

decider blocks replica allocation even though primaries stay assigned. The fix is disk capacity or watermark relief — not deleting the index or forcing an empty primary.

Data needed: decider name,

explanation

text, and affected node names from

node_allocation_decisions

Recommend remediation — read-only triage ends here. Report the single most likely cause (decider class + verbatim
```
allocate_explanation
```
) and one primary remediation path. Match urgency to color and shard role.
Yellow / replica unassigned (no data loss):
- State clearly: all primaries are assigned; only replicas are missing; no data loss.
- Name the real decider (e.g. disk high watermark on
```
es-node-2
```
  ), not merely “a node left”.
- Recommend: free disk space, expand storage, add data nodes, or adjust disk watermarks after reviewing
```
GET /_cat/allocation
```
  .
- Do not recommend: deleting the index,
```
allocate_empty_primary
```
  , force-allocating over a healthy primary, or restarting the entire cluster without evidence.
Red / primary unassigned with
no_valid_shard_copy
(data loss risk):
- State clearly: a primary shard is unassigned; queries/routing for that shard fail; treat as urgent and localized to the named index.
- Explain: the only copy was on the departed node; Elasticsearch cannot allocate a primary because no valid copy exists on any remaining node (
```
can_allocate: no_valid_shard_copy
```
  ).
- Recovery paths in order:
  1. Bring the departed node back if its data directory is intact — the shard copy returns.
  2. Restore from snapshot into the index (or a new index followed by reindex) when snapshots exist.
  3. Last resort only:
```
POST /_cluster/reroute
```
    with
```
allocate_empty_primary
```
    — this creates an empty primary and permanently loses all documents on that shard. State data loss explicitly; never present this as the first or casual fix.
- Do not recommend: deleting the index without discussing data loss, or
```
allocate_empty_primary
```
  without the data-loss warning.
Self-healing in progress:
- When deciders show throttling or active peer recovery, recommend waiting and re-checking read-only APIs above.
Do not execute reroutes, snapshot restores, or settings changes — surface cause and remediation only.

读取整体状态。调用
```
GET /_cluster/health
```
。
```
status
```
字段是判断依据：
- ```
green
```
  —— 所有主分片和副本分片均已分配。报告集群健康并停止操作。
- ```
yellow
```
  —— 所有主分片已分配，但至少有一个副本分片未分配。数据仍可读取；冗余性下降。这不属于数据丢失。
- ```
red
```
  —— 至少有一个主分片未分配。该分片的数据不可用；需视为紧急情况。
同时读取
```
unassigned_shards
```
、
```
initializing_shards
```
和
```
relocating_shards
```
字段。决策逻辑：仅当状态为黄色或红色时继续排查。如果
```
initializing_shards > 0
```
且
```
unassigned_shards == 0
```
，说明集群正在自行恢复——调用
```
GET /_cat/recovery
```
确认进度，等待后重新调用
```
GET /_cluster/health
```
再判断是否需要升级处理。
所需数据：集群级别的
```
status
```
和分片统计数据。
将问题定位到单个索引。调用
```
GET /_cluster/health?level=indices
```
，选择影响集群整体状态的索引：
- 任何红色状态的索引优先级高于黄色状态的索引。
- 在红色或黄色状态的索引中，优先选择
```
unassigned_shards
```
  数量最多的索引。
- 红色状态的系统索引（
```
.security
```
  、
```
.kibana*
```
  、
```
.fleet-*
```
  ）优先级高于业务索引，因为整个技术栈依赖这些系统索引。
可选择性调用
```
GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason
```
，列出该索引下所有未分配的分片，查看失败的是主分片（
```
prirep=p
```
）还是副本分片（
```
prirep=r
```
）。
决策逻辑：将后续步骤的焦点完全集中在一个索引上——恢复该索引即可解除集群阻塞。
所需数据：各索引的
```
status
```
和
```
unassigned_shards
```
；分片角色（主分片/副本分片，如有）。
区分触发事件与根本原因。调用不带请求体的
```
POST /_cluster/allocation/explain
```
，让Elasticsearch自动选择一个未分配分片；或明确指定问题最严重的分片：
json
```
{ "index": "<index>", "shard": <id>, "primary": <true|false> }
```
按顺序读取以下字段：
- ```
primary
```
  ——
```
false
```
  表示副本分片未分配（典型黄色状态场景）；
```
true
```
  表示主分片未分配（典型红色状态场景）。
- ```
can_allocate
```
  —— 顶级分配决策结果（
```
no
```
  、
```
yes
```
  、
```
throttled
```
  、
```
no_valid_shard_copy
```
  等）。
- ```
unassigned_info.reason
```
  —— 触发重新分配的事件（例如
```
NODE_LEFT
```
  、
```
INDEX_CREATED
```
  ）。当
```
can_allocate
```
  为
```
no
```
  时，这不是根本原因；它仅解释分片为何变为未分配状态。
- ```
allocate_explanation
```
  —— 人类可读的摘要；在报告中直接引用原文。
- ```
node_allocation_decisions[].deciders[]
```
  —— 各节点的决策器结果。找到
```
decision: "NO"
```
  的决策器；决策器名称（例如
```
disk_threshold
```
  、
```
filter
```
  、
```
awareness
```
  ）即为根本原因类别。
决策逻辑：
- 黄色状态 +
  primary: false
  —— 影响仅限于副本冗余；无数据丢失。继续执行步骤4以确定阻塞的决策器（切勿在
```
NODE_LEFT
```
  处停止排查）。
- 红色状态 +
  primary: true
  —— 该分片的数据丢失。继续执行步骤4；如果
```
can_allocate
```
  为
```
no_valid_shard_copy
```
  ，需立即视为潜在数据丢失场景。
所需数据：所选索引上一个代表性未分配分片的分配解释响应。

分类决策器。将阻塞信号映射到原因类别。优先选择

decision: "NO"

的决策器，而非

unassigned_info.reason

中的触发事件。

信号	原因类别	典型修复方案（由操作员执行）
`decider: disk_threshold` , `decision: NO`	磁盘高低水位线超出阈值	在指定节点释放磁盘空间、添加数据节点容量，或通过 `GET /_cat/allocation` 确认使用情况后调整 `cluster.routing.allocation.disk.watermark.*` 参数
`decider: filter` 或 `decider: awareness` , `decision: NO`	分配过滤或区域感知	添加满足 `index.routing.allocation.*` /感知属性的节点，或调整索引/集群的分配设置
`decider: throttling` 或恢复正在进行中	临时恢复	等待；通过 `GET /_cat/recovery` 监控进度，并重新调用 `GET /_cluster/health` 检查状态
`can_allocate: no_valid_shard_copy` （通常伴随空的 `node_allocation_decisions` ）	无存活分片副本	查看步骤5 —— 数据丢失场景
`can_allocate: yes` 但分片仍未分配	延迟分配或集群状态同步延迟	检查 `unassigned_info.at` 延迟时间；等待后重新检查

对于磁盘压力（

NODE_LEFT

后的常见黄色状态场景）：副本分片会迁移到剩余节点；如果某个存活节点的磁盘使用率超过高水位线（

cluster.routing.allocation.disk.watermark.high

，默认90%），

disk_threshold

决策器会阻止副本分片分配，即使主分片仍处于已分配状态。修复方案是增加磁盘容量或调整水位线——不要删除索引或强制分配空主分片。

所需数据：决策器名称、

explanation

文本，以及

node_allocation_decisions

中的受影响节点名称。

推荐修复方案——只读分类排查到此结束。报告最可能的单一原因（决策器类别 + 原文引用的
```
allocate_explanation
```
）和一条主要修复路径。根据状态颜色和分片角色匹配紧急程度。
黄色状态 / 副本分片未分配（无数据丢失）：
- 明确说明：所有主分片均已分配；仅副本分片缺失；无数据丢失。
- 指明真正的决策器原因（例如
```
es-node-2
```
  节点磁盘高水位线超出），而非仅提及“节点离开”。
- 推荐方案：释放磁盘空间、扩展存储、添加数据节点，或查看
```
GET /_cat/allocation
```
  后调整磁盘水位线。
- 不推荐：删除索引、执行
```
allocate_empty_primary
```
  、在健康主分片上强制分配，或无证据地重启整个集群。
红色状态 / 主分片未分配且
no_valid_shard_copy
（存在数据丢失风险）：
- 明确说明：主分片未分配；该分片的查询/路由失败；需视为紧急情况，且问题局限于指定索引。
- 解释：该分片的唯一副本位于已下线的节点；Elasticsearch无法分配主分片，因为剩余节点上没有有效的副本（
```
can_allocate: no_valid_shard_copy
```
  ）。
- 恢复路径优先级：
  1. 恢复下线节点（如果其数据目录完好）——分片副本将恢复。
  2. 从快照恢复到该索引（或恢复到新索引后重新索引）（如果存在快照）。
  3. 仅作为最后手段： 执行
```
POST /_cluster/reroute
```
    并使用
```
allocate_empty_primary
```
    —— 这会创建空主分片并永久丢失该分片上的所有文档。需明确说明数据丢失风险；切勿将此作为首选或随意的修复方案。
- 不推荐：未讨论数据丢失风险就删除索引，或未给出数据丢失警告就执行
```
allocate_empty_primary
```
  。
集群正在自行恢复：
- 当决策器显示限流或对等恢复正在进行时，建议等待并重新调用上述只读API检查状态。
请勿执行路由调整、快照恢复或设置更改——仅呈现问题原因和修复建议。

Guidelines

指导原则

Read-only: Use only GET/POST explain APIs for triage. Remediation is advice; the operator performs writes.
Trigger ≠ cause:
```
unassigned_info.reason: NODE_LEFT
```
explains the event;
```
node_allocation_decisions
```
deciders explain why allocation still fails.
Replica vs primary: Yellow +
```
primary: false
```
= redundancy gap, not data loss. Red +
```
primary: true
```
= missing data for that shard.
One index, one cause: Pick the highest-impact index and the strongest NO decider; avoid listing every shard.
Cat helpers: Use
```
GET /_cat/allocation
```
for disk percentages per node and
```
GET /_cat/recovery
```
for ongoing recoveries when the decider class is unclear or recovery is in progress.

只读操作： 仅使用GET/POST解释API进行分类排查。修复方案仅为建议；由操作员执行写入操作。
触发事件 ≠ 根本原因：
```
unassigned_info.reason: NODE_LEFT
```
解释了事件本身；
```
node_allocation_decisions
```
中的决策器解释了分配仍失败的原因。
副本与主分片的区别： 黄色状态 +
```
primary: false
```
= 冗余缺口，而非数据丢失。红色状态 +
```
primary: true
```
= 该分片的数据丢失。
单一索引，单一原因： 选择影响最大的索引和最明确的NO决策器；避免列出所有分片。
Cat工具辅助： 当决策器类别不明确或恢复正在进行时，使用
```
GET /_cat/allocation
```
查看各节点磁盘使用率，使用
```
GET /_cat/recovery
```
监控正在进行的恢复操作。

Examples

示例

Yellow — disk watermark after node departure. Health shows yellow with unassigned replicas on

logs-2025-07

. Allocation explain returns

primary: false

unassigned_info.reason: NODE_LEFT

, but

disk_threshold

decider NO on

es-node-2

(“above the high watermark … 90%”). Report: no data loss; root cause is disk pressure on the receiving node; remediate disk/watermark — not “node left” alone.

Red — primary with no valid copy. Health shows red on

orders-2025

with one unassigned shard. Explain returns

primary: true

can_allocate: no_valid_shard_copy

last_allocation_status: no_valid_shard_copy

. Report: urgent; primary data missing; restore node or snapshot; mention

allocate_empty_primary

only as last resort with explicit data loss.

黄色状态 —— 节点下线后磁盘水位线超出。集群健康状态显示黄色，

logs-2025-07

索引存在未分配副本。分配解释返回

primary: false

、

unassigned_info.reason: NODE_LEFT

，但

es-node-2

节点的

disk_threshold

决策器返回NO（“超出高水位线……90%”）。报告内容：无数据丢失；根本原因是接收节点的磁盘压力；修复方案为释放磁盘空间或调整水位线——而非仅提及“节点离开”。

红色状态 —— 主分片无有效副本。集群健康状态显示

orders-2025

索引为红色，存在一个未分配分片。分配解释返回

primary: true

、

can_allocate: no_valid_shard_copy

、

last_allocation_status: no_valid_shard_copy

。报告内容：紧急情况；主分片数据丢失；恢复节点或从快照恢复；仅在最后手段时提及

allocate_empty_primary

并明确说明数据丢失风险。

Operations

操作

HTTP API (shorthand)	`elastic` CLI command
`GET /_cluster/health`	`elastic es cluster health`
`GET /_cluster/health?level=indices`	`elastic es cluster health --level indices`
`POST /_cluster/allocation/explain`	`elastic es cluster allocation-explain`
`POST /_cluster/allocation/explain` (specific shard)	`elastic es cluster allocation-explain --index '<index>' --shard <id> --primary true` (replica: `false` )
`GET /_cat/allocation`	`elastic es cat allocation`
`GET /_cat/recovery`	`elastic es cat recovery`
`GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason`	`elastic es cat shards --index '<index>' --h index,shard,prirep,state,unassigned.reason`
`POST /_cluster/reroute` (last-resort empty primary — operator only)	`elastic es cluster reroute --commands '<json>'`

HTTP API（简写形式）	`elastic` CLI命令
`GET /_cluster/health`	`elastic es cluster health`
`GET /_cluster/health?level=indices`	`elastic es cluster health --level indices`
`POST /_cluster/allocation/explain`	`elastic es cluster allocation-explain`
`POST /_cluster/allocation/explain` （指定分片）	`elastic es cluster allocation-explain --index '<index>' --shard <id> --primary true` （副本分片使用 `false` ）
`GET /_cat/allocation`	`elastic es cat allocation`
`GET /_cat/recovery`	`elastic es cat recovery`
`GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason`	`elastic es cat shards --index '<index>' --h index,shard,prirep,state,unassigned.reason`
`POST /_cluster/reroute` （最后手段：空主分片——仅操作员执行）	`elastic es cluster reroute --commands '<json>'`