Diagnose Cluster Health

Triage a non-green Elasticsearch cluster read-only: localize the problem, classify the allocation decider, and report the single most likely cause with remediation. Never mutate cluster state — surface findings and let the operator act.

Environment Configuration

This skill executes Elasticsearch operations through the

elastic

CLI. If the

elastic

CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g.,

GET /

GET /_cat/indices

GET /{index}/_mapping

GET /{index}/_settings/index.mode

POST /_query

). The Operations table at the end of this document maps each shorthand to the equivalent

elastic

CLI command — always use the CLI rather than calling the HTTP API directly.

Process

Read the overall status. Call
```
GET /_cluster/health
```
. The
```
status
```
field is the verdict:
- ```
green
```
  — every primary and replica is assigned. Report healthy and stop.
- ```
yellow
```
  — every primary is assigned but at least one replica is not. Data remains readable; redundancy is degraded. This is not data loss.
- ```
red
```
  — at least one primary is unassigned. Data for that shard is unavailable; treat as urgent.
Also read
```
unassigned_shards
```
,
```
initializing_shards
```
, and
```
relocating_shards
```
. The decision: continue only when status is yellow or red. If
```
initializing_shards > 0
```
and
```
unassigned_shards == 0
```
, the cluster is recovering on its own — call
```
GET /_cat/recovery
```
to confirm progress, wait, and re-check
```
GET /_cluster/health
```
before escalating.
Data needed: cluster-wide
```
status
```
and shard counters.
Localize the problem to one index. Call
```
GET /_cluster/health?level=indices
```
and pick the index that drives the cluster-wide status:
- Any red index outranks every yellow index.
- Among reds or yellows, prefer the index with the most
```
unassigned_shards
```
  .
- A red system index (
```
.security
```
  ,
```
.kibana*
```
  ,
```
.fleet-*
```
  ) outranks application indices because the rest of the stack depends on it.
Optionally call
```
GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason
```
to list every unassigned shard on that index and see whether failures are primaries (
```
prirep=p
```
) or replicas (
```
prirep=r
```
).
The decision: focus the next steps on exactly one index — the one whose recovery unblocks the cluster.
Data needed: per-index
```
status
```
and
```
unassigned_shards
```
; shard role (primary vs replica) when available.
Separate trigger from root cause. Call
```
POST /_cluster/allocation/explain
```
with no body so Elasticsearch selects an unassigned shard, or target the worst shard explicitly:
json
```
{ "index": "<index>", "shard": <id>, "primary": <true|false> }
```
Read these fields in order:
- ```
primary
```
  —
```
false
```
  means a replica is unassigned (typical yellow);
```
true
```
  means a primary is unassigned (typical red).
- ```
can_allocate
```
  — top-level allocation verdict (
```
no
```
  ,
```
yes
```
  ,
```
throttled
```
  ,
```
no_valid_shard_copy
```
  , …).
- ```
unassigned_info.reason
```
  — what triggered reassignment (e.g.
```
NODE_LEFT
```
  ,
```
INDEX_CREATED
```
  ). This is not the root cause when
```
can_allocate
```
  is
```
no
```
  ; it only explains why the shard became unassigned.
- ```
allocate_explanation
```
  — human-readable summary; quote it verbatim in the report.
- ```
node_allocation_decisions[].deciders[]
```
  — per-node decider results. Find deciders with
```
decision: "NO"
```
  ; the decider name (e.g.
```
disk_threshold
```
  ,
```
filter
```
  ,
```
awareness
```
  ) is the root cause class.
The decision:
- Yellow +
  primary: false
  — impact is limited to replica redundancy; no data loss. Continue to step 4 to name the blocking decider (do not stop at
```
NODE_LEFT
```
  ).
- Red +
  primary: true
  — data for that shard is missing. Continue to step 4; if
```
can_allocate
```
  is
```
no_valid_shard_copy
```
  , treat as potential data loss immediately.
Data needed: allocation-explain response for one representative unassigned shard on the chosen index.

Classify the decider. Map the blocking signal to a cause class. Prefer the decider with

decision: "NO"

over the

unassigned_info.reason

trigger.

Signal	Cause class	Typical remediation (operator applies)
`decider: disk_threshold` , `decision: NO`	Disk high/low watermark exceeded	Free disk on the named node, add data-node capacity, or adjust `cluster.routing.allocation.disk.watermark.*` after confirming usage via `GET /_cat/allocation`
`decider: filter` or `decider: awareness` , `decision: NO`	Allocation filtering or zone awareness	Add a node that satisfies `index.routing.allocation.*` / awareness attributes, or adjust index/cluster allocation settings
`decider: throttling` or recovery in progress	Transient recovery	Wait; monitor `GET /_cat/recovery` and re-check `GET /_cluster/health`
`can_allocate: no_valid_shard_copy` (often with empty `node_allocation_decisions` )	No surviving shard copy	See step 5 — data loss scenario
`can_allocate: yes` but shard still unassigned	Delayed allocation or cluster state catch-up	Check `unassigned_info.at` delay; wait and re-check

For disk pressure (common yellow scenario after

NODE_LEFT

): replicas relocate to remaining nodes; if a survivor is above the high watermark (

cluster.routing.allocation.disk.watermark.high

, default 90%), the

disk_threshold

decider blocks replica allocation even though primaries stay assigned. The fix is disk capacity or watermark relief — not deleting the index or forcing an empty primary.

Data needed: decider name,

explanation

text, and affected node names from

node_allocation_decisions

Recommend remediation — read-only triage ends here. Report the single most likely cause (decider class + verbatim
```
allocate_explanation
```
) and one primary remediation path. Match urgency to color and shard role.
Yellow / replica unassigned (no data loss):
- State clearly: all primaries are assigned; only replicas are missing; no data loss.
- Name the real decider (e.g. disk high watermark on
```
es-node-2
```
  ), not merely “a node left”.
- Recommend: free disk space, expand storage, add data nodes, or adjust disk watermarks after reviewing
```
GET /_cat/allocation
```
  .
- Do not recommend: deleting the index,
```
allocate_empty_primary
```
  , force-allocating over a healthy primary, or restarting the entire cluster without evidence.
Red / primary unassigned with
no_valid_shard_copy
(data loss risk):
- State clearly: a primary shard is unassigned; queries/routing for that shard fail; treat as urgent and localized to the named index.
- Explain: the only copy was on the departed node; Elasticsearch cannot allocate a primary because no valid copy exists on any remaining node (
```
can_allocate: no_valid_shard_copy
```
  ).
- Recovery paths in order:
  1. Bring the departed node back if its data directory is intact — the shard copy returns.
  2. Restore from snapshot into the index (or a new index followed by reindex) when snapshots exist.
  3. Last resort only:
    POST /_cluster/reroute
    with
    allocate_empty_primary
    — this creates an empty primary and permanently loses all documents on that shard. State data loss explicitly; never present this as the first or casual fix.
- Do not recommend: deleting the index without discussing data loss, or
```
allocate_empty_primary
```
  without the data-loss warning.
Self-healing in progress:
- When deciders show throttling or active peer recovery, recommend waiting and re-checking read-only APIs above.
Do not execute reroutes, snapshot restores, or settings changes — surface cause and remediation only.

Guidelines

Read-only: Use only GET/POST explain APIs for triage. Remediation is advice; the operator performs writes.
Trigger ≠ cause:
```
unassigned_info.reason: NODE_LEFT
```
explains the event;
```
node_allocation_decisions
```
deciders explain why allocation still fails.
Replica vs primary: Yellow +
```
primary: false
```
= redundancy gap, not data loss. Red +
```
primary: true
```
= missing data for that shard.
One index, one cause: Pick the highest-impact index and the strongest NO decider; avoid listing every shard.
Cat helpers: Use
```
GET /_cat/allocation
```
for disk percentages per node and
```
GET /_cat/recovery
```
for ongoing recoveries when the decider class is unclear or recovery is in progress.

Examples

Yellow — disk watermark after node departure. Health shows yellow with unassigned replicas on

logs-2025-07

. Allocation explain returns

primary: false

unassigned_info.reason: NODE_LEFT

, but

disk_threshold

decider NO on

es-node-2

(“above the high watermark … 90%”). Report: no data loss; root cause is disk pressure on the receiving node; remediate disk/watermark — not “node left” alone.

Red — primary with no valid copy. Health shows red on

orders-2025

with one unassigned shard. Explain returns

primary: true

can_allocate: no_valid_shard_copy

last_allocation_status: no_valid_shard_copy

. Report: urgent; primary data missing; restore node or snapshot; mention

allocate_empty_primary

only as last resort with explicit data loss.

Operations

HTTP API (shorthand)	`elastic` CLI command
`GET /_cluster/health`	`elastic es cluster health`
`GET /_cluster/health?level=indices`	`elastic es cluster health --level indices`
`POST /_cluster/allocation/explain`	`elastic es cluster allocation-explain`
`POST /_cluster/allocation/explain` (specific shard)	`elastic es cluster allocation-explain --index '<index>' --shard <id> --primary true` (replica: `false` )
`GET /_cat/allocation`	`elastic es cat allocation`
`GET /_cat/recovery`	`elastic es cat recovery`
`GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason`	`elastic es cat shards --index '<index>' --h index,shard,prirep,state,unassigned.reason`
`POST /_cluster/reroute` (last-resort empty primary — operator only)	`elastic es cluster reroute --commands '<json>'`

elasticsearch-cluster-health

NPX Install

Tags

SKILL.md Content