Elasticsearch Troubleshooting

Every cluster hits trouble sooner or later. A node runs out of disk, a shard refuses to start, or writes begin to fail with confusing messages. A calm, ordered approach solves most cases in minutes. Elasticsearch includes APIs that explain what went wrong, so guessing is rarely needed.

Start with Cluster Health

GET /_cluster/health

Status colors:

  GREEN   All primary and replica shards are active
  YELLOW  All primaries active, some replicas missing
  RED     At least one primary shard is missing, some data unavailable

Status	Searches	Urgency
Green	Fully available	None
Yellow	Work, no redundancy for some data	Moderate
Red	Missing data in some indexes	High

A single-node learning cluster often shows yellow. Replicas cannot live on the same node as their primaries, so they stay unassigned. Adding a second node or setting replicas to zero turns it green.

Find the Unhealthy Indexes

GET /_cluster/health?level=indices
GET /_cat/indices?v&health=red
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason

The last request lists each shard with its state and the reason it could not start. Words such as NODE_LEFT, ALLOCATION_FAILED, and INDEX_CREATED point to different causes.

Ask Elasticsearch Why

The allocation explain API answers the question directly. Without a body, it explains the first unassigned shard it finds.

GET /_cluster/allocation/explain

Typical explanation lines:

  "the node is above the high disk watermark"
  "a copy of this shard is already allocated to this node"
  "node does not match index setting [index.routing.allocation.require]"

Common Causes of Unassigned Shards

Cause	Clue	Fix
Disk almost full	Watermark message	Free space, delete old indexes, add storage
Node left the cluster	NODE_LEFT reason	Restart the node and check its logs
Not enough nodes for replicas	Same-shard message	Add nodes or lower the replica count
Allocation filter mismatch	Routing setting message	Correct the index or node attributes
Corrupt shard files	Repeated allocation failures	Restore from a snapshot

Disk Watermarks

Elasticsearch protects itself from full disks with three thresholds.

Disk usage      Behavior

  85%  low     No new shards placed on this node
  90%  high    Elasticsearch moves shards away from this node
  95%  flood   Indexes on the node become read-only for writes

  ┌─────────────────────────────────────────────────┐
  │ 0%                 85%       90%       95%  100%│
  │ normal ────────────low──────high──────flood     │
  └─────────────────────────────────────────────────┘

After freeing disk space, newer versions remove the read-only block automatically. Older setups require a manual release.

PUT /_all/_settings
{ "index.blocks.read_only_allow_delete": null }

Write Failures and Rejections

A response with status 429 means Elasticsearch rejected the request because its queues were full. The cluster protects itself from overload this way. Slow the sender down, shrink the bulk batch size, and retry with a pause between attempts.

GET /_cat/thread_pool/write?v&h=node_name,active,queue,rejected

A growing rejected count shows sustained overload. Add nodes, reduce indexing speed, or review expensive ingest pipelines.

Memory Problems

Circuit breaker errors appear when a request would use too much memory. The message names the breaker and the limit. Heavy aggregations on text fields and giant result sizes commonly trigger them.

GET /_nodes/stats/breaker
GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu

Heap usage that stays above 85% signals pressure. Reduce shard counts, trim field data usage, or add memory.

Stuck or Slow Tasks

GET /_cat/pending_tasks?v
GET /_tasks?actions=*search&detailed
GET /_nodes/hot_threads

Pending tasks reveal a busy master node. The hot threads report shows which operations burn the most CPU at that moment.

Troubleshooting Flow

Problem reported
      │
      ▼
GET /_cluster/health
      │
  ┌───┴───────────┬─────────────┐
 GREEN          YELLOW         RED
  │               │             │
Check slow      Check          GET /_cluster/
queries and     replicas and   allocation/explain
logs            node count     │
                               ▼
                       Fix cause: disk, node,
                       settings, or snapshot restore

Read the Logs

Elasticsearch writes detailed logs under the logs directory of each node. Search for the words ERROR and WARN around the time of the problem. Node-left messages, garbage collection warnings, and disk watermark notices appear there long before users complain. Keep a snapshot schedule running, since a restore resolves the worst cases of corrupt or lost shards.

Prevention Habits

Most emergencies announce themselves earlier. Monitor disk usage per node and alert at 75 percent. Watch heap usage, rejected thread pool counts, and the number of unassigned shards. Keep at least one replica for important indexes, and test snapshot restores on a schedule. A written runbook with the commands from this page shortens response time when a real incident arrives at night.

Previous lesson

Back to course

Next lesson