Elasticsearch Index Lifecycle

Time-based data loses value as it ages. Yesterday's logs get searched constantly. Last year's logs get searched rarely. Keeping everything on the fastest, most expensive storage wastes money, and keeping everything forever fills the disks. Index Lifecycle Management (ILM) moves indexes through stages automatically and deletes them when their time ends.

The Lifecycle Phases

An index passes through up to five phases. Each phase matches a different level of attention and a different type of hardware.

  HOT          WARM          COLD          FROZEN        DELETE
  ─────────    ─────────     ─────────     ─────────     ─────────
  Receives     Read often,   Read rarely,  Read very     Removed
  new data     no writes     cheaper disk  rarely        for good
  Fast disks   Slower disks  Slowest disks Snapshot-based

  Day 0        Day 7         Day 30        Day 90        Day 180
  ──────────────────────────── age of index ───────────────────→

Phase	Index Status	Typical Hardware
Hot	Accepts writes, searched often	Fast SSD nodes
Warm	Read-only, searched sometimes	Mid-range nodes
Cold	Read-only, searched rarely	Large, cheaper disks
Frozen	Searched occasionally from snapshots	Object storage
Delete	Removed	None

Smaller projects skip phases freely. A policy with only hot and delete phases is perfectly valid. Frozen storage and searchable snapshots belong to paid license tiers, so check your subscription before relying on them.

Rollover: Starting a New Index at the Right Moment

One endless index grows too large to manage. Rollover creates a fresh index when the current one reaches a size, age, or document limit. Writes move to the new index, and the old one stays searchable.

logs-000001  ──(reaches 50 GB or 7 days)──→  closed to writes
                                              │
                                              ▼
logs-000002  ←── new writes arrive here  ─────┘

Creating a Policy

A policy lists phases, the age when each begins, and the actions to run in each phase.

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

The min_age counts from the moment of rollover, not from the first write. Rollover time marks the start of the clock for later phases.

Common Actions

Action	Phase	Effect
rollover	Hot	Starts a new index when limits hit
shrink	Warm	Reduces the number of shards
forcemerge	Warm	Merges segments for faster reads and less disk use
allocate	Warm, Cold	Changes replica count or placement
readonly	Warm	Blocks writes
delete	Delete	Removes the index

Attaching the Policy with a Template

A template gives every new index the policy name. Combine this with the template topic from earlier in the course for a fully automatic setup.

PUT /_index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-policy",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

PUT /logs-000001
{
  "aliases": { "logs": { "is_write_index": true } }
}

Applications write to the alias "logs." Elasticsearch routes each write to the current write index and rolls over when the limits trigger.

Data Streams: The Simpler Route

Data streams wrap the alias and rollover setup into a single concept for append-only data. A template with a data_stream section creates one, and ILM handles the rest. New log and metric projects usually start here.

PUT /_index_template/app-stream
{
  "index_patterns": ["app-logs"],
  "data_stream": {},
  "template": { "settings": { "index.lifecycle.name": "logs-policy" } }
}

POST /app-logs/_doc
{ "@timestamp": "2026-11-04T10:00:00Z", "message": "Service started" }

Checking Progress

The explain API reveals the current phase, the action in progress, and any error that blocked a step.

GET /logs-000001/_ilm/explain

Output highlights:
  "phase": "warm"
  "action": "shrink"
  "step": "complete"

Policy checks run every ten minutes by default, so phase changes may lag the exact age slightly. A stuck index shows a failed step in the explain output, and the retry endpoint restarts it after the cause is fixed.

POST /logs-000001/_ilm/retry

Policy Design Tips

Match retention to business or legal needs, and write the number of days into the policy name or notes. Size rollover by shard size rather than only by age, since traffic varies from day to day. Test policies on a small index with short ages before applying them to production data.

Sizing the Phases

Estimate daily data volume first. A system that writes 20 GB per day fills a 50 GB shard in about two and a half days, so a seven-day age limit rarely triggers before the size limit does. Keep the policy honest by checking which limit fires most often, and adjust the numbers when the traffic pattern changes.