An ILM cookbook: data tiers without the surprise bill

Index lifecycle management is the mechanism that turns a retention decision — "keep 30 days hot, a year searchable, then delete" — into automated cluster behavior. Configured well, it is the single biggest cost lever on a logging cluster. Configured half-way, it quietly leaves everything on your most expensive storage forever. This is the recipe book we wish every cluster came with.

The mental model

An ILM policy is a sequence of phases — hot, warm, cold, frozen, delete — and each phase is a set of actions applied to an index once it qualifies. Two facts prevent most confusion:

1. min_age counts from rollover, not from index creation. An index that rolls over after 3 days with a warm min_age of 7 days enters warm at day 10 of its life.
2. Phases need somewhere to go. Warm/cold/frozen actions allocate shards to nodes carrying the matching data-tier role (data_warm, etc.). If no node has the role, indices qualify and then sit unallocated or stay put — the policy looks configured and does nothing.

On OpenSearch the equivalent is ISM: same ideas, different JSON, and the recipes below translate directly.

Recipe: the hot phase

Rollover is the only hot-phase action that matters:

"hot": {
  "actions": {
    "rollover": { "max_primary_shard_size": "45gb", "max_age": "30d" }
  }
}

Size first, age as a backstop. Skip max_docs unless you have a specific doc-count ceiling; every extra condition is another way to roll early and shrink your shards.

Recipe: the warm phase

Warm is where you bank the easy savings on data that is still queried but no longer written:

"warm": {
  "min_age": "3d",
  "actions": {
    "shrink": { "number_of_shards": 1 },
    "forcemerge": { "max_num_segments": 1 },
    "allocate": { "number_of_replicas": 1 }
  }
}

Shrink collapses ingest-driven primary counts you no longer need; force merge to one segment cuts disk and speeds queries; and warm nodes can run cheaper hardware with more disk per node — the read path tolerates it.

Recipe: cold and frozen

The cold phase can drop replicas entirely if you have snapshots to restore from — halving storage for data queried rarely. The frozen phase changes the economics completely: with searchable snapshots (Elastic license feature), the index lives in object storage and nodes cache only what queries touch. S3-class storage per gigabyte costs a small fraction of hot SSD, which is why a 90-day investigation tail that nobody queries most weeks belongs in frozen, not hot. On OpenSearch, the analogous capability is searchable snapshots / UltraWarm-style tiering on the managed service — check what your platform and license actually include before designing around it.

Recipe: the delete phase

"delete": { "min_age": "365d", "actions": { "delete": {} } }

Two notes: make sure the number reflects a real retention decision per stream (debug logs and audit logs should not share a policy), and confirm deletes are actually happening — a delete phase blocked for months is a classic silent cost leak.

Debugging: why is my index stuck?

The explain API is the tool:

GET my-index/_ilm/explain

It names the current phase, action, and step. The usual culprits:

• Waiting on allocation — a tier action with no nodes carrying that tier role, or disk watermarks blocking relocation.
• Force merge crawling — merges are I/O-heavy; on underpowered warm nodes a large index can take hours. That is normal once, suspicious always.
• ILM's poll cadence — steps advance on an interval (default 10 minutes), so a policy is never instant; don't chase seconds.
• An error step — explain shows the failing step and error; after fixing the cause, POST my-index/_ilm/retry resumes it.

The cost math that motivates all this

Take a stream ingesting 200 GB a day with 365-day retention. Held entirely hot with a replica, that is roughly 146 TB of premium SSD at steady state. The same stream with 7 days hot, 23 warm on dense nodes at one replica, and the remaining 11 months in frozen object storage needs a fraction of the SSD and puts most of the data on the cheapest storage you can buy — same data, same searchability for the workflows that actually query the tail, dramatically different invoice.

Retention policy is an engineering decision expressed in ILM. Write it down per stream, encode it, and then check _ilm/explain often enough to know it is really running.