Lifecycle Goals: Rollover, Retention, Tiering, Shrink/Force-Merge-Like Actions, and Cost/Recovery Tradeoffs

Learning outcomes

AtlasMart stores application logs for incident response and audit. The expensive mistake is to begin with vendor JSON instead of a retention contract. This lesson starts with lifecycle intent and turns it into observable invariants that can later be mapped to ILM or ISM.

1. Lifecycle intent before policy syntax

A lifecycle policy is automation around the changing value and access pattern of an index. It does not make data “old” by itself. The design inputs are ingest rate, shard growth, recovery time objective, search frequency, query latency target, retention/compliance dates, snapshot/restore posture and storage economics.

Rollover changes the active write generation. Retention decides how long generations remain. Tiering places data on hardware/storage appropriate to expected access. Shrink creates a new index with fewer primary shards. Force merge compacts immutable Lucene segments. None of these actions is a backup.

2. The two lifecycle control planes

Elasticsearch ILM organizes indices into named phases—hot, warm, cold, frozen and delete—and executes product-defined actions within those phases. Data tiers are node roles and placement preferences, not generic strings called “hot” or “warm.” ILM can automatically inject migration behavior for warm/cold phases when appropriate.

OpenSearch ISM instead manages an index through exactly one named state at a time. A state contains ordered actions and transition conditions. The state names are yours; naming a state warm does not magically create an Elastic-style warm data tier. Placement is a separate allocation decision using OpenSearch-supported attributes/capabilities.

3. Cost and recovery are coupled

Moving older data to cheaper storage can lower cost but increase recovery/search latency. Shrinking can reduce shard overhead but concentrates recovery into larger units. Force merge can reduce segment count but generates heavy I/O and temporary disk amplification. Deletion lowers cost only after it destroys the local copy. Therefore a lifecycle design is a sequence of bounded risks, not a cleanup script.

AtlasMart lifecycle lab contract

The chapter uses a small log-like fixture so lifecycle transitions are observable without consuming substantial resources. The application writes through aliases rather than physical index names. That isolates the lifecycle contract from generation numbers and lets the write target change atomically.

The production requirement used for comparison is: roll active logs before any primary shard violates the platform’s measured recovery/latency envelope; keep recent data on the performance tier that meets the SLO; make older data cheaper only when recovery and query latency permit; and delete only after the retention/compliance date and required recoverability evidence are satisfied.

4. Tiny cross-product observation plan

# Before lifecycle actions
GET _cat/aliases/atlasmart-life-*?v
GET atlasmart-life-es-000001/_settings
GET atlasmart-life-os-000001/_settings

# Elasticsearch
GET atlasmart-life-es-000001/_ilm/explain?human

# OpenSearch
GET _plugins/_ism/explain/atlasmart-life-os-000001?show_policy=true

# Always pair policy state with index/storage evidence
GET _cat/indices/atlasmart-life-*?v&h=index,docs.count,store.size,pri,rep
GET _cat/segments/atlasmart-life-*?v

An explain response proves what the lifecycle controller believes it is doing; it does not prove backups exist, that an older tier meets p99, or that a force merge is harmless. Those require separate evidence.

5. Production judgment

Choose lifecycle automation only after defining the data owner, retention authority, backup owner, acceptable recovery time, failure domain, tier capacity and operational escalation path. Never let lifecycle deletion outrun snapshot verification or legal/compliance state. Never run shrink/force merge on active write generations simply because the action exists.

The next lesson maps this intent specifically to Elastic ILM and data tiers.

Summary

Lifecycle design begins with retention, recovery, query and cost requirements. ILM and ISM can implement overlapping goals, but their state models, tiering semantics, policy updates and APIs differ. Chapter 17 now applies that intent to Elastic ILM.

Authoritative references

• Elastic index lifecycle management — ILM scope, availability, and lifecycle concepts for indices and data streams.
• Elastic ILM phases and actions — Hot/warm/cold/frozen/delete phases, cached phase execution, and transition rules.
• Elastic Explain lifecycle API — Current phase/action/step, failures, and phase execution evidence.
• Elastic data tiers — Content/hot/warm/cold/frozen roles and lifecycle placement concepts.
• Elastic policy updates — Policy versions and cached phase-definition behavior.
• Elastic ILM troubleshooting — ERROR steps, retry behavior, and safe diagnosis.
• OpenSearch Index State Management — ISM model, job cadence, policy attachment, and managed-index workflow.
• OpenSearch ISM policies — States, actions, transitions, rollover, force merge, allocation, and templates.
• OpenSearch ISM API — Policy OCC, explain, retry, change-policy, and simulation APIs.
• OpenSearch ISM error prevention — Pre-action validation and explain diagnostics.
• OpenSearch artifacts by version — Current OpenSearch release baseline.
• Elasticsearch downloads — Current Elasticsearch release baseline.