Chapter 17 · Index Lifecycle: Elastic ILM/Data Tiers and OpenSearch ISM
Design Equivalent Retention Strategies in Elastic and OpenSearch and Document Feature/Operational Differences
Design equivalent AtlasMart retention intent in Elastic and OpenSearch while documenting non-equivalent lifecycle, tiering, policy-update, simulation, and managed-service behavior.
Learning outcomes
AtlasMart’s architecture board wants one retention design that can be implemented on Elasticsearch or OpenSearch. The correct deliverable is not “same JSON.” It is an intent contract plus two product-specific implementations, measurable equivalence tests and an explicit list of non-equivalent capabilities.
Write lifecycle requirements in product-neutral invariants before selecting ILM or ISM syntax.
Produce parallel Elastic/OpenSearch policy designs and document where equivalence stops.
Validate rollover, retention, placement, merge/shrink and delete semantics with observable evidence.
Define portability tests for correctness, cost, recovery, latency, permissions and managed-service boundaries.
Hand off lifecycle ownership cleanly to Chapter 18 snapshot/restore and disaster-recovery design.
Examples target self-managed
Elasticsearch 9.5.3 / Kibana 9.5.3 and
OpenSearch 3.8.0 / OpenSearch Dashboards 3.8.0. AtlasMart keeps https://localhost:9200 for
Elasticsearch with CA verification and
https://localhost:9201 for the disposable
OpenSearch demo certificate using
OPENSEARCH_INITIAL_ADMIN_PASSWORD. Existing
containers remain atlasmart-es and
atlasmart-os. The lifecycle lab uses isolated
aliases atlasmart-life-es and
atlasmart-life-os, one primary shard and zero
replicas so a single-node local lab can run; that topology is
not production guidance. OpenSearch demo -k is
local-only. No moving latest tags are used.
Elastic ILM is a phase/action engine integrated with Elastic data tiers. OpenSearch ISM is a plugin state machine with states, ordered actions and transitions. Similar goals such as rollover and deletion do not make policy JSON, policy-update timing, tier semantics, troubleshooting APIs, permissions, managed-service behavior, or feature availability portable.
The generation environment does not run the AtlasMart Docker containers. Commands below are reproducible lab instructions, while response snippets are explicitly labeled expected shapes/invariants rather than fabricated measurements. Measure your own transition timing, I/O, CPU, merge time, p95/p99 search latency and storage consumption.
1. Start with the AtlasMart intent contract
| Invariant | Acceptance rule |
|---|---|
| write continuity | clients always write through stable logical target; rollover changes physical generation without application downtime |
| bounded generation | rollover threshold derived from measured recovery/search limits; no universal shard-size folklore |
| read-mostly optimization | shrink/merge-like work only after generation is no longer active write target and resource budget is available |
| retention | delete only after 30-day example intent plus compliance/hold/recoverability gates |
| observability | controller state, action error, allocation, disk/merge/recovery and latency evidence retained |
| rollback | policy version and prior implementation retained; destructive actions have separate approval gate |
2. Equivalent intent, product-specific implementation
| Intent | Elastic 9.5.3 | OpenSearch 3.8.0 | Not equivalent because… |
|---|---|---|---|
| lifecycle model | hot/warm/cold/frozen/delete phases | user-defined states and transitions | control vocabulary and update semantics differ |
| rollover | ILM rollover on data stream or rollover alias | ISM rollover on managed index with rollover alias prerequisites | data stream integration and alias prerequisites differ |
| tier movement | data tiers + tier preference/migrate action | allocation attributes / OpenSearch-specific storage/search modes | Elastic tier roles are not generic ISM states |
| policy update | version increment + cached phase definition | policy OCC + asynchronous change-policy semantics | running-definition propagation differs |
| preview | test fixture + explain | ISM simulate API plus explain | OpenSearch has explicit mutation-free policy simulation |
| error recovery | explain ERROR + retry; move-to-step exceptional | explain/validation + retry; change-policy available | failure state and intervention APIs differ |
| serverless/managed | Elastic Serverless uses data stream lifecycle, not self-managed ILM | Amazon OpenSearch Service/Serverless features/config differ from upstream | managed control planes abstract or restrict lifecycle features differently |
3. Portability test suite
TEST rollover_write_continuity:
ingest through logical target during/after rollover
assert no client physical-index dependency
TEST retention_scope:
enumerate candidate generations
assert only approved prefix/data stream is managed
assert delete state is unreachable before compliance gate
TEST read_only_compaction:
assert old generation is not write index before shrink/force-merge-like work
record merge/shrink duration, disk amplification, CPU and p99
TEST placement:
assert actual shard/node placement matches intended capacity class
run representative old-data query and recovery drill
TEST policy_change:
deploy v2 to fixture
prove when current managed generation sees new definition
TEST failure_recovery:
create reversible alias/allocation fault
prove explain surfaces cause
repair, retry, and confirm healthy statePassing these tests means the two implementations satisfy the same intent under the tested topology. It does not imply APIs, JSON, permissions, plugins or operational tooling are interchangeable.
4. Expected state evidence
GET atlasmart-life-es-*/_ilm/explain?human
GET _cat/aliases/atlasmart-life-es?v
GET _cat/indices/atlasmart-life-es-*?v
GET atlasmart-life-es-*/_settings
GET _cat/segments/atlasmart-life-es-*?vGET _plugins/_ism/explain/atlasmart-life-os-*?show_policy=true
GET _plugins/_ism/policies/atlasmart-life-os-v1
GET _cat/aliases/atlasmart-life-os?v
GET _cat/indices/atlasmart-life-os-*?v
GET atlasmart-life-os-*/_settings
GET _cat/segments/atlasmart-life-os-*?vAdd monitoring data for p95/p99 search latency, indexing rate, merge I/O, disk headroom and recovery time. Controller metadata alone cannot certify SLO compliance.
5. Deliberately wrong portability strategy
At best it fails schema validation. At worst, a hand-translated approximation hides a semantic difference—such as policy update timing, data-tier placement, rollover attachment or managed-service restrictions—and creates silent operational drift. The safer artifact is a common intent document plus two versioned product policies and a shared acceptance-test suite.
6. Production decision record
| Decision surface | Questions to record |
|---|---|
| retention correctness | What event starts age? rollover or creation? What overrides deletion? |
| storage cost | Which generations move where, and what measurable saving justifies the move? |
| recovery | What is RTO per lifecycle stage? Are larger shrunk shards still recoverable in time? |
| latency | What p95/p99 applies to older data and after cache loss? |
| capacity | Can the target tier/state absorb relocation/merge/recovery bursts? |
| security | Which identity updates/runs the policy and which privileges are captured? |
| portability | Which features have no direct counterpart and what is the fallback? |
| managed service | Which controls are exposed, replaced or automated by the provider? |
7. Cleanup
# Elasticsearch: remove managed disposable indices/template/policy in safe order.
DELETE atlasmart-life-es-*
DELETE _index_template/atlasmart-life-es-template-v1
DELETE _ilm/policy/atlasmart-life-es-v1
# OpenSearch: remove disposable indices/template/policy in safe order.
DELETE atlasmart-life-os-*
DELETE _index_template/atlasmart-life-os-template-v1
DELETE _plugins/_ism/policies/atlasmart-life-os-v1Do not delete shared telemetry data streams, snapshots or policies from other chapters. If lifecycle management blocks cleanup, inspect current state and detach/remove management through supported APIs before deleting artifacts.
8. Bridge to Chapter 18: lifecycle is not disaster recovery
A replica protects availability from a shard/node failure; lifecycle automation controls age/placement/retention; neither is a backup. Chapter 18 moves to snapshot repositories, incremental segment reuse, restore semantics, repository access control and measured RPO/RTO. The lifecycle delete gate therefore becomes a consumer of snapshot/restore evidence rather than pretending deletion itself completes the data-protection story.
Check your understanding
- What is the portable lifecycle artifact?
- What does passing equivalent tests prove?
- Why pair controller state with latency/recovery metrics?
- What is the safe alternative when a feature has no counterpart?
- Why does Chapter 18 follow lifecycle?
Review the answers
1. A product-neutral intent/acceptance contract plus separate versioned ILM and ISM implementations—not shared JSON.
2. That the tested implementations satisfy the same intended behavior under that topology, not that APIs/features are interchangeable.
3. Policy state proves automation progress, while SLO metrics prove the resulting placement/compaction actually meets operational objectives.
4. Document the gap and choose an explicit fallback or operationally different design.
5. Retention/deletion decisions require independent backup/restore and RPO/RTO evidence; lifecycle alone is not disaster recovery.
Summary
Chapter 17 turned lifecycle from vendor syntax into a measured operational contract. Elastic ILM/data tiers and OpenSearch ISM can express overlapping goals, but they differ in state model, placement semantics, policy updates, simulation, intervention and managed-service boundaries. The durable design is common intent, separate implementations and shared acceptance evidence.
Authoritative references
- Elastic index lifecycle management — ILM scope, availability, and lifecycle concepts for indices and data streams.
- Elastic ILM phases and actions — Hot/warm/cold/frozen/delete phases, cached phase execution, and transition rules.
- Elastic Explain lifecycle API — Current phase/action/step, failures, and phase execution evidence.
- Elastic data tiers — Content/hot/warm/cold/frozen roles and lifecycle placement concepts.
- Elastic policy updates — Policy versions and cached phase-definition behavior.
- Elastic ILM troubleshooting — ERROR steps, retry behavior, and safe diagnosis.
- OpenSearch Index State Management — ISM model, job cadence, policy attachment, and managed-index workflow.
- OpenSearch ISM policies — States, actions, transitions, rollover, force merge, allocation, and templates.
- OpenSearch ISM API — Policy OCC, explain, retry, change-policy, and simulation APIs.
- OpenSearch ISM error prevention — Pre-action validation and explain diagnostics.
- OpenSearch artifacts by version — Current OpenSearch release baseline.
- Elasticsearch downloads — Current Elasticsearch release baseline.