Chapter 17 · Index Lifecycle: Elastic ILM/Data Tiers and OpenSearch ISM
Lifecycle Policy Versioning, Safe Testing, Stuck Actions, Manual Intervention, and Compliance Retention
Version lifecycle policies safely, test state transitions, diagnose stuck actions, intervene without corrupting lifecycle state, and make retention deletion conditional on compliance and recoverability evidence.
Learning outcomes
Lifecycle policies change while indices are already moving through them. AtlasMart therefore needs a deployment discipline: version policies, simulate/test first, understand when updates take effect, diagnose stuck work, retry only after repair, and keep compliance deletion gates external to blind automation.
Explain Elastic cached-phase policy updates and OpenSearch ISM policy/OCC change behavior.
Promote policy changes through deterministic fixtures rather than editing production first.
Diagnose stuck ILM/ISM actions from controller state plus allocation/disk/permission evidence.
Use retry/change-policy/manual-intervention APIs only within their safety boundaries.
Make retention automation subordinate to legal hold, verified backup/restore, and deletion evidence.
Examples target self-managed
Elasticsearch 9.5.3 / Kibana 9.5.3 and
OpenSearch 3.8.0 / OpenSearch Dashboards 3.8.0. AtlasMart keeps https://localhost:9200 for
Elasticsearch with CA verification and
https://localhost:9201 for the disposable
OpenSearch demo certificate using
OPENSEARCH_INITIAL_ADMIN_PASSWORD. Existing
containers remain atlasmart-es and
atlasmart-os. The lifecycle lab uses isolated
aliases atlasmart-life-es and
atlasmart-life-os, one primary shard and zero
replicas so a single-node local lab can run; that topology is
not production guidance. OpenSearch demo -k is
local-only. No moving latest tags are used.
Elastic ILM is a phase/action engine integrated with Elastic data tiers. OpenSearch ISM is a plugin state machine with states, ordered actions and transitions. Similar goals such as rollover and deletion do not make policy JSON, policy-update timing, tier semantics, troubleshooting APIs, permissions, managed-service behavior, or feature availability portable.
The generation environment does not run the AtlasMart Docker containers. Commands below are reproducible lab instructions, while response snippets are explicitly labeled expected shapes/invariants rather than fabricated measurements. Measure your own transition timing, I/O, CPU, merge time, p95/p99 search latency and storage consumption.
1. Versioning means more than naming files v2
A lifecycle artifact should be source-controlled with an
application-level version and promoted like schema/code. But the
server also has its own version/concurrency behavior.
Elasticsearch increments ILM policy version when the policy is
updated; managed indices may continue using cached phase
definitions. OpenSearch policy documents expose
_seq_no and _primary_term for
optimistic-concurrency updates, and a managed index may still be
executing an older attached policy version.
| Question | Elastic ILM | OpenSearch ISM |
|---|---|---|
| How do policy edits propagate? | safe edits may update cached current phase; unsafe retroactive changes wait for phase progression | small config changes may apply next execution; state/action/order changes can be queued until state completion |
| Concurrency on policy document | policy replacement increments version | update uses if_seq_no + if_primary_term |
| See running definition | ILM explain phase_execution/version | ISM explain show_policy + policy version metadata |
| Preview change | disposable index + explain; no generic mutation-free ILM simulator | ISM simulate supports stored/inline policy preview |
2. Safe promotion workflow
1. Parse policy JSON and lint destructive actions.
2. Verify exact index/data-stream/alias match set.
3. Create a disposable generation with production-like settings.
4. Apply/attach the candidate policy only to the fixture.
5. Simulate where the product supports it; otherwise use explain + dry design review.
6. Trigger safe small rollover/transition conditions.
7. Observe controller state, allocation, segment/merge, disk, CPU and latency.
8. Test one deliberate recoverable failure (for example a missing rollover alias).
9. Repair the cause; retry through supported API.
10. Confirm deletion is disabled or far-future until compliance/restore gates pass.
11. Promote policy with immutable change record and rollback instructions.The most important test is not “policy accepted.” It is whether the controller reaches the intended state while the cluster still meets availability, recovery, latency and compliance criteria.
3. Deliberately create a safe stuck rollover
On a disposable index only, remove or misconfigure the rollover alias so the controller cannot roll over. Then inspect controller state. The purpose is to learn that a lifecycle error is a symptom; the repair is the underlying alias/allocation/configuration fix.
GET atlasmart-life-es-000001/_ilm/explain?human
GET atlasmart-life-es-000001/_alias
# Repair the alias/template/root cause first.
POST atlasmart-life-es-000001/_ilm/retry
GET atlasmart-life-es-000001/_ilm/explain?humanGET _plugins/_ism/explain/atlasmart-life-os-000001?show_policy=true&validate_action=true
GET atlasmart-life-os-000001/_alias
# Repair the rollover alias/root cause first.
POST _plugins/_ism/retry/atlasmart-life-os-000001
GET _plugins/_ism/explain/atlasmart-life-os-000001?show_policy=trueNever retry repeatedly under unresolved disk-watermark, permission, allocation or alias failures. That converts a clear safety signal into operational noise.
4. Policy changes and manual intervention
Elastic protects current phase execution by caching phase
definitions. A policy edit that cannot be safely applied
retroactively does not magically rewrite the current phase.
OpenSearch change_policy is asynchronous; changes
that alter state/actions/order can wait until the current state
completes. This design prevents abrupt mid-action state
corruption.
Elastic’s move-to-step API can execute a lifecycle step and is documented as potentially destructive. OpenSearch change-policy can explicitly select a target state. Both require change control, confirmed current state, and a proof that the action is safe. A manual state jump is not a substitute for fixing a broken policy.
5. Compliance retention is an external invariant
A lifecycle engine knows index age and policy state, not the full legal/business context. A legal hold, investigation, contractual retention extension or failed backup can invalidate deletion even when the age condition is true. AtlasMart therefore defines a deletion gate outside the lifecycle JSON: policy release approval requires retention authority, legal-hold check, verified restore evidence where mandated, and auditable ownership.
| Evidence before deletion | Why |
|---|---|
| retention/compliance date | age alone may not equal legal eligibility |
| legal-hold state | active hold must override normal expiration |
| snapshot/restore verification if required | a snapshot name is not proof of usable recovery |
| index/alias/data-stream scope | avoid deleting unrelated generations |
| policy explain state | prove the controller is deleting the intended object |
| owner/change record | support audit and incident reconstruction |
6. Cleanup and bridge
DELETE _index_template/atlasmart-life-es-template-v1
DELETE _ilm/policy/atlasmart-life-es-v1
DELETE atlasmart-life-es-*
DELETE _index_template/atlasmart-life-os-template-v1
DELETE _plugins/_ism/policies/atlasmart-life-os-v1
DELETE atlasmart-life-os-*If a policy is still attached to an index, remove/stop management in the product-supported order before deleting the policy. Do not delete shared AtlasMart telemetry/data-stream fixtures from earlier chapters.
The next lesson turns the two implementations into one intent-level portability matrix.
Check your understanding
- Why can an ILM policy edit fail to affect an index already in a phase?
- How should an OpenSearch policy document be updated safely?
- What should happen before retrying a failed lifecycle action?
- Why is manual state movement high risk?
- Why is index age insufficient for compliance deletion?
Review the answers
1. The phase definition is cached; only safe changes are applied retroactively.
2. Use the current seq_no and primary_term so stale updates fail instead of silently overwriting newer policy state.
3. Repair the actual alias/allocation/disk/permission/policy cause and then retry.
4. It can execute destructive actions or bypass assumptions that normal state progression would enforce.
5. Legal holds, contracts, backup/restore obligations and ownership approvals can override a simple age condition.
Summary
Lifecycle automation is deployed software. Version it, test it, observe its running state, repair causes before retry, and never let the controller become the sole authority for irreversible deletion.
Authoritative references
- Elastic index lifecycle management — ILM scope, availability, and lifecycle concepts for indices and data streams.
- Elastic ILM phases and actions — Hot/warm/cold/frozen/delete phases, cached phase execution, and transition rules.
- Elastic Explain lifecycle API — Current phase/action/step, failures, and phase execution evidence.
- Elastic data tiers — Content/hot/warm/cold/frozen roles and lifecycle placement concepts.
- Elastic policy updates — Policy versions and cached phase-definition behavior.
- Elastic ILM troubleshooting — ERROR steps, retry behavior, and safe diagnosis.
- OpenSearch Index State Management — ISM model, job cadence, policy attachment, and managed-index workflow.
- OpenSearch ISM policies — States, actions, transitions, rollover, force merge, allocation, and templates.
- OpenSearch ISM API — Policy OCC, explain, retry, change-policy, and simulation APIs.
- OpenSearch ISM error prevention — Pre-action validation and explain diagnostics.
- OpenSearch artifacts by version — Current OpenSearch release baseline.
- Elasticsearch downloads — Current Elasticsearch release baseline.