Node Replacement, Disk Expansion, Tier Migration, Rebalancing, and Decommissioning

Learning outcomes

AtlasMart must replace a data node whose disk is near its operational headroom limit. A safe replacement is not “add disk, then delete the old VM.” It is a capacity transfer: add/validate destination capacity, move shard responsibility intentionally, preserve voting membership, observe recovery, and only then remove the source from discovery and infrastructure.

1. Four operations that look similar but have different risk

2. Add capacity before subtracting it

For a permanent data-node replacement, first start the replacement with the intended exact server version, node roles, TLS trust, discovery settings, JVM policy and compatible plugins. Confirm it joins the existing cluster—never bootstrap a new cluster. Then use allocation filters to drain the old node.

# Inspect first.
GET /_cat/allocation?v
GET /_cat/shards?v&s=node,index,shard
GET /_cluster/health?pretty

# Exclude a node from shard allocation. Both products support built-in _name filtering.
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._name": "atlasmart-data-old"
  }
}

# Observe; do not stop the node yet.
GET /_cat/recovery?v&active_only=true
GET /_cat/shards?v&h=index,shard,prirep,state,node
GET /_cluster/allocation/explain?include_disk_info=true

# After the node owns no required shards and the destination state is healthy,
# clear the temporary filter.
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._name": null
  }
}

3. Data drain and voting removal are separate workflows

A node can be empty of shards but still be part of the coordination voting configuration. Conversely, a data-only node can be drained without any voting change. For a permanent removal of multiple master/cluster-manager-eligible nodes, inspect and use the product-specific voting exclusion/reconfiguration mechanism so a majority remains throughout. Remove one eligible node at a time when possible.

4. Disk expansion does not erase watermarks or recovery physics

Expanding a volume may relieve a disk watermark, but the cluster still needs enough free space for merges, shard relocation, snapshots/restores and failure recovery. After expansion, verify the operating system sees the new filesystem size, then verify node filesystem stats and allocation decisions. Do not lower watermarks merely to postpone capacity work.

GET /_cat/allocation?v&bytes=gb
GET /_nodes/stats/fs?filter_path=nodes.*.name,nodes.*.fs.total
GET /_cluster/settings?include_defaults=true&flat_settings=true
GET /_cluster/allocation/explain?include_disk_info=true

5. Tier migration is policy plus capacity

Elasticsearch data tiers and OpenSearch allocation/tier patterns are not API-identical. The invariant is: the destination tier must have role/attribute eligibility, enough disk/IOPS/CPU, awareness placement and recovery headroom before policy or filters move shards. A lifecycle action that “should move warm” cannot manufacture warm capacity.

6. Wrong approach: decommission while recovery is still active

If AtlasMart shuts down atlasmart-data-old as soon as relocation begins, source reads may disappear mid-copy and the cluster can restart recovery from another copy, increasing network/disk load. If it was the only valid primary copy, the outcome can be worse. The correct gate is no required shards/tasks/coordination dependency plus healthy destination copies and application smoke success.

node: atlasmart-data-old
replacement: atlasmart-data-new
server/version: EXACT
plugins: EXACT COMPATIBLE SET
voting role: yes/no
shards_before: N
shards_after_drain: 0 required copies
active_recoveries: 0
cluster_health: green (or approved explicit state)
app_smoke: PASS
p99_delta: measured
snapshot/recovery_point: VERIFIED
seed_hosts/config updated: YES
old_node_poweroff_authorized: YES

7. Failure injection: replacement node cannot join

In a disposable lab, intentionally give the replacement one wrong discovery/TLS/plugin setting. The correct response is not to remove the old node anyway. Stop, inspect logs and node inventory, repair the single incompatibility, and require the new node to join before draining. This makes “add-before-remove” a real rollback boundary.

8. Bridge to compatibility planning

Node replacement can be a same-version infrastructure event or part of a version upgrade. The latter adds index-format, plugin, client and no-downgrade constraints. Lesson 4 freezes those dependencies before any binary changes.

References

• Elastic 9.5.3 release notes — Current Elasticsearch release baseline.
• Elastic self-managed Elasticsearch upgrade — Rolling order, allocation restriction, flush, plugin upgrade, recovery and no-downgrade boundary.
• Elastic plan/prepare upgrade — Upgrade path, compatibility and preparation guidance.
• Elastic add/remove nodes — Node removal, shard draining and master-eligible considerations.
• Elastic diagnose unassigned shards — Health meaning and allocation-explain workflow.
• Elastic shard allocation/routing settings — Allocation controls, filters, awareness and decommissioning primitives.
• OpenSearch 3.8 version history — Current OpenSearch release baseline.
• OpenSearch rolling upgrade — Health gate, allocation restriction, flush and role ordering.
• OpenSearch migrate or upgrade — Upgrade methods, plugin compatibility, configuration backup and snapshots.
• OpenSearch voting and quorum — Voting majority and maintenance safety.
• OpenSearch voting configuration — Voting membership and automatic shrink behavior.
• OpenSearch allocation explain API — Allocation diagnosis.
• OpenSearch cluster settings — Allocation include/require/exclude and recovery settings.
• OpenSearch nodes info API — Version/JVM/plugin/node-role inventory.