Rolling Restart/Upgrade Sequencing, Shard Allocation Control, Synced Recovery Concepts, and Validation

Learning outcomes

AtlasMart needs a certificate refresh and then a version upgrade. The operational temptation is to treat both as “restart one node at a time.” The reliable procedure is more explicit: establish a baseline, suppress unnecessary replica churn where appropriate, flush, change exactly one node, preserve version-compatible leadership, restore allocation, wait for recovery, validate the application, and only then move to the next node.

1. Rolling maintenance is a state machine

2. Product-specific role order

In Elasticsearch self-managed rolling upgrades, current guidance upgrades data nodes tier-by-tier from frozen → cold → warm → hot → other data/content, then non-data/non-master roles such as ML, ingest, coordinating, transform and remote-cluster-client nodes, and finally master/voting-only master-eligible nodes. This ordering keeps the older-version master compatible with nodes joining during the upgrade.

OpenSearch rolling upgrades use data nodes first, then ingest/ML/coordinating nodes, and cluster-manager nodes last. Identify the currently elected cluster manager and restart it last within that group.

3. Restrict replica allocation to avoid wasteful churn

When a data node is expected to return quickly, temporarily setting allocation to primaries prevents immediate replica recreation elsewhere. This is not “turning off recovery”; primary allocation can still occur. The setting must be restored after the node rejoins. Forgetting it is a common self-inflicted yellow-cluster incident.

# Before stopping one data node:
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

# Optional: reduce nonessential indexing and flush.
POST /_flush

# ... stop/change/start exactly one node ...

# After it rejoins, restore normal allocation.
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

GET /_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true&timeout=10m
GET /_cat/recovery?v&active_only=true

4. “Synced recovery” means understand recovery—not call an old endpoint

Older Elasticsearch operations literature often mentions a synced flush. In modern Elasticsearch the explicit synced-flush API was removed; a regular flush has had the same recovery-relevant effect since 7.6. Current Elasticsearch 9.5 and OpenSearch 3.8 rolling procedures use POST /_flush. A flush performs a Lucene commit and trims translog requirements; it does not guarantee instant recovery, and it does not make a missing shard copy magically valid.

Peer recovery can reuse existing local segment files and replay retained operations depending on the source/target histories, sequence-number state and retained recovery data. Measure /_cat/recovery rather than assuming “flush means zero-byte recovery.”

5. Version upgrade is not a reversible restart

Elasticsearch explicitly does not support downgrading an in-place upgraded cluster; after upgraded nodes participate, cluster metadata may advance. OpenSearch similarly states that nodes cannot be downgraded. The recovery boundary is therefore snapshot/rebuild, not “install the old binary back over the data directory.” Mixed versions are a temporary rolling-upgrade state only.

GET /
GET /_cat/nodes?v&h=name,node.role,master,version,uptime
GET /_nodes/plugins,jvm?filter_path=nodes.*.name,nodes.*.version,nodes.*.plugins.name,nodes.*.jvm.version
GET /_cluster/health?pretty
GET /_cat/shards?v&s=state
GET /_cat/recovery?v&active_only=true
GET /_cluster/pending_tasks

# App checks (examples, use least-privilege credentials):
GET /atlasmart-products-read/_search?size=1
# write/read-back only through the intended write principal/index

6. Wrong approach: suppress every symptom and keep going

An operator sees relocation and sets allocation to none, sees a plugin failure and removes the plugin, sees a yellow cluster and continues because search still works. Each action erases evidence and expands risk. The repair is one variable at a time: restore documented allocation state, install the target-compatible plugin, wait for recovery, and compare to the baseline. If the gate fails, stop the rollout.

7. AtlasMart rolling sequence

for node in PRODUCT_SPECIFIC_ORDER:
    assert snapshot_verified
    assert voting_majority_after_stop(node)
    assert cluster_gate_passes()
    if node.is_data:
        allocation = "primaries"
    flush_if_useful()
    stop(node)
    assert cluster_still_coordinated()
    upgrade_or_maintain(node)
    install_exact_compatible_plugins(node)
    start(node)
    assert node_joined_expected_cluster_and_version()
    allocation = "all/default"
    wait_for_recovery()
    assert app_smoke_and_p99_gate()
# never run next node merely because previous process started successfully

References

• Elastic 9.5.3 release notes — Current Elasticsearch release baseline.
• Elastic self-managed Elasticsearch upgrade — Rolling order, allocation restriction, flush, plugin upgrade, recovery and no-downgrade boundary.
• Elastic plan/prepare upgrade — Upgrade path, compatibility and preparation guidance.
• Elastic add/remove nodes — Node removal, shard draining and master-eligible considerations.
• Elastic diagnose unassigned shards — Health meaning and allocation-explain workflow.
• Elastic shard allocation/routing settings — Allocation controls, filters, awareness and decommissioning primitives.
• OpenSearch 3.8 version history — Current OpenSearch release baseline.
• OpenSearch rolling upgrade — Health gate, allocation restriction, flush and role ordering.
• OpenSearch migrate or upgrade — Upgrade methods, plugin compatibility, configuration backup and snapshots.
• OpenSearch voting and quorum — Voting majority and maintenance safety.
• OpenSearch voting configuration — Voting membership and automatic shrink behavior.
• OpenSearch allocation explain API — Allocation diagnosis.
• OpenSearch cluster settings — Allocation include/require/exclude and recovery settings.
• OpenSearch nodes info API — Version/JVM/plugin/node-role inventory.