Node Roles, Cluster State, Master/Cluster-Manager Responsibilities, Data Tiers, and Coordinating Work

Learning outcomes

Documents do not choose machines directly. Cluster metadata describes indices, mappings, settings and shard routing; an elected control-plane node publishes authoritative cluster-state changes; eligible data nodes host shard copies; and whichever node receives a client request also coordinates work. Elasticsearch and OpenSearch share these broad mechanisms but use different role names and have diverged in specialized node/tier capabilities.

1. Control-plane leader is not the primary shard

Elasticsearch calls an eligible node role master and elects a master node. OpenSearch uses cluster_manager and elects a cluster manager. This elected role manages cluster-wide metadata and coordination: membership changes, index metadata, routing table changes and shard-allocation decisions. It is not the permanent data writer for every document.

Each primary shard is the write authority for its own replication group. An index with two primaries therefore has two independent primary-shard groups even though the cluster has one elected master/cluster-manager at a time. Conflating these levels causes bad incident reasoning: replacing a primary shard copy is a data-plane replication event; electing a new control-plane leader is a cluster-coordination event.

2. Every node can coordinate, but not every node should do every job

A client can connect to a node that does not hold the target shard. That node uses cluster-state routing information to forward the request to the appropriate shard copies and combine results where needed. Both products therefore treat coordination as an implicit responsibility of nodes. A dedicated coordinating-only node is created by assigning no explicit roles, but it still joins the cluster, receives cluster state and needs CPU/memory for fan-out and reduce work.

Data nodes host shards and execute CRUD/search/aggregation work. Ingest nodes execute ingest pipelines. As clusters grow, separating control-plane eligibility from heavy search/indexing work can reduce interference. Do not copy a “three masters + N data nodes” diagram into every deployment without workload and failure-domain analysis; managed services and serverless products can hide or reshape these choices.

GET /_nodes?filter_path=cluster_name,nodes.*.name,nodes.*.roles,nodes.*.version,nodes.*.jvm.versionGET /_cat/nodes?v=true&h=name,ip,node.role,versionGET /_cat/shards/atlasmart-products-v2?v=true&h=index,shard,prirep,state,node

On the one-node Chapter 01 baseline, one process necessarily carries multiple responsibilities. That is acceptable for a learning fixture because the lesson states the limitation. It is not evidence that combined roles are optimal for a large production workload.

3. Data tiers and specialized roles have diverged

Elasticsearch exposes specialized data roles such as data_content, data_hot, data_warm, data_cold and data_frozen as part of its data-tier architecture, alongside generic data. These roles integrate with Elastic lifecycle and deployment behavior.

OpenSearch has its own role model. General data and ingest roles remain, while current OpenSearch also documents specialized warm nodes for searchable snapshots and search nodes that can host search replicas in remote-store/segment-replication workload-separation designs. OpenSearch also documents hot/warm placement through allocation attributes in conventional clusters. These are not JSON-compatible aliases for Elastic data tiers. A migration must map the lifecycle/storage intent to the target platform’s actual capabilities.

4. Cluster state is the routing/metadata map

Cluster state contains metadata such as index settings/mappings, node membership and routing tables. The elected master/cluster-manager publishes changes so nodes can route requests consistently. It is not where AtlasMart stores the full product JSON body. Cluster-state size and update frequency matter because every node needs enough metadata to participate, which is one reason excessive numbers of tiny indices/shards and uncontrolled mapping growth become operational problems.

GET /_cluster/state/metadata,routing_table,nodes/atlasmart-products-v2?filter_path=cluster_name,master_node,metadata.indices.atlasmart-products-v2.settings.index.number_of_shards,metadata.indices.atlasmart-products-v2.settings.index.number_of_replicas,routing_table.indices.atlasmart-products-v2.*,nodes.*.nameGET /_cluster/health/atlasmart-products-v2?level=shards

5. Controlled primary-unavailable drill: allocation, red health, repair

The prompt requires learners to see an unavailable-shard signal, but the drill must not threaten unrelated data. Use only atlasmart-products-v2 in the disposable one-node cluster. First record the actual node name. Then temporarily exclude that node from allocation for this one index. With no other eligible node, its primaries can become unassigned and index health becomes red. Finally clear the filter and wait for recovery.

GET /_cat/nodes?v=true&h=name,ip,node.role# Replace <LAB_NODE_NAME> with the exact node name from the response.PUT /atlasmart-products-v2/_settings{  "index.routing.allocation.exclude._name": "<LAB_NODE_NAME>"}GET /_cluster/health/atlasmart-products-v2?prettyGET /_cat/shards/atlasmart-products-v2?v=trueGET /_cluster/allocation/explain{  "index": "atlasmart-products-v2",  "shard": 0,  "primary": true}# RESET: clear only the course-index allocation exclusion.PUT /atlasmart-products-v2/_settings{  "index.routing.allocation.exclude._name": null}GET /_cluster/health/atlasmart-products-v2?wait_for_status=yellow&timeout=60sGET /_cat/shards/atlasmart-products-v2?v=true

Blast radius: searches/writes to this one course index can fail while its primary is unassigned. Do not run the exclusion against *, system indices, the data stream, production names or cluster-wide allocation settings. If the exact product/version rejects a request shape, stop and consult its current allocation API rather than improvising broader commands.

Once allocation is restored, the target can return to green because its replica count is zero. If you intentionally left replicas at one, yellow would be an acceptable intermediate state until another eligible node exists.

Verification checklist

• The node-role API identifies the actual single-node role set for each product rather than assuming defaults.
• The control-plane role is described as master in Elasticsearch and cluster_manager in OpenSearch.
• The shard inventory maps both product primaries to the one local data-capable node before failure injection.
• The allocation exclusion affects only atlasmart-products-v2 and produces an explainable unassigned-primary signal.
• Clearing the setting reallocates primaries and restores index availability.

Production judgment

Role separation is an interference/failure-domain decision, not a badge of maturity. Dedicated control-plane nodes need stable resources and should not be routine client targets; coordinating-only nodes consume resources and enlarge the cluster-state acknowledgement set; specialized tiers/search roles can improve cost or workload isolation but tie architecture to product-specific mechanisms. Before changing node roles in production, verify data paths, shard evacuation, voting configuration, managed-service constraints and rollback procedures. Chapter 21 will handle maintenance and replacement in depth.

Summary and next step

Cluster state tells nodes where shard copies belong; the elected master/cluster-manager publishes metadata changes; primary shards own write sequencing for their replication groups; and any receiving node can coordinate requests. Next, trace a write and a search across those components, including replication, scatter/gather and partial failure behavior.

Authoritative references

• Elasticsearch node roles — Current master/data/tier/ingest/coordinating-role semantics.
• Elasticsearch node settings — Current role configuration and coordinating-only behavior.
• Elasticsearch cluster state API — Operational cluster-state surface.
• OpenSearch cluster architecture — Current cluster-manager, data and coordinating-node terminology.
• OpenSearch discovery and cluster formation — Election and authoritative cluster-state concepts.
• OpenSearch separate index/search workloads — Current search role/search-replica behavior and constraints.
• OpenSearch searchable snapshots — Current 3.x warm role requirement for searchable snapshots.