Model a Capacity/Shard Plan and Simulate Node Loss Without Exceeding Recovery or Latency Objectives

Learning outcomes

The platform review now asks a concrete question: “Can AtlasMart survive one node or one availability-zone loss without missing the recovery objective or breaching search latency?” This lesson converts topology folklore into a small capacity model and an acceptance test. The model is deliberately conservative and parameterized; the numbers become valid only after you replace assumptions with measurements.

1. Start from service objectives and measured workload

Capacity planning should begin with what the service must do, not with how many nodes are currently available. Define a growth horizon, expected indexed bytes, writes/s, query mix/concurrency, p95/p99 targets, recovery RTO, tolerated availability state during a zone loss, and snapshot RPO. The last item is intentionally separate: replicas help availability; snapshots handle independent recovery.

Use ranges for uncertain inputs. A plan with 40% annual growth should also show what happens at a higher band, because reindexing under emergency disk pressure is the worst time to discover the forecast was optimistic.

2. Build a transparent model—not a magic calculator

The script below intentionally avoids pretending to know Elasticsearch/OpenSearch throughput. It takes measured values and calculates only simple envelopes: average primary bytes, replica-inclusive storage, a lower-bound single-shard copy time, and post-failure disk pressure. It cannot predict p99 query latency; that still requires a workload test.

A useful model exposes assumptions so reviewers can challenge them. Hidden constants are where “best practice” turns into folklore.

from dataclasses import dataclass
from math import ceil

@dataclass
class Plan:
    primary_store_gib: float
    primaries: int
    replicas: int
    data_nodes: int
    node_disk_gib: float
    safe_recovery_mib_s: float

    @property
    def avg_primary_gib(self):
        return self.primary_store_gib / self.primaries

    @property
    def total_copies_gib(self):
        return self.primary_store_gib * (1 + self.replicas)

    @property
    def avg_disk_used_pct(self):
        return 100 * self.total_copies_gib / (self.data_nodes * self.node_disk_gib)

    @property
    def one_shard_copy_minutes_floor(self):
        mib = self.avg_primary_gib * 1024
        return mib / self.safe_recovery_mib_s / 60

for p in [
    Plan(1200, 6, 1, 6, 1000, 80),
    Plan(1200, 12, 1, 6, 1000, 80),
    Plan(1200, 24, 1, 6, 1000, 80),
]:
    print({
      "primaries": p.primaries,
      "avg_primary_gib": round(p.avg_primary_gib,1),
      "avg_disk_used_pct": round(p.avg_disk_used_pct,1),
      "one_shard_copy_min_floor": round(p.one_shard_copy_minutes_floor,1),
    })

# Replace every value with measurements. This is not a benchmark or sizing recommendation.

3. Add failure-domain math before performance testing

With six equal data nodes across three zones and one replica, losing one zone removes roughly one third of data-node capacity at once. The survivors must serve traffic and reconstruct missing copies if the policy allows it. Average steady-state disk utilization can therefore look comfortable while zone-loss utilization becomes unsafe.

Compute a conservative survivor capacity ratio before running the cluster. If ordinary awareness will rebuild replicas into surviving zones, budget the temporary copy amplification. If forced awareness will leave replicas unassigned, budget the extra search/write load but not immediate full replica restoration until the missing zone returns.

def survivor_disk_pct(total_copy_gib, node_disk_gib, total_nodes, lost_nodes):
    survivors = total_nodes - lost_nodes
    return 100 * total_copy_gib / (survivors * node_disk_gib)

print("steady:", survivor_disk_pct(2400, 1000, 6, 0))
print("one node lost:", survivor_disk_pct(2400, 1000, 6, 1))
print("one 2-node zone lost:", survivor_disk_pct(2400, 1000, 6, 2))

# This assumes all copies are eventually present on survivors.
# Forced awareness may intentionally leave some replicas unassigned instead.

4. Define the acceptance matrix before the game-day

A failure test without pass/fail criteria is a demo. Define the workload, failure, expected health state, maximum recovery time and application SLO before stopping anything. Keep failures isolated to disposable containers or use a deterministic trace if the local machine cannot host the topology.

5. Execute with observable APIs and application metrics

GET _cluster/health?level=shards
GET _cat/nodes?v&h=name,node.role,cpu,heap.percent,ram.percent,disk.avail
GET _cat/shards?v&h=index,shard,prirep,state,docs,store,node,relocating_node
GET _cat/recovery?v&active_only=true
GET _cat/allocation?v
GET _nodes/stats/fs,indices,transport,thread_pool,jvm
POST _cluster/allocation/explain
{}

# Pair this with application p50/p95/p99, error rate, QPS/write rate and test timestamps.

Do not use _cluster/health alone. Green health can coexist with terrible latency, and yellow can be an intentional forced-awareness state. The acceptance artifact must pair cluster state with user-facing workload metrics and explain any shard that is unassigned or throttled.

6. Deliberately wrong approach: certify from averages

Average shard size, average CPU and average latency hide the failure domains that matter. One hot routing key, one slow disk, one oversized shard or one availability zone can determine the incident. p95/p99 and per-shard/per-node distributions are more useful than a single mean.

Another wrong pattern is to test node loss on an idle cluster. Recovery that meets RTO only when traffic is zero is not evidence for the production objective. Replay or generate a fixed representative workload during the experiment.

7. AtlasMart final shard-plan record

Write the plan as a versioned decision record so future chapters can reason about the same topology. Include: server versions, node roles/resources, zones, primaries/replicas, routing policy, lifecycle/tier rules, current and forecast primary bytes, workload classes, recovery settings, disk safety margin, snapshots, failure objectives, benchmark harness and the exact date of the last game-day.

Add re-evaluation triggers rather than pretending the decision is permanent. Examples: 25% primary-store growth, a new vector field, a high-cardinality aggregation workload, a new tenant that changes routing skew, new instance/storage type, changed zone count, or changed RTO/SLO.

• No universal shard-size threshold appears in the decision record.
• Every tuning override has an owner and reset condition.
• Replica topology is documented separately from snapshot/RPO policy.
• Managed-service restrictions are listed explicitly.
• The next migration path—reindex, split/shrink where applicable, or new index generation—is known before capacity becomes critical.

Production judgment

A shard plan is accepted evidence, not a static best practice. Keep the model in source control, keep the workload harness reproducible, and rerun the failure matrix after meaningful version/topology/workload changes. The cheapest cluster that passes steady-state benchmarks but fails a zone-loss test is not production-ready.

Summary and next step

You can now design shard topology from capacity, fan-out, routing distribution and failure objectives, explain allocator decisions, and validate node/zone loss without relying on folklore. Chapter 14 goes inside each shard to Lucene segments, refresh, merge, translog, flush and storage behavior—the mechanisms that make shard size and recovery cost tangible.

Authoritative references

• Elastic shard allocation settings — Current allocation, rebalance, disk-watermark and recovery controls.
• Elastic shard allocation awareness — Awareness and forced-awareness behavior across failure domains.
• Elastic allocation explain API — Explain why a shard is or is not allocatable.
• Elastic index recovery API — Recovery stage and byte/file progress.
• OpenSearch cluster settings — Current routing/allocation, awareness and recovery settings.
• OpenSearch cluster tuning — Shard allocation awareness and forced awareness concepts.
• OpenSearch CAT shards — Shard state and placement inspection.
• OpenSearch cluster allocation explain — Allocation diagnostics and decider evidence.
• Elastic production guidance: size shards — Workload-driven shard sizing and benchmark guidance.
• Elastic CAT allocation — Per-node shard count and disk allocation view.
• OpenSearch CAT allocation — Per-node disk and shard allocation evidence.
• OpenSearch performance analyzer — Optional operational metrics for self-managed performance analysis.