Build a Saturation Diagnosis from GC, CPU, Heap, Rejections, Latency, Disk, and Workload Evidence

Learning outcomes

AtlasMart now has all the pieces to misdiagnose a slowdown: high heap could mean healthy cache use or GC pressure; high CPU could mean useful throughput or one pathological query; a low cache hit rate might be irrelevant; a queue might be the first saturation signal or merely a brief burst. The goal is therefore not a dashboard screenshot—it is a causal narrative supported by synchronized workload, latency and server evidence.

1. Start with the service symptom, not the JVM knob

Write the incident in client terms first: “From 14:03–14:11, product-search p99 rose from its normal band, 429s appeared, and indexing freshness exceeded the agreed window while request rate increased.” Do not begin with “heap is 78%” because that already presumes the cause.

Then align server evidence to the same window. A runtime diagnosis is strongest when independent signals agree: queue depth/rejections, CPU/hot threads, heap/GC, breaker/indexing-pressure counters, disk latency, shard/indexing/search rates and client concurrency.

2. One API evidence bundle

GET _cluster/health?level=shards
GET _cat/nodes?v&h=name,roles,cpu,heap.percent,ram.percent,load_1m,node.role
GET _nodes/stats/jvm,process,os,breaker,thread_pool,indexing_pressure?human
GET _nodes/stats/indices/search,indexing,merge,refresh,query_cache,request_cache,fielddata?human
GET _nodes/hot_threads?threads=5&ignore_idle_threads=true
GET _tasks?detailed=true&actions=*search*,*write*,*bulk*

GET _nodes/stats/shard_indexing_pressure,search_backpressure?human
GET _cluster/settings?include_defaults=true&flat_settings=true

Not every API field is identical across products/releases. Preserve raw JSON alongside your normalized incident summary so future reviewers can see what the server actually returned. Scrub credentials and sensitive query/user data before sharing evidence.

3. Turn observations into a falsifiable hypothesis

Bad hypothesis: “Elasticsearch needs more heap.” Good hypothesis: “The sale traffic increased client search concurrency beyond the search pool’s sustainable drain rate; queued requests raised p99 and retries amplified arrival rate, while GC and disk remained stable. Capping client concurrency should reduce queue/rejection growth and p99 without reducing steady-state useful throughput.”

The good hypothesis predicts both confirming and disconfirming evidence. If queue depth and p99 remain high after concurrency is capped while disk wait spikes, the hypothesis is incomplete or wrong.

4. A decision matrix for common saturation shapes

5. Build the resource-bounded AtlasMart mixed workload

The mandatory lab uses atlasmart-runtime-lab-v1 and a finite query set: one full-text product search, one filtered size-zero facet aggregation, one exact lookup and a small deterministic bulk batch. Use a single-node local container only. Do not target a shared cluster.

The workload runner can be any local client capable of recording per-request latency and status. Keep the server API payloads exactly fixed across candidates. If you implement a custom runner, cap workers and request count in constants—do not generate unbounded load.

GET atlasmart-runtime-lab-v1/_search
{
  "size": 10,
  "query": {
    "bool": {
      "must": [{"match":{"name":"wireless keyboard"}}],
      "filter": [
        {"term":{"available":true}},
        {"range":{"price":{"lte":150}}}
      ]
    }
  },
  "sort": [{"_score":"desc"},{"sku":"asc"}]
}

GET atlasmart-runtime-lab-v1/_search?request_cache=true
{
  "size": 0,
  "query":{"term":{"available":true}},
  "aggs":{
    "category":{"terms":{"field":"category","size":10}},
    "price":{"percentiles":{"field":"price","percents":[50,95]}}
  }
}

• Baseline: concurrency 1, fixed request count, warm-up then measurement.
• Candidate pressure step: increase to the next bounded concurrency only if stop conditions remain clear.
• Safe saturation signal may be rising queue/p99 or a controlled 429/rejection; you do not need an OOM or breaker trip.
• Stop before host swapping, container kill risk, prolonged GC, unrelated impact or runaway retries.
• After cooldown, apply one repair—e.g., cap client concurrency—and rerun the identical mix.

6. Calculate latency percentiles without pretending they are universal targets

Sort the measured client durations. The 95th percentile approximates the latency below which 95% of requests completed; p99 focuses further into the tail. Use the same percentile method and sample size for baseline/candidate. Do not publish a universal “Elasticsearch p99 should be X ms”: acceptable latency depends on query, data, topology, hardware and product requirements.

Throughput alone is insufficient. A candidate that increases operations/second by 5% while doubling p99 and introducing rejections may be worse for an interactive service. Conversely, a batch pipeline may accept higher latency if useful throughput and recovery behavior improve within its SLA.

7. Change one variable, then try to falsify yourself

Suppose concurrency 8 creates search queue growth and 429s while CPU is high, but concurrency 4 has similar useful throughput and much lower p99. Set the application limit to 4 for the lab and rerun. If queues/rejections disappear and throughput stays similar, the evidence supports client admission control as part of the repair.

Now try to falsify it: repeat after a restart/cold cache, after enough time for merge/background work to settle, and with the same data/query mix. If the result depends entirely on warmed filesystem cache, the conclusion must say so. If a hot shard dominates, revisit Chapter 13 routing rather than attributing everything to concurrency.

8. What to scale and what to tune

9. Operational and security boundaries

Runtime evidence can contain query strings, tenant names, hostnames and potentially sensitive fields. Restrict node/task/hot-thread APIs to operators, store diagnostics with access control, and redact before external sharing. On managed Elastic/Amazon/OpenSearch offerings, some host/JVM controls are intentionally abstracted; use provider metrics and supported knobs rather than trying to reproduce self-managed internals exactly.

A node replica is not a backup. Capacity tuning must preserve snapshot/restore and recovery objectives. A “fast” cluster that cannot recover within its failure budget or has no disk/memory headroom is not production-ready.

Production judgment and bridge to Chapter 16

A credible saturation diagnosis says what changed, which resource became constrained, how the mechanism produced the client symptom, what evidence supports it, what evidence would disprove it, and how the repair was verified. It does not end with a generic tuning checklist.

Chapter 16 moves from node-level saturation into search execution itself: distributed query/fetch phases, profiling, slow logs, tasks, deep pagination, search_after, point-in-time readers and cancellation. The runtime evidence learned here becomes the context for interpreting those search-specific tools.

Summary and next step

You can now separate heap, filesystem cache, logical caches, breaker pressure, queues, indexing/search backpressure and client concurrency, then connect them to synchronized workload and tail-latency evidence. That is the runtime foundation for Chapter 16’s query/fetch profiling and pagination work.

Authoritative references

• Elastic JVM settings — Automatic heap sizing, filesystem-cache headroom, container memory, and compressed ordinary object pointer guidance.
• Elastic node query cache settings — Filter-context query-cache eligibility, segment-level caching and invalidation behavior.
• Elastic shard request cache — Shard-level request-result caching and request/index controls.
• Elastic field data cache settings — On-heap fielddata/global-ordinal cache behavior and breaker interaction.
• Elastic circuit breaker settings — Parent, fielddata, request and in-flight breaker semantics.
• Elastic thread pool settings — Current thread-pool types, sizing and queue behavior.
• Elastic indexing pressure — Outstanding indexing-byte accounting and rejection behavior.
• Elastic Nodes Stats API — JVM, process, caches, breakers, thread pools and indexing-pressure observations.
• OpenSearch circuit breaker settings — Parent/child breaker settings and failure-protection intent.
• OpenSearch caching overview — On-heap request/query/fielddata caching concepts.
• OpenSearch index request cache — Shard-level request-result caching, invalidation and metrics.
• OpenSearch field data cache — Fielddata/global ordinals and the keyword-subfield alternative.
• OpenSearch thread pool settings — Thread pools, queue sizing and monitoring guidance.
• OpenSearch search backpressure — OpenSearch-specific search task resource tracking and cancellation.
• OpenSearch shard indexing backpressure — OpenSearch-specific per-shard indexing pressure and rejection mechanism.
• OpenSearch Nodes Stats API — JVM, process, caches, indexing pressure and backpressure statistics.
• OpenSearch Nodes Hot Threads API — CPU/wait/block thread samples for diagnosis.