Tune a Slow Search by Changing Query, Mapping, Routing, Shards, Aggregation Strategy, or Pagination—Then Measure

Learning outcomes

AtlasMart’s search SLO is violated, but the team has six plausible fixes: simplify the query, add a keyword field, route by tenant, change shard count, replace a large terms tree with composite paging, or replace deep from with PIT + search_after. Applying all six at once might improve latency—and destroy the ability to know why. This lesson turns optimization into a falsifiable experiment.

1. Start with a symptom and a hypothesis

2. Baseline contract

• Pinned Elasticsearch 9.5.3 or OpenSearch 3.8.0; record distribution, plugins and JVM from node APIs.
• Same fixture or production-like snapshot, same shard topology unless shard count is the variable.
• Same query corpus with expected hits/aggregations/relevance judgments.
• Same client concurrency, request rate, timeout/retry policy and warm-up procedure.
• Record p50/p95/p99 and throughput—not only took.
• Record _shards, failures/skips, CPU, GC, thread-pool rejection, breaker/backpressure and storage state.
• Run enough repetitions to see variability; report cold/warm state explicitly.

3. Lab A: pagination hypothesis

PUT atlasmart-search-exec-v1
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "index.max_result_window": 100
  },
  "mappings": {
    "properties": {
      "sku":        {"type":"keyword"},
      "name":       {"type":"text","fields":{"raw":{"type":"keyword"}}},
      "category":   {"type":"keyword"},
      "price":      {"type":"double"},
      "available":  {"type":"boolean"},
      "updated_at": {"type":"date"},
      "popularity": {"type":"integer"}
    }
  }
}

# Save as chapter16_fixture.py and redirect stdout to fixture.ndjson.
import json
from datetime import datetime, timedelta, timezone
base = datetime(2026, 9, 1, tzinfo=timezone.utc)
for i in range(240):
    sku = f"P-{1000+i:04d}"
    meta = {"index":{"_index":"atlasmart-search-exec-v1","_id":sku}}
    doc = {
      "sku": sku,
      "name": f"AtlasMart {'wireless' if i%3==0 else 'wired'} headset model {i:03d}",
      "category": ["audio","mobile","office"][i%3],
      "price": round(20 + (i%80)*1.25, 2),
      "available": i%5 != 0,
      "updated_at": (base + timedelta(minutes=i)).isoformat().replace('+00:00','Z'),
      "popularity": (i*17)%101
    }
    print(json.dumps(meta,separators=(',',':')))
    print(json.dumps(doc,separators=(',',':')))
# Then POST fixture.ndjson to /_bulk?refresh=true with Content-Type application/x-ndjson.

GET atlasmart-search-exec-v1/_search
{
  "from":80,
  "size":10,
  "query":{"term":{"available":true}},
  "sort":[{"updated_at":"asc"},{"sku":"asc"}],
  "profile":true
}

Capture profile only for structural evidence. Then measure unprofiled repeated requests. Build the equivalent traversal with PIT + search_after. Validate the exact same SKU order for the fixed PIT and compare late-page client p95/p99. The expected mechanism is reduced deep-offset candidate work, not a guaranteed numeric speedup on a 240-document laptop fixture.

4. Lab B: aggregation and fetch hypotheses

GET atlasmart-search-exec-v1/_search
{
  "size":10,
  "_source":["sku","name","price"],
  "query":{"match":{"name":"headset"}},
  "aggs":{
    "category":{"terms":{"field":"category","size":10}},
    "price_stats":{"stats":{"field":"price"}}
  }
}

If an application needs every distinct high-cardinality key, a huge terms.size is not pagination. Use composite with after_key and accept its ordering/feature tradeoffs. If fetch dominates, trim response fields only when the API contract allows it. Every performance change must re-run correctness assertions.

5. Routing/shard/mapping changes require migration evidence

A routing or shard-count change is not a runtime toggle. It changes document placement and usually requires a new index plus reindex/cutover as taught in Chapters 09 and 12. A mapping change such as adding a keyword subfield can be additive for future documents in some cases, but retroactive values may require reindexing depending on the change.

Measure recovery and failure-domain consequences before accepting a topology optimization. Reducing shard fan-out can improve search while making one shard too large for recovery objectives. There is no universal “one shard per X GB” answer.

6. Before/after evidence table

7. Failure injection and rollback

Safe performance work includes failure behavior. During the disposable lab, cancel one long-running task, expire/delete a PIT, and issue one request that exceeds the reduced result window. Confirm the client handles each as an explicit error/state rather than infinite retry.

For production rollout, keep the previous query template/index alias/API version available. Roll back if correctness, relevance, error rate or tail latency breaches the acceptance criteria even if average latency improves.

8. Cleanup

DELETE atlasmart-search-exec-v1
# Also delete any open PIT/async IDs created during the lab using the product-specific APIs.
# Restore any temporary slow-log thresholds to -1.

Do not delete shared AtlasMart indices from prior chapters. The Chapter 16 fixture is intentionally isolated so failure experiments and changed result-window settings remain reversible.

9. Production judgment and bridge to Chapter 17

A search tuning change is complete only when it has a mechanism, repeatable evidence, correctness/relevance checks, operational side-effect analysis and rollback. “Profile got smaller” is not enough. The production decision surface includes p95/p99, throughput, shard/recovery cost, freshness, failure behavior, tenant security and version/API compatibility.

Chapter 17 moves from per-request/runtime behavior to index lifecycle. The same discipline applies: rollover, tiering, shrink/force-merge-like actions and retention must be justified by recovery, cost and compliance evidence—and Elastic ILM and OpenSearch ISM must not be treated as the same control plane.

Summary

Chapter 16 made distributed search cost visible from shard selection through query/fetch, established a diagnostic toolkit, replaced deep paging with stable PIT-backed cursors, defined long-running async/cancellation semantics, and ended with a falsifiable tuning workflow. You are ready to apply the same operational discipline to lifecycle automation.

Authoritative references

• Elastic search API — Current search request options, search type, pre-filter shard behavior and partial-results controls.
• Elastic search profiling — Profile operator trees, collectors and DFS profiling; also documents what profile does not measure.
• Elastic pagination — from/size result window, search_after, PIT consistency and PIT cleanup guidance.
• Elastic point in time API — Opening PITs, keep_alive, changing PIT IDs and retained-segment resource implications.
• Elastic async search submit — Long-running asynchronous search submission and response-size constraints.
• Elastic async search results — Polling, ownership/security and keep_alive behavior for async search.
• Elastic slow logs — Shard-level query/fetch slow logs and current query-logging guidance.
• OpenSearch pagination — from/size, search_after and PIT-backed pagination tradeoffs.
• OpenSearch PIT — Create/list/delete PIT APIs, security permissions and resource lifetime.
• OpenSearch profile API — Search component timings and explicit omissions such as network/queue/coordinator idle time.
• OpenSearch asynchronous search — Plugin endpoint, partial results and long-running search model.
• OpenSearch async settings — Maximum running time, concurrency, retention and wait timeout settings.
• OpenSearch logs — Request-level and shard-level search slow logs plus task-resource logging.
• OpenSearch tasks API — Task inspection and cancellation mechanisms.