Trace One Document from Index Request Through Refresh to Search Result and Explain Every Distributed Step

Learning outcomes

The chapter closes by tracing one AtlasMart document end to end. The goal is not another API recipe: it is to predict each state transition and prove it using metadata. You will choose a business ID and routing key, observe the selected primary-shard group, capture concurrency metadata, separate write acknowledgement from search refresh, then search and correlate the returned hit with the original write.

1. Define the trace before sending the request

The trace document is P-TRACE-01 for tenant TENANT-TRACE. The product index is atlasmart-products-v2 behind alias atlasmart-products. It has two primary shards and zero replicas in the mandatory one-node baseline. The search-event stream is atlasmart-search-events. These names are not arbitrary: they make every cleanup command narrow and every observation attributable to the course.

2. Step A — resolve routing, then index the document

Ask the cluster which shard group the routing value maps to before writing. Then send the document with the same routing value. The receiving node coordinates the request; routing chooses one primary shard; the primary executes the operation and assigns sequence metadata; configured active replicas receive the operation according to the replication model. In the mandatory baseline there are no replicas, so the response should not be misrepresented as replicated high availability.

GET /_search_shards/atlasmart-products-v2?routing=TENANT-TRACEGET /_cat/shards/atlasmart-products-v2?v=true&h=index,shard,prirep,state,node

PUT /atlasmart-products-v2/_doc/P-TRACE-01?routing=TENANT-TRACE{  "product_id": "P-TRACE-01",  "name": "Atlas Trace Keyboard",  "category": "keyboards",  "price": 88.00,  "updated_at": "2026-09-10T17:30:00Z"}

Capture the actual response fields _index, _id, _version, _seq_no, _primary_term, result and _shards. Do not substitute the example values from documentation; your trace is evidence only if it records the engine’s response.

3. Step B — prove current document state before proving search visibility

Immediately perform a routed GET. The correct routing key is part of the lookup because the document was custom-routed. The GET should return the current source and concurrency metadata. Then perform a routed search for the exact product key before the explicit refresh. Depending on whether an automatic refresh occurred, it may already be visible. Record the observation either way; the contract is that refresh governs ordinary search visibility, not that your manual timing always wins a race.

GET /atlasmart-products-v2/_doc/P-TRACE-01?routing=TENANT-TRACEGET /atlasmart-products-v2/_search?routing=TENANT-TRACE&seq_no_primary_term=true{  "query": {"term": {"product_id": "P-TRACE-01"}}}POST /atlasmart-products-v2/_refreshGET /atlasmart-products-v2/_search?routing=TENANT-TRACE&seq_no_primary_term=true{  "query": {"term": {"product_id": "P-TRACE-01"}}}

After the explicit refresh completes, the routed search should see the indexed document unless another failure changed the state. Correlate the hit’s _id, _seq_no, _primary_term and _source.product_id with the GET/write evidence. If they differ because the document was updated between steps, that is a useful concurrency observation, not a reason to edit the evidence.

4. Step C — make a guarded change and record the resulting search event

Take the current sequence number and primary term from the routed GET and perform one conditional price change. This demonstrates that the trace can protect against stale replacement while still preserving the same business ID/routing key. Then append a search event to the data stream. The two resources now show why one course needs both mutable-index and append-only data-stream semantics.

PUT /atlasmart-products-v2/_doc/P-TRACE-01?routing=TENANT-TRACE&if_seq_no=<CURRENT_SEQ>&if_primary_term=<CURRENT_TERM>{  "product_id": "P-TRACE-01",  "name": "Atlas Trace Keyboard",  "category": "keyboards",  "price": 84.00,  "updated_at": "2026-09-10T17:35:00Z"}POST /atlasmart-products-v2/_refresh

POST /atlasmart-search-events/_doc{  "@timestamp": "2026-09-10T17:36:00Z",  "event_id": "SE-TRACE-01",  "customer_id": "C-TRACE",  "query": "atlas trace keyboard",  "result_count": 1}POST /atlasmart-search-events/_refreshGET /atlasmart-search-events/_search{  "query": {"term": {"event_id": "SE-TRACE-01"}}}

The product document is updated in place by stable identity in the current index; the event stream records a new timestamped observation. Later chapters will add templates, lifecycle policies, ingest pipelines and zero-downtime index changes without collapsing these workloads into one generic “JSON collection.”

5. Build the evidence bundle: what each observation proves

A useful course lab preserves these observations as text/JSON alongside the time and commands used. Do not publish secret-bearing headers, passwords, CA private keys or cloud credentials. Redact only secrets; keep version, index, shard and error metadata that makes diagnosis possible.

6. Reset Chapter 02 without damaging the Chapter 01 environment

The chapter reset removes only the new product index/alias, data stream and its template. It leaves the pinned containers, persistent volumes, network and Chapter 01 certificate/credentials in place for Chapter 03. Because deleting a data stream also deletes its backing indices, that operation is intentionally destructive to the stream fixture.

DELETE /atlasmart-products-v2DELETE /_data_stream/atlasmart-search-eventsDELETE /_index_template/atlasmart-search-events-template# Verify only the named Chapter 02 resources are gone.GET /_cat/indices/atlasmart-products-v2?v=trueGET /_data_stream/atlasmart-search-eventsGET /_index_template/atlasmart-search-events-template

If you want to continue directly into Chapter 03 instead of resetting, keep atlasmart-products-v2 and the stream, but record their exact mappings/settings first. Chapter 03 will deliberately revisit field types, and the course must know whether it is evolving an existing index or creating a clean mapping fixture.

Chapter 02 acceptance checklist

• All document metadata terms are defined and demonstrated with actual response shapes.
• OCC stale writes fail visibly with current sequence/primary-term semantics.
• Alias, index, data stream and backing index are kept distinct.
• Custom routing is correlated with shard selection and its missing-routing failure mode.
• Replica count is varied safely and allocation failures are explained, not ignored.
• Elasticsearch master and OpenSearch cluster_manager terminology remains product-specific.
• Write and search paths are traced through coordination, primary/replica and scatter/gather mechanisms.
• Partial-search handling is an explicit application correctness choice.
• No benchmark, latency, recovery or failover number is invented.
• All cleanup operations name only Chapter 02 course resources.

Production judgment

A durable search architecture must make five contracts explicit: business document identity, shard/routing topology, replication/failure tolerance, search freshness, and completeness under partial failure. Changing any one can alter latency, recovery time, write cost and correctness. Data streams solve a particular append-heavy lifecycle problem; aliases solve a different naming/cutover problem. Replica copies improve availability but are not backups. Routing can reduce fan-out but can create hot shards. The next chapter turns mappings and field types into the schema layer that determines how these documents are actually indexed and queried.

Summary and next step

You can now follow one document from JSON source and stable business identity through routing, primary-shard sequencing, replica policy, refresh and distributed search evidence. You can also distinguish mutable product indices from append-only data streams and explain node/control-plane roles. Chapter 03 uses this architecture to answer the next question: how should each field be mapped so indexing and queries have the intended semantics?

Authoritative references

• Elasticsearch optimistic concurrency control — Sequence-number/primary-term revision control.
• Elasticsearch near real-time search — Refresh-to-searchability mechanism.
• Elasticsearch routing field — Shard selection and custom-routing requirements.
• Elasticsearch data streams — Backing/write-index semantics for append-heavy data.
• OpenSearch document APIs — Document metadata and replication model.
• OpenSearch Refresh Index API — Current refresh/search visibility and production guidance.
• OpenSearch data streams — Backing indices, current write index and time-series constraints.
• OpenSearch search shard routing — Shard selection and routing for searches.