Refresh vs Flush vs Force Merge: Search Visibility, Durability, Segments, and Performance

Learning outcomes

AtlasMart operators see three tempting commands—refresh, flush, and force merge—and each seems to “make indexing more complete.” They solve different problems. Refresh changes what search can see. Flush establishes a Lucene commit and allows older translog generations to be discarded according to the engine’s recovery rules. Segment merge rewrites Lucene segments and can reclaim deletes. Confusing them can produce tiny segments, heavy I/O, long pauses, or false claims about durability.

1. Acknowledged does not mean searchable yet

Elasticsearch and OpenSearch are near-real-time search systems. A document mutation can be acknowledged before a periodic refresh opens a new searcher over recently written Lucene segment data. A direct GET by ID is designed for current document retrieval and can observe state before a search query sees the same mutation. That difference is exactly what the lab should prove; it is not evidence that the write was “lost.”

PUT atlasmart-products-v4-write-lab/_doc/P-720?refresh=false{"product_id":"P-720","name":"Laptop Stand","price":44.00,"stock":7,"updated_at":"2026-09-11T05:40:00Z","event_id":"evt-720"}GET atlasmart-products-v4-write-lab/_doc/P-720GET atlasmart-products-v4-write-lab/_search{"query":{"term":{"product_id":"P-720"}}}POST atlasmart-products-v4-write-lab/_refreshGET atlasmart-products-v4-write-lab/_search{"query":{"term":{"product_id":"P-720"}}}

The first search may already see the document if an automatic refresh happened before you query. That race is expected. To demonstrate the distinction reliably, use a long disposable refresh_interval and record timing rather than asserting that search must miss the document every time.

2. Request-level refresh choices

For Bulk requests, only shards that received operations participate in request-level refresh behavior. That detail matters when measuring latency and when a test incorrectly assumes every shard in the index has been refreshed by one small batch.

3. Flush is about recovery boundaries, not search visibility

Each engine uses a transaction log/translog so recently acknowledged operations can be recovered even when they are not yet represented in a durable Lucene commit. A flush creates a Lucene commit and starts a new translog generation, allowing older translog data to become unnecessary for recovery according to engine rules. Both products perform flushes automatically. Manual flush exists for operational cases; it is not something an application should issue after every mutation.

GET atlasmart-products-v4-write-lab/_stats?filter_path=indices.*.total.translog,indices.*.total.refresh,indices.*.total.flush,indices.*.primaries.segmentsGET atlasmart-products-v4-write-lab/_segmentsPOST atlasmart-products-v4-write-lab/_flushGET atlasmart-products-v4-write-lab/_stats?filter_path=indices.*.total.translog,indices.*.total.refresh,indices.*.total.flush,indices.*.primaries.segments

A manual flush is not “make my latest write durable.” Durability of an acknowledged write depends on translog durability configuration, primary/replica acknowledgement conditions, storage behavior, and the exact operation. Nor is a flush a backup: it does not create an independent recovery copy outside the cluster.

4. Force merge rewrites segments; it does not certify safety

Lucene segments are immutable. Background merge policy combines smaller segments and eventually reclaims space from deleted documents. Force merge asks the engine to perform additional merge work. Official guidance warns against routine force merge on actively written indices because it can create very large segments, consume disk and I/O, and interfere with normal merge behavior. Later lifecycle chapters can apply force merge after rollover when a backing index becomes read-only.

# First stop writes to this disposable index and verify that assumption.POST atlasmart-products-v4-write-lab/_forcemerge?max_num_segments=1GET atlasmart-products-v4-write-lab/_segments

5. Evidence lab: build a state-transition timeline

Create ten small documents with refresh=false, capture the write acknowledgement timestamps, GET/search visibility, stats, and segment inventory. Then perform one refresh, one flush, and—only after stopping writes—an optional force merge. Record what changed after each step. Do not claim that any operation made p99 latency “better”; the lab is about causal state evidence, not invented performance.

Production judgment

Freshness is a product SLO and must be measured separately from write acknowledgement and disaster recovery. Tight refresh intervals improve visibility but increase segment churn; long intervals improve write efficiency but increase search lag. Manual flush and force merge are operational controls, not application acknowledgements. Capacity plans must leave disk and I/O headroom for translogs and merges, and benchmark reports must state refresh policy and segment state because they materially affect indexing/search results.

Summary and next step

You can now name which mechanism controls visibility, recovery boundaries, and segment rewriting. The next lesson uses sequence numbers and primary terms to prevent a stale client from overwriting a newer document, then connects that OCC contract to external versioning and idempotency keys.

Authoritative references

• Elastic Bulk API — Current bulk action, per-item result, refresh, versioning, routing, and OCC reference.
• Elastic refresh parameter — Current visibility semantics for index/update/delete/bulk requests.
• Elastic optimistic concurrency control — Sequence-number and primary-term concurrency semantics.
• OpenSearch Bulk API — Current NDJSON, per-item failure, OCC, versioning, and refresh behavior.
• OpenSearch Document APIs — Current document operation and sequence-number/primary-term overview.
• OpenSearch Refresh API — Near-real-time visibility and refresh guidance.
• OpenSearch Flush API — Flush and transaction-log recovery semantics.
• OpenSearch Force Merge API — Segment merge behavior and write-active warnings.
• Elastic Flush API — Current Elasticsearch flush API.
• Elastic Force Merge API — Current Elasticsearch force-merge API.