Segment Merging, Deleted Docs, Merge Policy, Force Merge Risks, and Write Amplification

Learning outcomes

AtlasMart updates prices and inventory frequently. Store bytes rise even though logical document count barely changes, and segment counts fluctuate. The reason is immutable indexing: updates create replacement documents, deletes mark old Lucene documents as deleted, and background merges later rewrite segments to consolidate live data and reclaim space.

1. Immutable segments turn updates into future merge work

When AtlasMart updates P-1430, Lucene does not overwrite bytes in the segment that contains the old version. The old Lucene document becomes deleted (logically invisible), and a new document is indexed. Deletes behave similarly: a deletion marker makes a document invisible, while physical bytes are reclaimed when a merge rewrites affected segments without the deleted documents.

This is why docs.deleted can be non-zero and why store size can lag behind logical deletion. It is also why update-heavy workloads create more write amplification than append-only workloads.

DELETE atlasmart-merge-lab
PUT atlasmart-merge-lab
{
  "settings":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"},
  "mappings":{"properties":{"sku":{"type":"keyword"},"name":{"type":"text"},"price":{"type":"scaled_float","scaling_factor":100}}}
}

PUT atlasmart-merge-lab/_doc/P-1430
{"sku":"P-1430","name":"AtlasMart Dock","price":79.00}
POST atlasmart-merge-lab/_refresh

PUT atlasmart-merge-lab/_doc/P-1430
{"sku":"P-1430","name":"AtlasMart Dock","price":74.00}
POST atlasmart-merge-lab/_refresh

GET atlasmart-merge-lab/_stats/docs,segments,merge,store
GET atlasmart-merge-lab/_segments

2. Background merge is normal housekeeping with real I/O cost

Lucene’s merge policy selects sets of segments and writes their live documents into larger replacement segments. Once readers no longer need old files, those files can be deleted. Merging reduces segment fragmentation and reclaims deleted documents, but it reads and writes substantial bytes and consumes CPU, disk bandwidth and filesystem-cache capacity.

A high merge time counter does not automatically mean a fault—the total is cumulative. Diagnose with rates and concurrency: current merges, bytes/docs being merged, indexing/search latency, disk utilization and whether merge backlog grows under steady load.

GET atlasmart-merge-lab/_stats/merge,docs,segments,store
GET _nodes/stats/indices,fs,process,jvm
GET _cat/segments/atlasmart-merge-lab?v
GET atlasmart-merge-lab/_segments

3. Write amplification is workload + mapping + merge behavior

If the application writes 1 GB of new source data, the storage device may write more than 1 GB because analysis builds index structures and merges rewrite existing segments. Updates can amplify further because old versions remain until merged. There is no universal amplification factor: measure it using host/device metrics under a fixed dataset and workload.

Refresh frequency interacts with this system. Very frequent refreshes can produce more small searchable segments that must later be merged. Chapter 04’s “refresh every write” anti-pattern therefore reappears here as storage/merge debt.

4. Force merge is not a maintenance ritual for hot indices

The force merge API requests aggressive merging, for example toward one segment. Both products warn that it is resource intensive and can temporarily require significant additional disk space. OpenSearch documents that a one-segment force merge can temporarily double shard storage; Elastic similarly recommends force merge only after an index has stopped receiving writes in lifecycle scenarios.

Force-merging a hot write index can create very large segments, compete with indexing/search, and future writes immediately create new segments again. It does not make writes durable, does not replace flush, and does not replace snapshots.

5. Safe lab: clone/reindex to a disposable read-only copy, then force merge

PUT atlasmart-merge-coldcopy
{
  "settings":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"-1"},
  "mappings":{"properties":{"sku":{"type":"keyword"},"name":{"type":"text"},"price":{"type":"scaled_float","scaling_factor":100}}}
}

POST _reindex
{
  "source":{"index":"atlasmart-merge-lab"},
  "dest":{"index":"atlasmart-merge-coldcopy"}
}
POST atlasmart-merge-coldcopy/_refresh
PUT atlasmart-merge-coldcopy/_settings
{"index.blocks.write":true}

GET atlasmart-merge-coldcopy/_stats/docs,store,segments,merge
GET atlasmart-merge-coldcopy/_segments
POST atlasmart-merge-coldcopy/_forcemerge?max_num_segments=1
GET atlasmart-merge-coldcopy/_stats/docs,store,segments,merge
GET atlasmart-merge-coldcopy/_segments

Record free disk before starting. The fixture is intentionally tiny; it teaches state transitions, not production performance. On real large shards, force merge can run for a long time and consume large temporary disk capacity.

6. Merge policy is not a first-line tuning surface

Changing merge-policy settings without a workload trace can trade one bottleneck for another. Start with mapping correctness, sensible refresh/freshness requirements, bulk indexing, adequate fast storage and lifecycle design. If merge debt remains causal, benchmark one change at a time with the exact workload.

Managed services may restrict low-level settings or perform storage/lifecycle operations differently. Document the deployment flavor before copying a self-managed setting.

Production judgment

Treat merge activity as background debt service. The objective is not zero merges; it is sustainable merge throughput while foreground indexing/search meets SLOs and disk headroom remains safe. Deleted-doc percentage alone is not an incident—connect it to store growth, merge backlog and user-visible behavior.

Summary and next step

You can now interpret deleted-doc and merge evidence as consequences of immutable segments rather than as mysteries. Lesson 4 moves from logical storage structures to the operating system: compression, mmap, filesystem cache, cold/warm behavior and storage latency.

Authoritative references

• Elastic index segments API — Low-level Lucene segment metadata for index shards.
• Elastic index stats API — Refresh, flush, merge, segment, translog, docs and store statistics.
• Elastic translog settings — Flush as Lucene commit plus new translog generation and request durability semantics.
• Elastic tune for indexing speed — Filesystem-cache and indexing guidance.
• Elastic tune for search speed — Filesystem cache, storage latency and local-versus-remote storage guidance.
• OpenSearch Index Segments API — Segment committed/search flags and Lucene writer-version evidence.
• OpenSearch Index Stats API — Refresh, flush, merge, segments, translog and deleted-document statistics.
• OpenSearch Refresh API — Search-visibility semantics and refresh cost guidance.
• OpenSearch Flush API — Flush and transaction-log lifecycle semantics.
• OpenSearch Force Merge API — Merge/deleted-document behavior and temporary disk-space risk.
• Elastic force merge API — Current force merge behavior and operational cautions.
• Elastic ILM force merge action — Lifecycle-stage force merge guidance.
• OpenSearch index settings — Merge/translog/index settings and defaults.