Immutable Segments, Inverted Index/Postings, Stored Fields, Doc Values, Norms, and Term Dictionaries

Learning outcomes

AtlasMart search works because every Elasticsearch/OpenSearch shard is backed by a Lucene index. A Lucene index is not one mutable table file: it is a collection of immutable segments. Each segment contains purpose-built structures for matching, scoring, sorting, aggregating and retrieving fields. Understanding those structures prevents incorrect claims such as “_source is the inverted index” or “doc values are just another cache.”

1. One shard is one Lucene index composed of segments

A segment is an immutable unit written by Lucene. New indexing work accumulates in memory and, after refresh, becomes visible through newly opened segment readers. Existing segment bytes are not edited to change a document. An update is logically a delete of the old Lucene document plus indexing of a new one; physical reclamation waits for merge work.

Immutability simplifies concurrent search because readers can continue using old segment files while new readers open newer segments. The cost is write amplification: updates, deletes and merges eventually rewrite data.

GET _cat/shards/atlasmart-products*?v&h=index,shard,prirep,state,docs,store,node
GET atlasmart-products*/_segments
GET atlasmart-products*/_stats/docs,store,segments?include_segment_file_sizes=true
GET _nodes/jvm

2. Inverted index = term dictionary + postings, not the original JSON

For an analyzed text field, Chapter 05’s analyzer produces tokens. Lucene’s term dictionary organizes distinct indexed terms so a query can locate a term efficiently. A postings list records which Lucene document IDs contain that term and may also retain frequencies, positions and offsets depending on mapping/index options.

A match query analyzes user text and looks up resulting terms in these structures. A phrase query needs positional information. This is why changing an analyzer changes the indexed token contract and normally requires reindexing: the old postings do not magically change when a new analyzer definition exists.

PUT atlasmart-storage-lab
{
  "settings":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"-1"},
  "mappings":{"properties":{
    "sku":{"type":"keyword"},
    "name":{"type":"text","fields":{"raw":{"type":"keyword"}}},
    "price":{"type":"scaled_float","scaling_factor":100},
    "popularity":{"type":"integer"}
  }}
}

PUT atlasmart-storage-lab/_doc/P-1401
{"sku":"P-1401","name":"Café USB-C Dock","price":89.00,"popularity":42}

POST atlasmart-storage-lab/_refresh
GET atlasmart-storage-lab/_termvectors/P-1401?fields=name&positions=true&offsets=true&term_statistics=true

Term vectors are optional evidence derived from indexed terms and settings; they are not the global term dictionary itself. Use them to inspect token/position behavior for a document, then use _analyze when debugging the analyzer pipeline.

3. Stored fields, _source and doc values answer different questions

4. Multi-fields deliberately create multiple index structures

The AtlasMart name field is indexed as analyzed text and also as name.raw keyword. That is not one representation with two query modes; it is two field mappings with different indexed structures. The text field supports lexical matching through analyzed terms, while the keyword subfield supports exact value semantics and typically doc values for sorting/aggregation.

This is a storage-for-capability tradeoff. Multi-fields, positions, norms and doc values should exist because a query or aggregation requires them, not because “more indexing is safer.”

GET atlasmart-storage-lab/_mapping
GET atlasmart-storage-lab/_field_caps?fields=name,name.raw,price,popularity
GET atlasmart-storage-lab/_search
{
  "query":{"match":{"name":"cafe dock"}},
  "sort":[{"price":"asc"}]
}

5. Boundary case: do not infer physical bytes from logical field names

Compression, codec selection, compound files and Lucene versions can change the physical representation. A field that looks small in JSON can create more index bytes because of terms/positions/doc values; conversely, repeated values may compress efficiently. Measure store, segment file sizes and actual workload behavior.

Elasticsearch 9.5.3 and OpenSearch 3.8.0 both expose segment version metadata. Use it as diagnostic evidence, not as a promise that every segment in a rolling-upgraded cluster was written by one identical Lucene patch.

6. AtlasMart lab and production judgment

Index a deterministic 1,000-document fixture twice: once with only fields required by the query contract and once with an intentionally redundant multi-field or unnecessary scoring feature. Refresh both, capture mapping, segment, store and query behavior, and explain the size/capability difference. Do not fabricate a “% faster” result: record your machine, warm/cold state, corpus and measured values.

Production decisions should link every mapping/storage feature to a user-visible capability. Segment internals explain cost, but application code should depend on supported mapping/query semantics rather than codec/file layouts.

• Capture _segments and _stats/store,segments after the same document count.
• Verify matching, sorting and aggregation behavior before comparing size.
• Keep the fixture and commands in source control so mapping changes are regression-testable.
• Cleanup: delete only atlasmart-storage-lab* disposable indices.

Summary and next step

You can now name the structures inside a shard without conflating them. Lesson 2 follows one acknowledged write through in-memory indexing, real-time GET, refresh, translog durability and flush/Lucene commit so visibility and durability are separated precisely.

Authoritative references

• Elastic index segments API — Low-level Lucene segment metadata for index shards.
• Elastic index stats API — Refresh, flush, merge, segment, translog, docs and store statistics.
• Elastic translog settings — Flush as Lucene commit plus new translog generation and request durability semantics.
• Elastic tune for indexing speed — Filesystem-cache and indexing guidance.
• Elastic tune for search speed — Filesystem cache, storage latency and local-versus-remote storage guidance.
• OpenSearch Index Segments API — Segment committed/search flags and Lucene writer-version evidence.
• OpenSearch Index Stats API — Refresh, flush, merge, segments, translog and deleted-document statistics.
• OpenSearch Refresh API — Search-visibility semantics and refresh cost guidance.
• OpenSearch Flush API — Flush and transaction-log lifecycle semantics.
• OpenSearch Force Merge API — Merge/deleted-document behavior and temporary disk-space risk.
• Elastic term vectors API — Token/term statistics for document-level inspection.
• OpenSearch term vectors API — Document term/position/offset inspection.
• Apache Lucene core — Lower-level implementation reference; not an Elasticsearch/OpenSearch application contract.