Index-Time vs Search-Time Analysis and Why Token Streams Define Search Semantics

Learning outcomes

AtlasMart search now has a stable mapping and write pipeline, but users still judge the system by what text does and does not match. A search engine does not index the original sentence as one opaque value: it applies an analysis chain that can rewrite characters, split text into tokens, and transform those tokens before Lucene records searchable terms. This chapter makes that transformation observable so relevance changes are engineered and tested rather than guessed.

1. The indexed unit is the token stream, not the sentence

A character filter transforms the input character stream before tokenization—for example stripping HTML or applying a configured character mapping. A tokenizer then emits token boundaries. An analyzer has exactly one tokenizer. Ordered token filters may lowercase, stem, remove, expand, or otherwise transform those tokens. Each emitted token has text plus metadata such as a position and character offsets. Lucene builds its inverted index from those analyzed terms.

That model immediately explains why copying the visible product title into a query is insufficient to predict matching. The index analyzer determines what terms exist in the index; the search analyzer determines which terms the user query asks for. Search succeeds only when those two contracts are compatible enough for the query semantics.

POST _analyze
{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "filter": ["lowercase", "asciifolding"],
  "text": "<b>Café USB-C Charger</b>"
}

For this fixture, expect terms equivalent to cafe, usb, c, and charger, with positions increasing through the stream. The exact response also reports offsets/types. That evidence proves what this analysis request emitted; it does not prove how an existing field is mapped unless you invoke the field's configured analyzer.

2. Index-time and search-time analysis are separate contracts

For a text field, the field's analyzer is used when text is indexed. By default the same analyzer is also used for full-text search, but a field may declare a separate search_analyzer. That separation is useful when indexing needs extra terms—autocomplete prefixes are the classic example—while queries should remain compact. It is also where synonym operations often belong because changing search-time expansion can avoid rewriting every indexed document.

3. Build AtlasMart's first explicit analysis contract

Product names need full-text search plus an exact sortable/facetable representation. Descriptions need full-text search but not an exact-value aggregation. SKUs and category codes are identifiers: they should not be split into words. This is query-driven schema design from Chapter 03 carried into analysis.

PUT atlasmart-products-analysis-v1
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "analysis": {
      "normalizer": {
        "atlas_keyword_lc": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      },
      "analyzer": {
        "atlas_name_index": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        },
        "atlas_name_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "atlas_name_index",
        "search_analyzer": "atlas_name_search",
        "fields": {"raw": {"type": "keyword", "normalizer": "atlas_keyword_lc"}}
      },
      "description": {"type": "text", "analyzer": "atlas_name_index"},
      "sku": {"type": "keyword", "normalizer": "atlas_keyword_lc"},
      "category": {"type": "keyword", "normalizer": "atlas_keyword_lc"}
    }
  }
}

PUT atlasmart-products-analysis-v1/_doc/p-100?refresh=true
{"name":"Café USB-C Charger","description":"65W fast charger for laptops","sku":"USB-C-65W","category":"POWER"}

PUT atlasmart-products-analysis-v1/_doc/p-101?refresh=true
{"name":"Cafe Travel Mug","description":"Insulated steel mug","sku":"MUG-450","category":"KITCHEN"}

4. The deliberate failure: term query against analyzed user text

A common bug is to send a user's phrase directly to a term query on name. Term-level queries do not run the field's full-text analyzer. The indexed field contains analyzed terms such as cafe and charger, not the original phrase as one term. Use match when you want query text analyzed, or query name.raw when you truly want normalized exact-value semantics.

GET atlasmart-products-analysis-v1/_search
{"query":{"term":{"name":"Café USB-C Charger"}}}

GET atlasmart-products-analysis-v1/_search
{"query":{"match":{"name":"cafe charger"}}}

GET atlasmart-products-analysis-v1/_search
{"query":{"term":{"name.raw":"cafe usb-c charger"}}}

The first request is intentionally misleading. The second invokes the field search analyzer. The third targets the single-token normalized multi-field. This distinction becomes essential when Chapter 06 adds Query DSL composition.

5. Verification checklist and reset

• GET atlasmart-products-analysis-v1/_mapping shows the named index and search analyzers plus name.raw.
• POST atlasmart-products-analysis-v1/_analyze with field: "name" exposes the field's index analyzer token stream.
• A match search for unaccented cafe can match the accented product because folding is part of both relevant analysis paths.
• sku remains one normalized keyword token; it is not split at hyphens.
• Deleting this disposable index is sufficient reset for this lesson.

DELETE atlasmart-products-analysis-v1

Production judgment

Analysis is part of the search API contract. Record analyzer names/version, representative token fixtures, language assumptions, and the queries that depend on them. Measure recall, precision, index growth, and query latency on judged traffic before deploying a broader analyzer. More tokens can improve recall while increasing postings, query work, and false positives. Analyzer changes require the same change discipline as schema migrations.

Next, we isolate the tokenizer stage and compare standard, keyword, whitespace, pattern, n-gram, edge-ngram, hierarchy, and language-aware patterns on the same AtlasMart inputs.

Summary and next step

This lesson’s analysis contract, observable evidence, failure boundaries, and production implications should now be concrete enough to verify rather than assume.

Authoritative references

• Elastic: standard analyzer — Current reference for the default standard analysis chain.
• Elastic: Analyze API — Token/position/offset inspection for built-in, field-bound, and custom analysis.
• Elastic: tokenizer reference — Official tokenizer families and behavior.
• Elastic: configure synonyms — Search-time synonym-set workflow, validation, and reload behavior.
• Elastic: reload search analyzers API — Reload semantics for file-backed, updateable search analyzers.
• OpenSearch: text analysis — Analyzer, tokenizer, token-filter, and normalizer fundamentals.
• OpenSearch: Analyze API — Index/global analysis requests and detailed token evidence.
• OpenSearch: synonym_graph token filter — Multiword synonym graph behavior.
• OpenSearch: refresh search analyzer — Product-specific analyzer refresh workflow and plugin requirement.
• OpenSearch: normalizers — Single-token keyword normalization constraints.