Standard, Keyword, Whitespace, Pattern, N-Gram/Edge-N-Gram, Path, and Language Tokenization Patterns

Learning outcomes

AtlasMart search now has a stable mapping and write pipeline, but users still judge the system by what text does and does not match. A search engine does not index the original sentence as one opaque value: it applies an analysis chain that can rewrite characters, split text into tokens, and transform those tokens before Lucene records searchable terms. This chapter makes that transformation observable so relevance changes are engineered and tested rather than guessed.

1. Tokenizers answer one question: where are token boundaries?

The tokenizer receives the character-filter output and turns it into a token stream. The same characters can become one token, many word tokens, overlapping substrings, anchored prefixes, or hierarchy paths depending on the tokenizer. Choosing a tokenizer is therefore choosing the primitive terms available to later token filters and the inverted index.

2. Run the same text through several tokenizer contracts

POST _analyze
{"tokenizer":"standard","text":"USB-C Café Charger/65W"}

POST _analyze
{"tokenizer":"keyword","text":"USB-C Café Charger/65W"}

POST _analyze
{"tokenizer":"whitespace","text":"USB-C Café Charger/65W"}

POST _analyze
{"tokenizer":{"type":"pattern","pattern":"[-_/]+"},"text":"USB-C/65W_POWER"}

Do not memorize only token text. Inspect positions and offsets: phrase queries and graph-producing filters build on positional structure. Also note that keyword tokenizer is not the same as a keyword field. The tokenizer is an analysis component; the field type has mapping, doc-values, exact-match, sort, and aggregation semantics.

3. Partial-word retrieval: n-gram vs edge n-gram

An n-gram tokenizer emits sliding substrings; edge n-gram emits prefixes anchored at a token edge. Both can multiply indexed terms. For AtlasMart autocomplete, a controlled edge-ngram index analyzer with a normal search analyzer is usually easier to reason about than applying edge-ngram to every query token. The correct gram range depends on product vocabulary, result quality, and measured resource cost—not folklore.

PUT atlasmart-tokenizers-v1
{
  "settings": {"number_of_shards":1,"number_of_replicas":0,
    "analysis": {
      "tokenizer": {"atlas_edge":{"type":"edge_ngram","min_gram":2,"max_gram":12,"token_chars":["letter","digit"]}},
      "analyzer": {
        "atlas_autocomplete_index":{"type":"custom","tokenizer":"atlas_edge","filter":["lowercase","asciifolding"]},
        "atlas_autocomplete_search":{"type":"custom","tokenizer":"standard","filter":["lowercase","asciifolding"]}
      }
    }
  },
  "mappings":{"properties":{"name":{"type":"text","analyzer":"atlas_autocomplete_index","search_analyzer":"atlas_autocomplete_search"}}}
}

POST atlasmart-tokenizers-v1/_analyze
{"analyzer":"atlas_autocomplete_index","text":"Charger"}

POST atlasmart-tokenizers-v1/_analyze
{"analyzer":"atlas_autocomplete_search","text":"Char"}

4. Paths and language-aware analysis solve different problems

POST _analyze
{"tokenizer":{"type":"path_hierarchy","delimiter":"/"},"text":"electronics/power/chargers"}

A hierarchy tokenizer can emit ancestor-like path terms such as electronics, electronics/power, and the full path. A language analyzer, by contrast, is a packaged analyzer that may combine standard-like tokenization with language-specific lowercasing, stopwords, normalization, and stemming. Language behavior must be evaluated with real catalog/query pairs; stemming can merge forms that users consider different, and multilingual fields may need separate language fields or routing rather than one aggressive analyzer.

POST _analyze
{"analyzer":"standard","text":"Running chargers for laptops"}

POST _analyze
{"analyzer":"english","text":"Running chargers for laptops"}

5. AtlasMart tokenizer regression fixture

Create a small table of inputs and expected token invariants before changing production analyzers: punctuation-heavy SKU, accented brand, hyphenated product phrase, short autocomplete prefix, category path, and at least one language-specific sentence. Store expected tokens and positions in source control and compare both products where portability matters.

DELETE atlasmart-tokenizers-v1

Production judgment

Tokenizer changes alter the vocabulary of the index and therefore normally require reindexing. Before adopting partial-word or language-specific tokenization, quantify term growth, recall/precision changes, phrase behavior, and p95/p99 search latency under representative load. Keep a simple field for ordinary full-text search when a specialized autocomplete or hierarchy field serves only one interaction.

Next, we keep token boundaries fixed and study filters that normalize, remove, stem, expand, or combine those tokens.

Summary and next step

This lesson’s analysis contract, observable evidence, failure boundaries, and production implications should now be concrete enough to verify rather than assume.

Authoritative references

• Elastic: standard analyzer — Current reference for the default standard analysis chain.
• Elastic: Analyze API — Token/position/offset inspection for built-in, field-bound, and custom analysis.
• Elastic: tokenizer reference — Official tokenizer families and behavior.
• Elastic: configure synonyms — Search-time synonym-set workflow, validation, and reload behavior.
• Elastic: reload search analyzers API — Reload semantics for file-backed, updateable search analyzers.
• OpenSearch: text analysis — Analyzer, tokenizer, token-filter, and normalizer fundamentals.
• OpenSearch: Analyze API — Index/global analysis requests and detailed token evidence.
• OpenSearch: synonym_graph token filter — Multiword synonym graph behavior.
• OpenSearch: refresh search analyzer — Product-specific analyzer refresh workflow and plugin requirement.
• OpenSearch: normalizers — Single-token keyword normalization constraints.