Use Analyze APIs to Debug Unexpected Matching and Build Tests for Tokenization/Normalization Contracts

Learning outcomes

AtlasMart search now has a stable mapping and write pipeline, but users still judge the system by what text does and does not match. A search engine does not index the original sentence as one opaque value: it applies an analysis chain that can rewrite characters, split text into tokens, and transform those tokens before Lucene records searchable terms. This chapter makes that transformation observable so relevance changes are engineered and tested rather than guessed.

1. Debug from evidence, not from the visible sentence

When an AtlasMart query unexpectedly matches or misses, capture four things: the field mapping/settings, the index-time token stream for the document text, the search-time token stream for the user text, and the actual query type. This avoids the common error of staring at _source and assuming the engine searches those exact characters.

POST atlasmart-products-analysis-v1/_analyze
{"field":"name","text":"Café USB-C Charger"}

POST atlasmart-products-analysis-v1/_analyze
{"analyzer":"atlas_name_search","text":"cafe charger"}

Use the index-scoped API when you need custom analyzer names from that index. Global _analyze is useful for built-in or fully inline chains. With custom components, tying the probe to the index reduces accidental mismatch between the test and deployed settings.

2. Positions and offsets explain phrase and highlighting boundaries

Token position is logical order in the token stream; some filters create gaps or alternatives at the same position. Offsets map tokens back to character ranges in the original/filtered text and can matter for highlighting/debugging. Token type is implementation metadata useful for inspection but should not be treated as a stable business API unless the design explicitly depends on documented behavior.

POST _analyze
{
  "tokenizer":"standard",
  "filter":["lowercase",{"type":"shingle","min_shingle_size":2,"max_shingle_size":2,"output_unigrams":true}],
  "text":"wireless phone charger",
  "explain":true
}

Detailed/explain-style analysis is diagnostic. It can show how each stage transformed the stream, but it is not a relevance benchmark and should not be used to infer production throughput.

3. Term vectors connect a document to indexed-term evidence

Term-vector APIs can expose terms and statistics for document fields when supported/configured. They are useful for debugging why a particular document contributes certain terms, but the returned statistics depend on index state and options. Treat them as evidence about terms, not as a guarantee of final ranking.

GET atlasmart-products-analysis-v1/_termvectors/p-100
{
  "fields":["name","description"],
  "term_statistics":false,
  "field_statistics":false,
  "positions":true,
  "offsets":true
}

4. Build deterministic token-contract tests

A lightweight test harness can call _analyze, extract ordered token text/positions, and compare them with version-controlled fixtures. Keep portability fixtures for the common analyzers and separate product-specific fixtures for synonym administration or plugin components. A token test should fail loudly when a release, plugin, configuration, or synonym change alters the contract.

{
  "case": "accented-product-name",
  "index": "atlasmart-products-analysis-v1",
  "analyzer": "atlas_name_search",
  "input": "Café Charger",
  "expected_tokens": ["cafe", "charger"]
}
{
  "case": "sku-normalization",
  "field": "sku",
  "input": "USB-C-65W",
  "expected_single_token": "usb-c-65w"
}

for each fixture:
  response = POST /{index}/_analyze with fixture analyzer/field/input
  actual = ordered response.tokens[].token
  assert actual == expected_tokens
  record server product + exact version + analyzer settings hash

then run judged search queries separately;
do not treat token equality as proof of relevance quality or latency.

5. Failure drills: make the wrong assumption reproducible

Use small fixtures to demonstrate three recurring failures: (1) a term query does not analyze free-form user text; (2) an edge-ngram search analyzer produces more query terms than intended; (3) a synonym/stemmer change alters phrase or recall behavior. For each, capture the before token stream, the failing query/result, the corrected analyzer/query, and the after result. This creates a diagnosis record rather than an anecdote.

6. Chapter acceptance gate and safe cleanup

• Every analyzer used by AtlasMart has at least one positive and one boundary token fixture.
• Product-name punctuation, accents, case, SKUs, category paths, multiword synonyms, and representative language text are covered.
• Index-time analyzer changes are tied to a versioned reindex plan; search-only synonym changes use the exact supported product workflow.
• Query tests distinguish match/phrase behavior from term-level exact lookup.
• Relevance and performance are tested separately from token correctness.

DELETE atlasmart-products-analysis-v1
DELETE atlasmart-tokenizers-v1
DELETE atlasmart-filters-v1
DELETE atlasmart-normalizer-v1
DELETE atlasmart-products-analysis-v2
DELETE atlasmart-es-syn-v1
DELETE atlasmart-os-syn-v1

Production judgment

Text analysis deserves version control, tests, ownership, and rollback just like application code. Record analyzer settings, synonym sources, plugin/product versions, and judged query fixtures. Watch for relevance regressions, token/index growth, cache and CPU effects, and query tail latency after changes. Keep multilingual behavior explicit; do not assume English stemming/folding strategies generalize to other languages.

Chapter 06 builds on this evidence by composing Query DSL full-text, term-level, boolean, range, nested, geo, and safe user-intent queries. The analysis contract you can now inspect explains why those queries match what they match.

Summary and next step

This lesson’s analysis contract, observable evidence, failure boundaries, and production implications should now be concrete enough to verify rather than assume.

Authoritative references

• Elastic: standard analyzer — Current reference for the default standard analysis chain.
• Elastic: Analyze API — Token/position/offset inspection for built-in, field-bound, and custom analysis.
• Elastic: tokenizer reference — Official tokenizer families and behavior.
• Elastic: configure synonyms — Search-time synonym-set workflow, validation, and reload behavior.
• Elastic: reload search analyzers API — Reload semantics for file-backed, updateable search analyzers.
• OpenSearch: text analysis — Analyzer, tokenizer, token-filter, and normalizer fundamentals.
• OpenSearch: Analyze API — Index/global analysis requests and detailed token evidence.
• OpenSearch: synonym_graph token filter — Multiword synonym graph behavior.
• OpenSearch: refresh search analyzer — Product-specific analyzer refresh workflow and plugin requirement.
• OpenSearch: normalizers — Single-token keyword normalization constraints.