Benchmark Vector Recall, p99 Latency, Index Build Time, Memory, and Filter Selectivity with Representative Queries

Learning outcomes

1. AtlasMart problem: the fastest demo loses the products users need

A developer benchmarks one query against warm cache, reports average latency, and picks the configuration with the smallest number. The chosen system misses relevant filtered products for a minority tenant and has unpredictable p99 during indexing. The benchmark optimized a demo, not a service.

2. Benchmark contract

3. Deterministic larger fixture generator

The eight-product fixture proves semantics; it is too small to prove performance. Generate a larger synthetic corpus only for systems mechanics. It still is not a substitute for representative production embeddings.

import json, math, random
R=random.Random(25092026)
DIMS=32; N=5000

def unit(v):
    n=math.sqrt(sum(x*x for x in v)); return [x/n for x in v]

with open("atlasmart-v25.ndjson","w",encoding="utf-8") as f:
    for i in range(N):
        # Four broad clusters; deterministic noise creates realistic-ish neighborhoods.
        c=i % 4
        center=[0.0]*DIMS
        for j in range(8): center[c*8+j]=1.0
        v=unit([center[j] + R.gauss(0,0.25) for j in range(DIMS)])
        doc={"product_id":f"S-{i:05d}","tenant_id":f"tenant-{i%5}",
             "category":f"c{c}","embedding_model":"synthetic-v25","embedding":v}
        f.write(json.dumps({"index":{"_id":doc["product_id"]}})+"\n")
        f.write(json.dumps(doc)+"\n")
print("generated",N,"documents; seed=25092026; dims=32")

Label every chart/table from this corpus synthetic mechanics. Production decisions require real embedding distributions, query vectors, filter selectivity, and judged relevance.

4. Exact ground truth harness

import heapq, math

def cosine(a,b):
    return sum(x*y for x,y in zip(a,b))/(math.sqrt(sum(x*x for x in a))*math.sqrt(sum(y*y for y in b)))

def exact_topk(docs, q, k, predicate=lambda d: True):
    scored=((cosine(q,d["embedding"]), d["product_id"]) for d in docs if predicate(d))
    return [pid for score,pid in heapq.nlargest(k, scored)]

# Persist exact IDs for every benchmark query/filter tuple.
# The exact file is the quality oracle used for both Elasticsearch and OpenSearch.

5. Query matrix

query_id  k   filter_class           concurrency  cache_state
Q01       10  none                   1            cold+warm
Q02       10  tenant=20% corpus      4            warm
Q03       10  tenant+category=5%     8            warm
Q04       50  none                   4            warm
Q05       10  highly-selective <1%   4            warm
# Add real AtlasMart query classes before production approval.

6. Timing harness rules

• Use a monotonic client timer around the full HTTP request; record status/error and returned IDs.
• Run a documented warm-up phase but keep cold-start observations separate rather than discarding them silently.
• Use fixed concurrency and connection settings per run.
• Do not run indexing/build and query benchmarks together unless the test is intentionally mixed-load.
• Capture cluster/node/index stats before and after so tail latency can be correlated with GC, CPU, disk, merge, cache, rejection, and recovery signals from Chapters 14–16 and 24.
• Repeat enough samples to make p99 meaningful; state sample count.

7. Result schema: no invented numbers

run_id,product,version,engine,index_options,model,dims,docs,shards,replicas,query_set,filter_class,k,candidates_or_ef,recall_at_k,p50_ms,p95_ms,p99_ms,index_build_s,index_bytes,memory_evidence,sample_count,notes
RUN-001,elasticsearch,9.5.3,elastic,int8_hnsw,synthetic-v25,32,5000,1,0,Q01,none,10,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,fill from run
RUN-002,opensearch,3.8.0,lucene,hnsw,synthetic-v25,32,5000,1,0,Q01,none,10,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,MEASURED,fill from run

8. Filter selectivity must be recorded

Define selectivity = eligible_docs / total_docs. Report it for every filtered query. Two filters with the same syntax can have radically different selectivity by tenant or category, which changes both exact cost and ANN behavior. If a filter leaves fewer than k eligible documents, the correct result can contain fewer than k hits; do not mark that as ANN recall loss.

9. Build-time and recovery evidence

Time bulk ingestion, refresh completion, and the period until the index is in the intended searchable state. Record store size after merges stabilize according to a documented policy; do not force-merge a hot benchmark index merely to make a chart smaller. For production readiness, also test restart/recovery and snapshot/reindex procedures because an index that searches quickly but cannot be recovered within the service objective is not acceptable.

10. Decision gates

11. Cleanup/reset

DELETE /atlasmart-vector-es-v25
DELETE /atlasmart-vector-os-v25
# Delete only the product/index that exists on the cluster you are connected to.
# Remove generated atlasmart-v25.ndjson and benchmark outputs after archiving the evidence bundle.
# Do not delete shared course indices such as atlasmart-products-v1.

12. Bridge to Chapter 26

Chapter 25 establishes the vector contract and measurement discipline. Chapter 26 adds model invocation, sparse/neural retrieval, lexical+dense/sparse hybrid fusion, normalization, reciprocal-rank-like fusion, and reranking. The same rule carries forward: every new retrieval signal needs a versioned contract, a judged evaluation set, operational limits, and measurable failure behavior.

References and version checks

• Elastic dense_vector field — dimensions, similarity, exact/script scoring, HNSW and quantized index options.
• Elastic kNN search — approximate versus brute-force kNN, filters, nested vector search, and query forms.
• Elastic sparse_vector field — weighted sparse features, pruning, and specialized query boundaries.
• OpenSearch vector index creation — knn_vector, dimensions, workload modes, and storage choices.
• OpenSearch methods and engines — HNSW/IVF and Lucene/Faiss/NMSLIB/JVector capability boundaries.
• OpenSearch k-NN query — k, filters, and method parameters such as ef_search.
• OpenSearch efficient k-NN filtering — filter-aware exact/approximate behavior.
• OpenSearch vector quantization — byte/binary, scalar, product, and engine-specific compression choices.
• OpenSearch sparse_vector — neural sparse ANN support introduced in OpenSearch 3.3.
• Elastic Stack 9.5.3 release and OpenSearch version history — pinned September/August 2026 baselines.