Evaluate Search Quality with Queries/Judgments Instead of Tuning Only by Anecdotal Examples

Search teams often tune on the last embarrassing query: add a synonym for “camra,” boost a field for one product, change the analyzer, celebrate the demo, then unknowingly damage dozens of other intents. AtlasMart needs a search acceptance suite: versioned queries, relevance judgments, reproducible retrieval settings, ranking metrics, failure slices and explicit freshness/latency SLOs. The goal is not one perfect number; it is evidence that a change improves the intended population without unacceptable regressions elsewhere.

Learning outcomes

Lab contract and exact assumptions

Direct-entry setup

Run the fixture and wait for indexes to be ONLINE. If you test eventual mode, force a refresh barrier before comparing relevance variants so freshness is not a confounder unless freshness is the thing being evaluated.

CYPHER 25
// Safe to rerun in the disposable Chapter 21 lab.
CREATE CONSTRAINT ch21_product_id IF NOT EXISTS
FOR (p:Product) REQUIRE p.productId IS UNIQUE;
CREATE CONSTRAINT ch21_category_id IF NOT EXISTS
FOR (c:Category) REQUIRE c.categoryId IS UNIQUE;
CREATE CONSTRAINT ch21_store_id IF NOT EXISTS
FOR (s:Store) REQUIRE s.storeId IS UNIQUE;
CREATE CONSTRAINT ch21_customer_id IF NOT EXISTS
FOR (c:Customer) REQUIRE c.customerId IS UNIQUE;

MERGE (cam:Category {categoryId:'CAT-21-CAM'}) SET cam.name='Cameras', cam.labTag='ch21'
MERGE (out:Category {categoryId:'CAT-21-OUT'}) SET out.name='Outdoor', out.labTag='ch21'
MERGE (sec:Category {categoryId:'CAT-21-SEC'}) SET sec.name='Security', sec.labTag='ch21'
MERGE (acc:Category {categoryId:'CAT-21-ACC'}) SET acc.name='Accessories', acc.labTag='ch21'
MERGE (st:Store {storeId:'ST-21-CENTRAL'}) SET st.name='AtlasMart Central', st.labTag='ch21'
MERGE (c1:Customer {customerId:'C-2101'}) SET c1.name='Mina Rahimi', c1.labTag='ch21'
MERGE (c2:Customer {customerId:'C-2102'}) SET c2.name='Omid Karimi', c2.labTag='ch21';

UNWIND [
 {id:'P-2101',name:'Trail Camera Pro',description:'Weatherproof wildlife trail camera with infrared night vision and long battery life',tags:['wildlife','trail','infrared','outdoor'],active:true,category:'CAT-21-CAM',qty:5,featured:true},
 {id:'P-2102',name:'Trail Camera Mini',description:'Compact wildlife camera for trails, gardens, and backyard monitoring',tags:['wildlife','trail','compact'],active:true,category:'CAT-21-CAM',qty:0,featured:false},
 {id:'P-2103',name:'Indoor Security Camera',description:'Wi-Fi home security camera with motion alerts and night vision',tags:['security','indoor','night vision'],active:true,category:'CAT-21-SEC',qty:7,featured:false},
 {id:'P-2104',name:'Trail Running Hydration Vest',description:'Lightweight hydration vest for long trail runs and mountain races',tags:['running','trail','hydration'],active:true,category:'CAT-21-OUT',qty:11,featured:false},
 {id:'P-2105',name:'Action Camera 4K',description:'Water-resistant action sports camera for cycling, hiking, and travel',tags:['action','sports','camera'],active:true,category:'CAT-21-CAM',qty:3,featured:true},
 {id:'P-2106',name:'Wildlife Field Guide',description:'Illustrated guide to birds and mammals for outdoor observation',tags:['wildlife','book','outdoor'],active:true,category:'CAT-21-OUT',qty:6,featured:false},
 {id:'P-2107',name:'Solar Trail Charger',description:'Solar charger for outdoor cameras, sensors, and trail equipment',tags:['solar','trail','charger'],active:true,category:'CAT-21-ACC',qty:0,featured:false},
 {id:'P-2108',name:'Refurbished Trail Camera',description:'Older trail camera unit retained for support reference only',tags:['trail','camera','refurbished'],active:false,category:'CAT-21-CAM',qty:2,featured:false}
] AS row
MERGE (p:Product {productId:row.id})
SET p.name=row.name, p.description=row.description, p.tags=row.tags,
    p.active=row.active, p.featured=row.featured, p.labTag='ch21'
WITH row,p
MATCH (cat:Category {categoryId:row.category}), (st:Store {storeId:'ST-21-CENTRAL'})
MERGE (p)-[:IN_CATEGORY]->(cat)
MERGE (p)-[stock:STOCKED_AT]->(st)
SET stock.quantity=row.qty, stock.labTag='ch21';

MATCH (c1:Customer {customerId:'C-2101'}), (p1:Product {productId:'P-2101'})
MERGE (c1)-[r1:REVIEWED]->(p1)
SET r1.reviewId='REV-2101', r1.rating=5,
    r1.message='Excellent night vision for wildlife at the cabin', r1.labTag='ch21';
MATCH (c2:Customer {customerId:'C-2102'}), (p4:Product {productId:'P-2104'})
MERGE (c2)-[r2:REVIEWED]->(p4)
SET r2.reviewId='REV-2102', r2.rating=4,
    r2.message='Comfortable on long trail runs', r2.labTag='ch21';

CREATE FULLTEXT INDEX ch21_catalog_ft IF NOT EXISTS
FOR (p:Product) ON EACH [p.name, p.description, p.tags]
OPTIONS {indexConfig: {
  `fulltext.analyzer`: 'english',
  `fulltext.eventually_consistent`: false
}};

CREATE FULLTEXT INDEX ch21_reviews_ft IF NOT EXISTS
FOR ()-[r:REVIEWED]-() ON EACH [r.message]
OPTIONS {indexConfig: {`fulltext.analyzer`: 'english'}};

CALL db.awaitIndexes(300);

CYPHER 25
SHOW FULLTEXT INDEXES
YIELD name, state, populationPercent, entityType, labelsOrTypes,
      properties, indexProvider, options, lastRead, readCount
WHERE name STARTS WITH 'ch21_'
RETURN * ORDER BY name;

MATCH (p:Product {labTag:'ch21'})
RETURN count(p) AS products,
       count { (p)-[:STOCKED_AT]->(:Store {storeId:'ST-21-CENTRAL'}) } AS stockEdges;
// Deterministic graph invariant: products = 8 and stockEdges = 8.
// Full-text state invariant before search: both indexes are ONLINE and 100% populated.

1. Judgments describe user intent, not current ranking

Assign relevance labels before looking at the proposed ranking change whenever possible. Grade 3 can mean “ideal,” 2 “useful,” 1 “weakly relevant,” and 0/unlisted “not relevant.” Keep judgments versioned with intent notes. The fixture deliberately includes ambiguity: P-2106 contains “wildlife” but is a field guide, while P-2108 is a strong lexical trail-camera match that is inactive.

2. Capture ranked IDs; never make Lucene score the label

Run each query with identical index/analyzer/sourceK settings and store product IDs in returned order. Keep actual scores as diagnostics, but the evaluator consumes ranks and human/business judgments. That prevents the system from “proving itself correct” using its own score.

// Capture ranked IDs for the four judged queries. Run each query separately and
// put the returned productId order into results.json; do not copy score values into the judgments.
CYPHER 25
CALL db.index.fulltext.queryNodes('ch21_catalog_ft', $q, {limit: 10})
YIELD node, score
RETURN node.productId AS productId, node.name AS name, score
ORDER BY score DESC;

{
  "q1": ["P-2101", "P-2106", "P-2102"],
  "q2": ["P-2101", "P-2102", "P-2108"],
  "q3": ["P-2103", "P-2101"],
  "q4": ["P-2105", "P-2101", "P-2102"]
}

3. Evaluate with multiple metrics because they answer different questions

Precision@k asks how many shown items are relevant. Recall@k asks how much known relevant inventory was recovered. MRR emphasizes the first relevant hit. nDCG@k uses graded relevance and position. A retail autosuggest may prioritize early precision/MRR; a support discovery workflow may care more about recall.

#!/usr/bin/env python3
"""Evaluate AtlasMart Chapter 21 ranked product IDs captured from Neo4j.
Usage: python evaluate_search.py results.json
results.json example: {"q1":["P-2101","P-2102","P-2106"], ...}
"""
import json, math, sys

JUDGMENTS = {
  "q1": {"query":"wildlife camera", "grades":{"P-2101":3,"P-2102":2,"P-2106":1}},
  "q2": {"query":"trail camera", "grades":{"P-2101":3,"P-2102":3,"P-2108":1}},
  "q3": {"query":"night vision camera", "grades":{"P-2101":3,"P-2103":2}},
  "q4": {"query":"camera outdoor", "grades":{"P-2101":3,"P-2105":2,"P-2102":1}},
}
K = 3

def dcg(ids, grades, k):
    total = 0.0
    for rank, pid in enumerate(ids[:k], start=1):
        gain = grades.get(pid, 0)
        total += (2**gain - 1) / math.log2(rank + 1)
    return total

def metrics(ids, grades, k):
    rel = {pid for pid,g in grades.items() if g > 0}
    top = ids[:k]
    hits = [pid for pid in top if pid in rel]
    precision = len(hits) / k
    recall = len(hits) / len(rel) if rel else 0.0
    rr = 0.0
    for rank,pid in enumerate(ids, start=1):
        if pid in rel:
            rr = 1.0 / rank; break
    ideal = [pid for pid,_ in sorted(grades.items(), key=lambda x:x[1], reverse=True)]
    idcg = dcg(ideal, grades, k)
    ndcg = dcg(ids, grades, k) / idcg if idcg else 0.0
    return precision, recall, rr, ndcg

results = json.load(open(sys.argv[1], encoding='utf-8'))
rows=[]
for qid,spec in JUDGMENTS.items():
    ids = results.get(qid, [])
    p,r,rr,n = metrics(ids, spec['grades'], K)
    rows.append((qid,p,r,rr,n))
    print(f"{qid} P@{K}={p:.3f} R@{K}={r:.3f} RR={rr:.3f} nDCG@{K}={n:.3f}")
if rows:
    print('macro', ' '.join([
      f"P@{K}={sum(x[1] for x in rows)/len(rows):.3f}",
      f"R@{K}={sum(x[2] for x in rows)/len(rows):.3f}",
      f"MRR={sum(x[3] for x in rows)/len(rows):.3f}",
      f"nDCG@{K}={sum(x[4] for x in rows)/len(rows):.3f}",
    ]))

python evaluate_search.py results.json
# The script prints measured metrics from your captured ranks.
# Do not paste fixed expected values into the course: ranking can change with corpus/analyzer/release.

4. Compare variants with controlled experiments

Do not compare a cold rebuilt index, different corpus, different candidate count and new ranking rule in one experiment. If four things change, a better metric cannot tell you which mechanism helped.

5. Build failure slices, not only macro averages

6. Quality is only one release gate

A ranking variant that raises nDCG but doubles p99 latency or violates freshness is not automatically better. Search release evidence should join relevance, latency, resource, correctness and operations.

7. Wrong approach → failure → repair

8. Production runbook and bridge to Chapter 22

1. Freeze fixture/corpus snapshot, server/Cypher/index/analyzer configuration and judged-set version.
2. Wait for indexes ONLINE; if testing quality rather than freshness, synchronize eventual indexes.
3. Warm the workload consistently and execute the complete query suite.
4. Capture candidate IDs/ranks/scores, filter reasons, p50/p95/p99, zero-result rate and resource evidence.
5. Compute quality metrics and slices.
6. Review security/freshness/capacity regressions.
7. Approve or reject the single change; retain rollback commands/index definition.
8. When adding vector retrieval in Chapter 22, keep the lexical baseline unchanged and fuse ranked lists rather than raw score magnitudes.

9. Cleanup / reset

Run only against the disposable Chapter 21 lab. This removes the chapter indexes, synthetic graph entities and chapter constraints.

CYPHER 25
DROP INDEX ch21_catalog_ft IF EXISTS;
DROP INDEX ch21_reviews_ft IF EXISTS;
DROP INDEX ch21_freshness_ft IF EXISTS;
MATCH (n) WHERE n.labTag='ch21' DETACH DELETE n;
DROP CONSTRAINT ch21_product_id IF EXISTS;
DROP CONSTRAINT ch21_category_id IF EXISTS;
DROP CONSTRAINT ch21_store_id IF EXISTS;
DROP CONSTRAINT ch21_customer_id IF EXISTS;

10. Final production judgment

Summary and next step

Chapter 21 now treats lexical search as an engineered retrieval system: schema + analyzer + query contract + freshness + bounded candidates + graph/business context + judged evaluation. Chapter 22 can add embeddings/vector search without destroying that discipline: semantic/vector scores remain source-specific, and hybrid retrieval will fuse independently ranked lexical and vector lists before graph-aware context assembly.

Authoritative references

• Current Neo4j versions — Current database release 2026.07.1 and current 5.26 LTS patch 5.26.30.
• Full-text indexes — Cypher 25 — Current schema, analyzer, query, score, eventual-consistency, SHOW FULLTEXT INDEXES and procedure semantics.
• Semantic indexes — Why full-text and vector indexes are explicit semantic retrieval systems and why raw cross-source scores should not be compared.
• Built-in full-text procedures — Current signatures for queryNodes/queryRelationships/listAvailableAnalyzers/awaitEventuallyConsistentIndexRefresh and limit/skip/query-analyzer options.
• Index configuration — Default analyzer, eventual-consistency queue model, background update settings and operational implications.
• Configuration settings — Current db.index.fulltext.* defaults, including standard-no-stop-words and eventual-consistency settings.
• Index syntax — Current CREATE/SHOW/DROP and semantic-index query syntax; SHOW FULLTEXT INDEXES is the supported filtered SHOW form.
• Security limitations for semantic indexes — Lucene-backed full-text/vector security filtering can conservatively return partial or zero results under fine-grained restrictions.
• Hybrid search developer guide — Current rank-fusion guidance for lexical, vector and structural sources; fuse ranks, not incomparable raw scores.
• Hybrid search engineering article — 2026 worked explanation of combining words, meaning and graph topology with rank-based fusion.
• Metrics — Enterprise-only built-in metrics surface and monitoring responsibilities.
• Metrics reference — Current full-text queried/populated counters; edition boundary is Enterprise.
• System requirements — Neo4j 2026.07 supported Java 21/25 and current OS/runtime boundaries.
• Python driver 6.3 — Official driver API used by the optional evaluation harness; current 6.3 supports Neo4j 2026.x.
• Vector SEARCH clause — Bridge to Chapter 22 and explicit current warning to rank vector/full-text sources independently.