Build and Evaluate a Semantic Retrieval Feature with Recall / Quality, Latency, Security, and Cost Measurements

1. AtlasMart problem: “the demos look good” is not a retrieval acceptance test

Before shipping semantic search, AtlasMart needs evidence across quality, latency, authorization, model freshness and cost. A search feature can return plausible demos while failing rare categories, crossing tenant boundaries, scanning too much of the vector index, or degrading after a model/backfill change.

Learning outcomes

2. Evaluation dataset: write the expected neighbors down

[  {"id":"q-camera","query":[0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13],"relevant":["p-1001","p-1005"],"tenantId":null},  {"id":"q-camera-tenant-a","query":[0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13],"relevant":["p-1001"],"tenantId":"tenant-a"},  {"id":"q-power","query":[0.02,0.20,0.09,0.16,0.01,0.94,0.05,0.01],"relevant":["p-1006"],"tenantId":"tenant-a"}]

3. Quality harness: recall@k plus “no unauthorized result”

import { performance } from "node:perf_hooks";import rows from "./atlasmart-vectors.json" with { type:"json" };import judged from "./judged-queries.json" with { type:"json" };import { rank } from "./vector-math.mjs";function recallAtK(ids, truth, k) {  const t = new Set(truth);  return ids.slice(0,k).filter(id => t.has(id)).length / Math.max(1,t.size);}const latencies=[]; const reports=[];for (let warm=0; warm<25; warm++) for (const q of judged) rank(rows,q.query);for (let round=0; round<200; round++) {  for (const q of judged) {    const pool = q.tenantId ? rows.filter(x => x.tenantId === q.tenantId) : rows;    const t0=performance.now(); const ranked=rank(pool,q.query); const ms=performance.now()-t0;    latencies.push(ms);    const ids=ranked.map(x=>x.id);    if (q.tenantId && ranked.some(x=>x.tenantId!==q.tenantId)) throw new Error("tenant leak");    reports.push({id:q.id, recall2:recallAtK(ids,q.relevant,2)});  }}latencies.sort((a,b)=>a-b);const p = x => latencies[Math.min(latencies.length-1, Math.floor((x/100)*latencies.length))];console.log({p50:p(50),p95:p(95),p99:p(99), sampleCount:latencies.length});console.log(reports.slice(0,judged.length));console.log("LOCAL CPU timing only: not Firestore service latency");

4. Latency: report what you actually measured

The harness measures local exact-distance CPU time over six vectors. It teaches percentile calculation and reproducibility, not Firestore performance. For managed production evidence, record region, edition/mode, SDK/runtime, vector dimension, K, filters, index configuration, dataset cardinality, warmup, concurrency, Query Explain/index entries scanned where available, and billing dimensions. Never substitute emulator/local numbers for p95/p99 service latency.

5. Standard vector-query billing model

Current Standard pricing charges document reads for returned documents plus one read for each batch of up to 100 KNN vector-index entries read. The service’s actual scanned-entry count depends on the managed query/index; the local emulator cannot produce it. The following calculator turns measured managed evidence into billable read units without inventing the scan count.

export function standardVectorReadCharges({ returnedDocuments, vectorIndexEntriesRead }) {  if (returnedDocuments < 0 || vectorIndexEntriesRead < 0) throw new Error("non-negative inputs required");  return returnedDocuments + Math.ceil(vectorIndexEntriesRead / 100);}for (const sample of [  { returnedDocuments:5, vectorIndexEntriesRead:0 },  { returnedDocuments:5, vectorIndexEntriesRead:100 },  { returnedDocuments:5, vectorIndexEntriesRead:1550 }]) console.log({ ...sample, billedReads: standardVectorReadCharges(sample) });// Pricing model only. Supply real managed scan evidence before using for forecasting.

6. Filter selectivity and cost/security evidence

Track candidate population before/after metadata filters. A tenant filter can reduce candidates and is required for isolation, but it is not enough to say “filters make vector search cheaper.” Measure the actual managed vector-index entries scanned. Likewise, a server service account may have database-wide IAM; application authorization still must prevent a user from asking the backend to substitute another tenant ID.

export function authorizeSearch({ caller, requestedTenantId }) {  if (!caller?.uid) throw Object.assign(new Error("unauthenticated"), { status:401 });  if (caller.tenantId !== requestedTenantId) throw Object.assign(new Error("forbidden tenant"), { status:403 });  return { tenantId: caller.tenantId };}// Build the Firestore filter from the returned trusted object, not raw request JSON.

7. Model/version drift gate

Quality reports must be segmented by embeddingModel/embeddingVersion. If a v2 backfill is incomplete, either restrict v2 queries to v2-complete documents with a deliberate coverage tradeoff or keep v1 primary. Mixing vector versions into one index without proof they share the same space invalidates evaluation.

8. Failure injection matrix

9. When Firestore remains a good fit—and when to separate search

Firestore is attractive when vector retrieval is close to the transactional documents, query shapes fit supported KNN/filter capabilities, managed index/billing are acceptable, and one operational datastore simplifies consistency. A dedicated search/vector system can be warranted when retrieval needs exceed current Firestore features—for example advanced hybrid ranking, very specialized ANN controls, search-specific observability/relevance tooling, or independent scaling/cost boundaries. That decision should come from measured requirements, not fashion.

10. Edition/mode/client matrix

11. Final Chapter 17 verification checklist

• Five lesson labs use the same deterministic six-product/eight-dimensional fixture.
• Every vector carries model/version/dimension lineage.
• Local quality/latency are explicitly labeled local evidence.
• Managed Standard cost uses returned documents plus measured vector-index entries scanned.
• Tenant authorization is tested independently from relevance.
• Vector generation is external to Firestore and external providers are optional/governed.
• Standard Core, Enterprise Native Pipeline and MongoDB compatibility are not conflated.

Production judgment and bridge to Chapter 18

AtlasMart now has a complete vector-retrieval engineering loop: versioned embeddings, managed index contract, distance/threshold/filter semantics, replayable generation, judged quality, authorization tests, latency methodology and cost evidence. Chapter 18 moves to Firestore Enterprise Native mode and compares familiar Core operations with the separate Pipeline execution model—including where findNearest participates in that broader query engine.

Knowledge check

1. What two inputs are required for a credible Standard vector cost estimate?
2. Can local p99 from the six-product oracle be called Firestore p99?
3. What is the strongest security acceptance criterion for tenant vector search?
4. Why segment evaluation by embedding version?
5. When should AtlasMart consider a dedicated search/vector system?

Authoritative references

• Firebase · Search with vector embeddings
• Google Cloud · Firestore Standard pricing
• Firebase · Connect to the Firestore emulator and understand differences from production
• Google Cloud · Enterprise Native Pipeline findNearest stage
• Firebase · Enterprise Native supported data types
• Google Cloud · Vertex AI embedding sample