K-Nearest-Neighbor Queries, Distance Thresholds, Metadata Filters, and Result Ranking

1. AtlasMart problem: the nearest result changes when the distance and candidate set change

A KNN query is not “semantic search” in the abstract. It is: choose a candidate set, choose a vector field/model version, choose a distance measure, optionally apply a threshold, rank, and return K. AtlasMart must know which of those choices caused a result—not merely display a score.

Learning outcomes

2. One fixture, three distance functions

import rows from "./atlasmart-vectors.json" with { type: "json" };import { rank, euclidean, cosineDistance, dot } from "./vector-math.mjs";const q = [0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13];for (const [name, fn] of [["euclidean",euclidean],["cosine",cosineDistance],["negative-dot",(a,b)=>-dot(a,b)]]) {  const top = rank(rows, q, fn).slice(0,4).map(x=>[x.id,+x.distance.toFixed(6)]);  console.log(name, top);}

3. Distance measure selection is part of model evaluation

Euclidean considers absolute coordinate distance. Cosine normalizes magnitude and compares direction. Dot product can reward magnitude as well as alignment. If the embedding model documentation assumes normalized vectors, dot product and cosine may behave similarly; if not, they can diverge. Do not select a measure because one cherry-picked query “looks better.” Evaluate a judged set representative of production traffic.

4. Thresholds: K is a ceiling, not a promise of relevance

If the nearest item is still far away, forcing five results can manufacture bad recommendations. Current Standard vector search supports a distance threshold. AtlasMart’s local oracle uses the same idea: first rank, then discard candidates beyond a task-specific threshold. Thresholds are model- and measure-specific; they must be calibrated against labeled data and revisited after model changes.

import rows from "./atlasmart-vectors.json" with { type: "json" };import { rank } from "./vector-math.mjs";const query = [0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13];const maxDistance = 0.25; // fixture decision, NOT a universal production valueconst accepted = rank(rows, query).filter(x => x.distance <= maxDistance).slice(0,5);console.table(accepted.map(x => ({ id:x.id, distance:+x.distance.toFixed(6) })));if (accepted.some(x => x.distance > maxDistance)) throw new Error("threshold bug");

5. Metadata prefilters shrink the candidate set before KNN

For tenant-a camera retrieval, candidates from tenants B/C must not be considered merely because they are close. In a trusted server path the authenticated AtlasMart tenant is converted into a mandatory equality filter, not accepted from an arbitrary request body. Category may be a product filter; tenant is an authorization boundary.

const candidates = rows.filter(x => x.tenantId === "tenant-a" && x.category === "cameras");const ranked = rank(candidates, query).slice(0, 5);console.log({ total: rows.length, candidates: candidates.length, selectivity: candidates.length / rows.length });if (ranked.some(x => x.tenantId !== "tenant-a")) throw new Error("cross-tenant leak");

6. Managed prefilter query

import { Firestore, FieldValue } from "@google-cloud/firestore";const db = new Firestore({ projectId: process.env.GCLOUD_PROJECT });const q = [0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13];const snap = await db.collection("vectorProducts")  .where("tenantId", "==", "tenant-a")  .where("category", "==", "cameras")  .findNearest("embeddingV1", FieldValue.vector(q), {    limit: 5,    distanceMeasure: "EUCLIDEAN",    distanceResultField: "vectorDistance"  })  .get();for (const doc of snap.docs) {  console.log(doc.id, doc.get("vectorDistance"));}

The query requires a matching composite vector index. In Standard pricing, candidate/index work matters: vector search bills returned document reads plus one read for each batch of up to 100 vector index entries read. A selective prefilter can therefore affect both isolation and cost, but do not claim a specific scan reduction until Query Explain/managed evidence measures it.

7. Ground truth and recall@k

AtlasMart’s judged query “outdoor camera similar to Trail Camera” labels p-1001 and p-1005 as relevant. With that tiny ground truth, recall@2 is the fraction of those two relevant IDs returned in the top two. A six-document fixture is not statistically meaningful; its purpose is to make the metric executable before scaling to a real evaluation set.

export function recallAtK(rankedIds, relevantIds, k) {  const truth = new Set(relevantIds);  const hits = rankedIds.slice(0,k).filter(id => truth.has(id)).length;  return truth.size === 0 ? 1 : hits / truth.size;}const r = recallAtK(["p-1001","p-1005","p-1002"], ["p-1001","p-1005"], 2);console.log({ recallAt2: r });if (r !== 1) throw new Error("expected fixture recall@2 = 1");

8. Wrong approach: distance as a business truth score

A vector distance is not “92% relevant,” not an authorization score, and not a causal explanation. Repair the UI by labeling results as retrieval candidates, keeping model/measure/version in telemetry, evaluating user/task outcomes, and providing fallbacks when no candidate passes the quality threshold.

9. Ties, stability, and ranking contracts

Do not rely on undocumented tie ordering. If two items have effectively equal distance and business UX requires deterministic ordering, apply a documented second-stage ranker in application code using stable business fields after KNN, while keeping the vector distance available for debugging. This post-ranking must not reintroduce unauthorized documents.

10. Edition/mode/client matrix

Verification checklist

• The same fixture is evaluated under at least two distance measures.
• Threshold is explicitly labeled fixture-specific.
• Tenant filter is derived from trusted identity/context.
• Recall@k uses a written judged set, not subjective inspection.
• Any managed cost statement separates returned documents from vector-index entries scanned.

Production judgment and bridge to Lesson 4

Retrieval quality is now testable, but the vectors still need to be generated, refreshed and governed. Lesson 4 designs that embedding lifecycle and demonstrates why generation belongs outside retryable Firestore transactions.

Knowledge check

1. What does K control?
2. Why can a distance threshold be useful?
3. Is tenantId just a relevance filter?
4. What does recall@k require?
5. Why avoid undocumented tie ordering?

Authoritative references

• Firebase · Search with vector embeddings
• Google Cloud · Firestore Standard pricing
• Firebase · Connect to the Firestore emulator and understand differences from production
• Google Cloud · Enterprise Native Pipeline findNearest stage
• Firebase · Enterprise Native supported data types
• Google Cloud · Vertex AI embedding sample