Embedding Fundamentals, Model Versioning, Dimensions, Similarity / Distance, and Storing Vector Values

1. AtlasMart problem: semantic search needs a data contract before it needs a KNN query

AtlasMart wants “products like this camera” and natural-language discovery without copying the entire catalog into a separate system on day one. The dangerous shortcut is to treat an embedding as magic meaning. An embedding is a fixed-length numeric vector produced by an external model; Firestore stores/searches that vector but does not generate it. A model version identifies the generator and configuration that produced the vector. Dimension is the vector length. A distance measure turns two vectors into a numeric proximity score. A K-nearest-neighbor (KNN) query returns the closest indexed vectors according to that chosen measure. Those definitions form the storage/query contract.

Learning outcomes

2. Deterministic AtlasMart fixture: eight dimensions, six products, one judged query

The mandatory lab intentionally uses eight-dimensional vectors so every value can be inspected. Eight dimensions are not a production recommendation. They are a deterministic teaching fixture. Production models can emit far larger vectors, but Firestore Native vector indexing currently supports at most 2,048 dimensions. The important contract is that every indexed/query vector for one index has the same dimension and semantic model lineage.

{  "name": "atlasmart-firestore-ch17",  "private": true,  "type": "module",  "engines": { "node": ">=22" },  "dependencies": {    "firebase": "12.19.0",    "firebase-admin": "14.4.0"  },  "devDependencies": { "firebase-tools": "15.30.0" },  "scripts": {    "emulators": "firebase emulators:start --only firestore,auth --project demo-atlasmart-firestore",    "seed": "node seed-ch17.mjs",    "evaluate": "node evaluate-ch17.mjs",    "reset": "node reset-ch17.mjs"  }}

[  {"id":"p-1001","tenantId":"tenant-a","category":"cameras","name":"Trail Camera","embeddingV1":[0.93,0.05,0.02,0.00,0.20,0.06,0.01,0.12]},  {"id":"p-1002","tenantId":"tenant-a","category":"accessories","name":"USB-C Hub","embeddingV1":[0.04,0.91,0.08,0.15,0.01,0.18,0.04,0.02]},  {"id":"p-1003","tenantId":"tenant-b","category":"sensors","name":"Temp Sensor","embeddingV1":[0.09,0.14,0.86,0.05,0.05,0.18,0.02,0.03]},  {"id":"p-1004","tenantId":"tenant-b","category":"gateways","name":"Edge Gateway","embeddingV1":[0.10,0.55,0.31,0.51,0.04,0.28,0.08,0.04]},  {"id":"p-1005","tenantId":"tenant-c","category":"cameras","name":"PoE Camera","embeddingV1":[0.86,0.09,0.02,0.06,0.30,0.07,0.01,0.14]},  {"id":"p-1006","tenantId":"tenant-a","category":"power","name":"Bench PSU","embeddingV1":[0.02,0.22,0.11,0.18,0.01,0.89,0.06,0.02]}]

3. Store vector lineage beside the vector

process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { readFile } from "node:fs/promises";import { initializeApp } from "firebase-admin/app";import { getFirestore } from "firebase-admin/firestore";initializeApp({ projectId: "demo-atlasmart-firestore" });const db = getFirestore();const rows = JSON.parse(await readFile("atlasmart-vectors.json", "utf8"));for (const row of rows) {  if (row.embeddingV1.length !== 8) throw new Error(`bad dimension: ${row.id}`);  await db.doc(`vectorProducts/${row.id}`).set({    ...row,    embeddingModel: "atlasmart-local-v1",    embeddingVersion: 1,    embeddingDimension: 8,    embeddingSourceHash: `sha256:${row.id}:fixture-v1`,    searchable: true,    public: true  });}console.log(`seeded ${rows.length} versioned vector products`);

4. Similarity/distance is a measurement, not an explanation

Euclidean distance measures straight-line separation. Cosine distance focuses on angle and is useful when vector magnitude is not intended to carry meaning. Dot product combines direction and magnitude; Firestore Standard documents all three options. For cosine, a zero vector is a special case because normalization is undefined. A smaller Euclidean/cosine distance means nearer; dot-product APIs may expose a score/distance convention that you must read from the SDK documentation before comparing thresholds.

export function assertVector(v, dim = 8) {  if (!Array.isArray(v) || v.length !== dim || v.some(x => !Number.isFinite(x))) {    throw new TypeError(`expected ${dim} finite numbers`);  }}export function euclidean(a, b) {  assertVector(a, a.length); assertVector(b, a.length);  return Math.sqrt(a.reduce((s,x,i) => s + (x-b[i])**2, 0));}export function dot(a,b) { return a.reduce((s,x,i)=>s+x*b[i],0); }export function norm(a) { return Math.sqrt(dot(a,a)); }export function cosineDistance(a,b) {  const d = norm(a) * norm(b);  if (d === 0) throw new Error('cosine undefined for zero vector');  return 1 - dot(a,b)/d;}export function rank(items, query, distance = euclidean) {  return items.map(x => ({...x, distance: distance(x.embeddingV1, query)}))    .sort((a,b) => a.distance-b.distance || a.id.localeCompare(b.id));}

5. Observable ranking before Firestore is involved

import rows from "./atlasmart-vectors.json" with { type: "json" };import { rank } from "./vector-math.mjs";const query = [0.90,0.06,0.02,0.02,0.24,0.05,0.01,0.13];const ranked = rank(rows, query);console.table(ranked.map(x => ({ id:x.id, tenant:x.tenantId, category:x.category, distance:+x.distance.toFixed(6) })));if (ranked[0].id !== "p-1001" || ranked[1].id !== "p-1005") throw new Error("fixture drifted");console.log("ground-truth top-2 preserved");

6. Model migration: never overwrite lineage invisibly

If AtlasMart changes from atlasmart-local-v1 to another embedding model, vectors from the old and new spaces are not automatically comparable. A safe migration writes embeddingV2 plus explicit embeddingModelV2/embeddingVersion metadata, builds a matching index, evaluates the new retrieval contract, shifts reads, and only then retires v1. An in-place overwrite with no version marker makes rollback and offline quality comparison impossible.

7. Managed Firestore vector field: optional cloud step

In managed Standard Native, the server library writes a Firestore vector value and a flat vector index supports KNN. The local mandatory lab does not claim that its array field is a managed vector index; it is a deterministic oracle. When you run the optional managed step, write the same eight values with the server SDK’s vector type and create the matching index.

import { Firestore, FieldValue } from "@google-cloud/firestore";const db = new Firestore({ projectId: process.env.GCLOUD_PROJECT });await db.doc("vectorProducts/p-1001").set({  tenantId: "tenant-a",  category: "cameras",  embeddingModel: "atlasmart-local-v1",  embeddingVersion: 1,  embeddingDimension: 8,  embeddingV1: FieldValue.vector([0.93,0.05,0.02,0.00,0.20,0.06,0.01,0.12])}, { merge: true });

8. Authorization metadata belongs in the retrieval contract

Vector proximity cannot decide whether a caller may see a product. AtlasMart carries tenantId, visibility/category state, and model version as ordinary metadata. A trusted backend that bypasses Security Rules must validate the end-user/tenant and inject the corresponding metadata filter; a broad service-account IAM grant is not tenant authorization. If content is sent to an external embedding service, classify and minimize that content first. Product descriptions may be safe while private support tickets, PII, regulated data, or customer secrets may require a different architecture.

9. Edition/mode/client matrix

Verification checklist and cleanup

• Every fixture vector has exactly eight finite numbers and explicit model/version/dimension metadata.
• The deterministic query returns p-1001 then p-1005 under Euclidean distance.
• No claim is made that the emulator proves a production vector index or production billing.
• Changing model/version requires a separate evaluation/cutover path.
• Tenant/visibility metadata is treated as authorization input, never as semantic relevance alone.

process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { initializeApp } from "firebase-admin/app";import { getFirestore } from "firebase-admin/firestore";initializeApp({ projectId: "demo-atlasmart-firestore" });await getFirestore().recursiveDelete(getFirestore().collection("vectorProducts"));console.log("Chapter 17 vector fixture removed");

Production judgment and bridge to Lesson 2

Choose an embedding model and distance measure only after defining the retrieval task, privacy boundary, judged queries, model lineage and authorization metadata. Lesson 2 turns that data contract into actual Firestore vector indexes and examines where index configuration, supported SDKs and query limits change the implementation.

Knowledge check

1. Who generates an AtlasMart embedding in this architecture?
2. Why record model version beside a vector?
3. What is the current maximum Firestore embedding dimension for vector indexing?
4. Is a smaller vector distance proof that an item is semantically correct?
5. Can a tenant filter be omitted because the nearest vector belongs to another tenant?

Authoritative references

• Firebase · Search with vector embeddings
• Google Cloud · Firestore Standard pricing
• Firebase · Connect to the Firestore emulator and understand differences from production
• Google Cloud · Enterprise Native Pipeline findNearest stage
• Firebase · Enterprise Native supported data types
• Google Cloud · Vertex AI embedding sample