Firestore Does Not Generate Embeddings: Vertex AI / External Generation Pipelines, Backfills, and Updates

1. AtlasMart problem: Firestore can search a vector only after another system creates it

Product descriptions change, model versions change, and some content must never leave AtlasMart’s trust boundary. Firestore is the vector storage/retrieval layer here; embedding generation is a separate pipeline with its own authentication, quotas, privacy, retries, cost and model lifecycle.

Learning outcomes

2. Mandatory local generator: deterministic and free

The course does not require an external model. The mandatory path uses the checked-in vectors from atlasmart-vectors.json. That guarantees every learner sees the same ranking and allows model-lifecycle mechanics to be tested without network, billing, model drift or rate limits.

{  "name": "atlasmart-firestore-ch17",  "private": true,  "type": "module",  "engines": { "node": ">=22" },  "dependencies": {    "firebase": "12.19.0",    "firebase-admin": "14.4.0"  },  "devDependencies": { "firebase-tools": "15.30.0" },  "scripts": {    "emulators": "firebase emulators:start --only firestore,auth --project demo-atlasmart-firestore",    "seed": "node seed-ch17.mjs",    "evaluate": "node evaluate-ch17.mjs",    "reset": "node reset-ch17.mjs"  }}

[  {"id":"p-1001","tenantId":"tenant-a","category":"cameras","name":"Trail Camera","embeddingV1":[0.93,0.05,0.02,0.00,0.20,0.06,0.01,0.12]},  {"id":"p-1002","tenantId":"tenant-a","category":"accessories","name":"USB-C Hub","embeddingV1":[0.04,0.91,0.08,0.15,0.01,0.18,0.04,0.02]},  {"id":"p-1003","tenantId":"tenant-b","category":"sensors","name":"Temp Sensor","embeddingV1":[0.09,0.14,0.86,0.05,0.05,0.18,0.02,0.03]},  {"id":"p-1004","tenantId":"tenant-b","category":"gateways","name":"Edge Gateway","embeddingV1":[0.10,0.55,0.31,0.51,0.04,0.28,0.08,0.04]},  {"id":"p-1005","tenantId":"tenant-c","category":"cameras","name":"PoE Camera","embeddingV1":[0.86,0.09,0.02,0.06,0.30,0.07,0.01,0.14]},  {"id":"p-1006","tenantId":"tenant-a","category":"power","name":"Bench PSU","embeddingV1":[0.02,0.22,0.11,0.18,0.01,0.89,0.06,0.02]}]

3. Pipeline state is data, not log text

{  "productId": "p-1001",  "targetModel": "atlasmart-local-v2",  "targetVersion": 2,  "targetDimension": 8,  "sourceHash": "sha256:...",  "status": "queued",  "attempt": 0,  "nextAttemptAt": null,  "lastError": null,  "generatedAt": null,  "indexedAt": null,  "correlationId": "embed-p-1001-v2"}

4. Idempotent backfill algorithm

for (const product of products) {  const sourceHash = sha256(canonicalText(product));  const ref = db.doc(`embeddingJobs/${product.id}-v2`);  const existing = await ref.get();  if (existing.exists && existing.get("sourceHash") === sourceHash && existing.get("status") === "complete") continue;  await ref.set({ productId: product.id, targetVersion: 2, sourceHash, status: "running" }, { merge: true });  try {    const vector = deterministicLocalEmbeddingV2(product); // mandatory no-cost path    if (vector.length !== 8) throw new Error("dimension mismatch");    await db.doc(`vectorProducts/${product.id}`).set({      embeddingV2: vector,      embeddingModelV2: "atlasmart-local-v2",      embeddingVersionV2: 2,      embeddingDimensionV2: 8,      embeddingSourceHashV2: sourceHash    }, { merge: true });    await ref.set({ status: "complete", completedAt: new Date().toISOString() }, { merge: true });  } catch (error) {    await ref.set({ status: "retryable_error", lastError: String(error) }, { merge: true });    throw error;  }}

5. Why generation must not run inside a retryable Firestore transaction

A transaction callback can rerun after contention. Calling an external embedding API inside that callback can create duplicate charges, repeated sensitive-data transfer and long transaction duration. Repair the design by making the transaction commit only durable intent/state; a worker then performs the external call with an idempotency key/source hash and writes the result separately.

6. Optional Vertex AI path: explicitly separate and governed

If AtlasMart chooses Vertex AI, current Google Cloud samples show gemini-embedding-001 as a stable embedding model. That choice is not part of the mandatory lab and can change over time. Record the exact model ID, task type, output dimension, location, library/API version and source hash. If the model emits more than Firestore’s 2,048 supported dimensions, request a compatible output dimension where the model supports it or apply a governed dimensionality-reduction step; never truncate blindly.

{  "provider": "vertex-ai",  "model": "gemini-embedding-001",  "taskType": "RETRIEVAL_DOCUMENT",  "requestedOutputDimension": 768,  "firestoreVectorField": "embeddingVertex768V1",  "firestoreIndexDimension": 768,  "contentPolicy": "public product catalog text only",  "piiAllowed": false,  "secretsAllowed": false}

7. Privacy and data governance gate

8. Cutover state machine

A robust migration has observable phases: v2_index_building → v2_backfilling → v2_shadow_read → v2_quality_approved → v2_primary → v1_retiring. Shadow reads compare v1/v2 on judged traffic without changing user-visible results. Rollback means switching the read contract back to v1 while v1 index/data still exist.

9. Wrong approach: “backfill until the script exits 0”

A one-shot script with no per-document state cannot prove completeness after throttling, partial failure or source updates. Repair it with idempotent job records, source hashes, pagination/checkpoints, retry classes, completion counts, and a reconciliation query that identifies products missing the target model/version.

const missing = [];for (const doc of (await db.collection("vectorProducts").get()).docs) {  const d = doc.data();  if (d.searchable && (d.embeddingVersionV2 !== 2 || !Array.isArray(d.embeddingV2) || d.embeddingV2.length !== 8)) {    missing.push(doc.id);  }}console.log({ missingCount: missing.length, missing });if (missing.length) process.exitCode = 1;

10. Edition/mode/client matrix

Verification checklist

• Generation is outside retryable transaction callbacks.
• Every target vector is tied to a source hash and model/version/dimension.
• Backfill retry after injected failure is idempotent.
• External generation is optional and has an explicit content-governance gate.
• Cutover and rollback states are documented before retiring v1.

Production judgment and bridge to Lesson 5

A reliable vector pipeline can still deliver a bad search product. Lesson 5 closes the loop with judged relevance, recall, latency distribution, authorization tests, cost accounting, drift detection and a decision on whether Firestore remains the right retrieval engine.

Knowledge check

1. Does Firestore generate embeddings?
2. Why is an external embedding call unsafe inside a retryable transaction callback?
3. What makes a backfill replayable?
4. What should happen before sending customer text to an embedding provider?
5. Why keep v1 during v2 shadow evaluation?

Authoritative references

• Firebase · Search with vector embeddings
• Google Cloud · Firestore Standard pricing
• Firebase · Connect to the Firestore emulator and understand differences from production
• Google Cloud · Enterprise Native Pipeline findNearest stage
• Firebase · Enterprise Native supported data types
• Google Cloud · Vertex AI embedding sample