Index Fan-Out, Large Arrays / Maps, High-Cardinality Fields, and Write Latency / Cost

1. Why a scattered key can still be an expensive write

AtlasMart’s product telemetry uses random IDs and shows no obvious document-key hotspot, yet write tail latency grows after a schema change. The new document contains a 1,200-element tags array, an 800-element audiences array, a deeply nested debug map, and several automatic/composite indexes. Every document write can produce many index-entry mutations. This is index fan-out: one logical write expands into many physical index updates.

Learning outcomes

2. Fan-out is the number of index structures one write must maintain

Firestore documents have a hard maximum of 40,000 index entries per document and 8 MiB total index-entry size. These are failure boundaries. Good designs leave operating margin; they do not deliberately approach them.

3. Use query contracts to justify every index

{  "indexes": [    {      "collectionGroup": "events",      "queryScope": "COLLECTION",      "fields": [        { "fieldPath": "tenantId", "order": "ASCENDING" },        { "fieldPath": "kind", "order": "ASCENDING" },        { "fieldPath": "occurredAt", "order": "DESCENDING" }      ]    }  ],  "fieldOverrides": [    {      "collectionGroup": "events",      "fieldPath": "ingestedAt",      "indexes": []    },    {      "collectionGroup": "events",      "fieldPath": "rawPayload",      "indexes": []    },    {      "collectionGroup": "events",      "fieldPath": "debugAttributes",      "indexes": []    }  ]}

In Chapter 6 AtlasMart retained an index manifest. Chapter 15 adds a rule: every composite index must point to at least one named query contract and every exemption must point to a statement that the field is not queried—or to the replacement access path. Index configuration is code, so review it with the same dependency discipline as an API change.

4. Design-time estimator: catch suspicious arrays/maps before deployment

function estimateArrayEntries(document, indexedArrayFields) {  return indexedArrayFields.reduce((sum, field) => {    const value = document[field];    return sum + (Array.isArray(value) ? value.length : 0);  }, 0);}const candidate = {  tags: Array.from({ length: 1200 }, (_, i) => `tag-${i}`),  audiences: Array.from({ length: 800 }, (_, i) => `aud-${i}`),  rawPayload: "x".repeat(20_000)};console.log({  approximateArrayMembershipValues: estimateArrayEntries(candidate, ["tags", "audiences"]),  warning: "Actual Firestore index-entry count depends on every single-field and composite index. Verify the real index configuration; do not treat this script as billing evidence."});

This small script is intentionally conservative and incomplete. It counts obvious array membership values so code review can flag an extreme document shape. It cannot compute Firestore’s exact index-entry count because that depends on the full deployed single-field and composite index configuration. Exact backend behavior belongs to official index-entry calculation rules and, where appropriate, production/pre-production measurements.

5. Deliberately wrong approach: disable every index to make writes fast

In Standard Native, that breaks required queries. In Enterprise Native, an unindexed query may still execute, which can hide the mistake until data growth turns a prototype scan into expensive latency. “Fewer indexes” is not the objective; “only indexes that buy a required query/performance outcome” is.

The repair sequence is query inventory → index dependency mapping → candidate removal/exemption → contract tests → bounded write/query measurement → staged deployment → observe → retain rollback. Performance changes without query correctness evidence are incomplete.

6. Bounded before/after experiment

process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { initializeApp } from "firebase-admin/app";import { getFirestore, FieldValue } from "firebase-admin/firestore";import { randomUUID } from "node:crypto";initializeApp({ projectId: "demo-atlasmart-firestore" });const db = getFirestore();const percentile = (sorted, p) => {  if (!sorted.length) return null;  const index = Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1);  return sorted[index];};async function runStage({ mode, writes, concurrency }) {  const latencies = [];  let next = 0;  let errors = 0;  async function worker(workerId) {    while (true) {      const i = next++;      if (i >= writes) return;      const started = performance.now();      try {        if (mode === "hot-document") {          await db.doc("benchmarks/ch15-hot").set({            n: FieldValue.increment(1),            updatedAt: FieldValue.serverTimestamp()          }, { merge: true });        } else {          const id = mode === "sequential"            ? `evt-${String(i).padStart(10, "0")}`            : `evt-${randomUUID()}`;          await db.doc(`benchmarks/ch15-${mode}/events/${id}`).set({            mode,            ordinal: i,            observedAt: new Date().toISOString(),            workerId          });        }      } catch (error) {        errors++;        console.error(JSON.stringify({ mode, i, code: error.code, message: error.message }));      } finally {        latencies.push(performance.now() - started);      }    }  }  const t0 = performance.now();  await Promise.all(Array.from({ length: concurrency }, (_, i) => worker(i)));  const elapsedMs = performance.now() - t0;  latencies.sort((a, b) => a - b);  return {    mode, writes, concurrency, elapsedMs,    observedOpsPerSecond: writes / (elapsedMs / 1000),    errors,    latencyMs: {      p50: percentile(latencies, 50),      p95: percentile(latencies, 95),      p99: percentile(latencies, 99),      max: latencies.at(-1)    }  };}for (const stage of [  { mode: "random", writes: 600, concurrency: 12 },  { mode: "sequential", writes: 600, concurrency: 12 },  { mode: "hot-document", writes: 250, concurrency: 12 }]) {  console.log(JSON.stringify(await runStage(stage)));}

Run the same local write harness against two document shapes: a lean event and a deliberately bloated event with large arrays/maps. Record raw latency distributions and serialized document sizes. Then modify the local firestore.indexes.json manifest and rerun. The emulator does not reproduce managed index write cost faithfully enough to claim a production percentage improvement, but it validates the experiment workflow and catches application-level regressions.

7. Standard vs Enterprise index-cost reasoning

Verification checklist

• Every retained composite maps to a named query contract.
• Every exemption names the field’s non-query purpose or replacement query path.
• Large arrays/maps have an explicit reason to be in one document and an index policy.
• Index limits are treated as hard ceilings, not desired operating points.
• Before/after results preserve environment, document shape, index manifest and raw latency samples.
• Enterprise unindexed scans are not described as free.

Production judgment and bridge to Lesson 5

Index fan-out is part of the write path. Reducing it can improve write efficiency, but only when query capability remains intentional. Lesson 5 combines document-key distribution, sequential-index behavior, fan-out and traffic ramp into one repeatable load-test and redesign workflow.

Knowledge check

1. What is index fan-out?
2. Why is 40,000 index entries not a tuning target?
3. Can Enterprise simply omit every index?
4. Why map indexes to named query contracts?
5. What must accompany a before/after write-latency claim?

Summary

AtlasMart now treats indexes as write-path structures with measurable amplification, not merely query configuration. Large arrays/maps and redundant indexes are reviewed against query contracts and hard index-entry limits.

Authoritative references

• Understand reads and writes at scale
• Firestore best practices
• Standard index overview and indexing best practices
• Sharded timestamps
• Key Visualizer overview
• Key Visualizer document-key patterns
• Enterprise Native Core/Pipeline overview
• Enterprise Native index overview
• Enterprise latency troubleshooting
• Firestore quotas and limits
• Firebase current releases
• Firebase Admin Node.js SDK release notes