Audit Index Usage, Remove Redundant Structures, and Validate Query Latency / Cost Before and After Changes

Learning outcomes

After several teams add screens, AtlasMart's index file can accumulate “just in case” composites, abandoned experiments, field exemptions with no explanation, and future vector definitions. The final lesson establishes an audit loop: every index has an owner, every deletion has dependency tests, and every performance/cost claim has measured evidence from the environment that can actually prove it.

1. An index inventory needs ownership, not just JSON

firestore.indexes.json tells Firestore what to build, but it does not tell humans why the structure exists. Add a neighboring manifest that records the owning query/test, edition, scope, data surface, risk, and rollback. This is the bridge between product access patterns from Chapter 03, query contracts from Chapter 05, and index implementation from this chapter.

2. Canonicalize definitions so duplicate structures are detectable

import fs from "node:fs";import assert from "node:assert/strict";const cfg=JSON.parse(fs.readFileSync("firestore.indexes.json","utf8"));const normalized = cfg.indexes.map(x=>JSON.stringify({  collectionGroup:x.collectionGroup,  queryScope:x.queryScope,  fields:x.fields}));const duplicates = normalized.filter((x,i,a)=>a.indexOf(x)!==i);assert.deepEqual(duplicates,[],"duplicate manual index definitions found");const experimental=cfg.indexes.find(x=>x.collectionGroup==="catalogItems" &&  x.fields?.some(f=>f.fieldPath==="tags"&&f.arrayConfig==="CONTAINS") &&  x.fields?.some(f=>f.fieldPath==="rating"&&f.order==="DESCENDING"));console.log({manualIndexes:cfg.indexes.length, fieldOverrides:cfg.fieldOverrides?.length??0,  experimentalPresent:Boolean(experimental)});

This check catches exact duplicates and the known experiment. A mature audit should also compare semantically overlapping indexes, but automated deletion based on set similarity is dangerous: direction, field order, scope, array/vector mode, density, API scope, and the implicit/explicit document-name dimension can all matter.

3. Prove dependencies before removing anything

Start with the query-owner suite, not a delete command. Search application code/tests for the exact access pattern, run all query-contract tests, inspect saved dashboards/Query Insights where available, and ask whether background jobs or old clients still depend on the structure. For a public mobile application, version skew matters: an older client can keep issuing a query after the newest code stops.

Candidate: IDX-TAGS-RATING-EXPERIMENTCurrent definition: tags CONTAINS + rating DESC, COLLECTION scopeKnown query owner: noneStatic config audit: PASSRepository query search: no owner foundEmulator semantic regression suite: PASS (does not prove managed index dependency)Managed Query Insights / logs checked: [record real evidence if available]Old-client compatibility window: [record policy]Rollback JSON preserved: YESRemoval approved by: [owner]Post-removal observation window: [define before change]

4. Define before/after metrics that the environment can actually prove

Use the emulator for correctness and Rules regression. Use managed Firestore for index readiness, Query Explain, actual scan/index evidence, storage/usage, and real latency. Query Explain planning mode can show index selection without executing the full query and has its own documented charge; analyze mode executes the query and is billed normally. One analyzed request is not a percentile benchmark.

{  "environment": {    "project": "DISPOSABLE_PROJECT_ID",    "database": "(default)",    "edition": "standard",    "location": "RECORD_ACTUAL_LOCATION",    "sdk": "@google-cloud/firestore 9.1.0"  },  "fixture": { "documents": "RECORD_ACTUAL_COUNT", "averageBytes": "RECORD_ACTUAL" },  "query": "RECORD_EXACT_FILTERS_AND_ORDER",  "before": { "indexState": "RECORD", "explain": "PASTE_SANITIZED_REAL_METRICS" },  "after":  { "indexState": "RECORD", "explain": "PASTE_SANITIZED_REAL_METRICS" },  "latencyBenchmark": "separate repeated-run distribution; do not infer p95/p99 from Query Explain once"}

For Standard cost, include document reads, applicable index-entry reads, storage, and network. For Enterprise, use read/write units and index-write implications instead. Do not merge the two models into one spreadsheet row labeled “Firestore cost.”

5. Remove the deliberate experiment in configuration-as-code

The end-state file removes tags + rating because no current query owner justifies it. It keeps the two Chapter 05 query-owned composites and the two proven field exemptions. TTL and vector examples remain documented outside the deployable baseline until their future chapters create real owners.

{  "indexes": [    {      "collectionGroup": "catalogItems",      "queryScope": "COLLECTION",      "fields": [        { "fieldPath": "category", "order": "ASCENDING" },        { "fieldPath": "price", "order": "ASCENDING" }      ]    },    {      "collectionGroup": "orders",      "queryScope": "COLLECTION_GROUP",      "fields": [        { "fieldPath": "tenantId", "order": "ASCENDING" },        { "fieldPath": "createdAt", "order": "DESCENDING" }      ]    }  ],  "fieldOverrides": [    { "collectionGroup": "catalogItems", "fieldPath": "description", "indexes": [] },    { "collectionGroup": "catalogItems", "fieldPath": "attributes", "indexes": [] }  ]}

node audit-indexes.mjs# local semantic/rules suite firstnpx firebase-tools@15.30.0 emulators:exec --project demo-atlasmart-firestore --only firestore,auth "node seed.mjs && node query-contracts.mjs"# OPTIONAL MANAGED PROJECT ONLY — requires authenticated IAM and may use quota/billingnpx firebase-tools@15.30.0 deploy --only firestore:indexes --project DISPOSABLE_PROJECT_IDnpx firebase-tools@15.30.0 firestore:indexes --database="(default)" --project DISPOSABLE_PROJECT_ID

Before the optional deploy, diff the generated plan/config so you know which index will be deleted or created. Do not run these managed commands against an unrelated production project simply because your shell has a default Firebase alias.

6. Rollback is re-add + wait ready + verify—not “git revert completed”

Source control rollback is necessary but not sufficient. If removing an index breaks a Standard query, restoring the JSON and redeploying starts a managed index build. The application should not resume the dependent traffic until the index reaches a usable state and the owning query passes. This is why risky index removals need feature gates, old-client awareness, and an observation window.

1. Freeze/gate the affected query path if errors or latency regression appear.2. Restore the exact prior index definition from version control.3. Deploy the index to the correct project/database.4. Monitor managed index build status until READY.5. Run the owning query contract against the managed database.6. Capture Explain/latency/error evidence.7. Re-enable traffic gradually and monitor.8. Record the incident and prevent the same unowned deletion path.

7. Audit cadence and ownership

Run the static manifest audit in CI on every index change. Perform a deeper usage audit when major screens, query shapes, tenant cardinality, or editions change—not on an arbitrary “delete indexes every month” schedule. A configuration that was redundant at 10,000 documents may become valuable after a new query arrives; an index that was critical can become dead after an API version retires.

Include index configuration in backup/recovery/runbook thinking. Firestore data backup does not replace source-controlled Rules/index definitions and application deployment metadata. Recovery drills should verify that query-supporting configuration is restored alongside data.

Hands-on lab and acceptance checklist

1. Start from the experimental manifest and confirm the audit reports experimentalPresent: true.
2. Run all Chapter 05/06 emulator query contracts and static configuration contracts.
3. Create the removal-gate record for IDX-TAGS-RATING-EXPERIMENT.
4. Replace the experimental file with the audited end-state baseline and rerun static + emulator suites.
5. Confirm the two field exemptions remain and no vector/TTL experimental definition accidentally entered the deployable baseline.
6. Optional managed disposable project: deploy, wait for readiness/deletion state, rerun query owners, capture real Explain/usage evidence, and practice restoring the removed index definition in a non-production environment.
7. Archive the manifest with exact CLI/SDK versions, edition, mode, database ID, location, billing status, fixture shape, and what was/was not executed.

Production judgment

Good index hygiene is conservative: remove structures only with evidence, because cleanup has a correctness blast radius in Standard and a performance blast radius in Enterprise. Cost optimization that creates incident risk is not optimization. Treat index changes like schema/API migrations with tests, staged rollout, observability, and rollback time.

Knowledge check

1. Why is firestore.indexes.json alone an incomplete index inventory?
2. What evidence should precede deleting a Standard composite index?
3. Why is “git revert” not an instantaneous index rollback?
4. Which evidence belongs to the emulator versus managed Firestore?
5. Why keep the vector example out of the final deployable baseline?

Summary and next step

Chapter 06 ends with a minimal, owned index baseline, explicit exemptions, and an evidence/rollback workflow. Chapter 07 adds realtime listeners; indexing remains underneath those queries, while listener lifecycle, reconnects, metadata, and read cost introduce a new operational dimension.

Authoritative references

• Index types in Cloud Firestore — Standard automatic/manual index modes, scopes, entry limits, exemptions, and index fan-out guidance.
• Manage indexes in Cloud Firestore — Missing-index workflow, roles, build state, CLI/console management, and vector indexes.
• Cloud Firestore Index Definition Reference — Current firestore.indexes.json schema, vector configuration, field overrides, and TTL configuration.
• Best practices for Cloud Firestore — Index fan-out, sequential-field, TTL, large string/array/map exemption guidance.
• Understand query performance using Query Explain — Planner versus analyze evidence and billing/scan statistics for managed Firestore.
• Enterprise edition index overview — Optional indexing, sparse/dense behavior, and query-performance reasoning in Enterprise Native mode.
• Firestore Native mode Core/Pipeline overview — Standard/Enterprise indexing requirements and interface differences.
• Search with vector embeddings — Vector index management, flat index type, supported dimensions, and vector-search limitations.
• Firestore pricing — Current document/index-entry billing semantics; re-check region and edition before budgeting.
• Firebase release notes — Current CLI/SDK versions used by the pinned lab.
• Firestore release notes — Enterprise Native/Pipeline launch-stage changes and emulator support.