Chapter 18 · Firestore Enterprise Native Mode: Core vs Pipeline Operations and Advanced Querying
Index-Optional Querying in Enterprise: Query Explain, Performance, and When to Add Indexes
Compare indexed and unindexed Enterprise queries with Query Explain concepts, scan evidence, read-unit consequences, sparse/non-sparse/unique indexes, and a repeatable add-or-remove-index decision process.
1. AtlasMart problem: Enterprise lets an unindexed query work, which can hide a production regression
Use the Emulator Suite, a Firebase demo project, or an isolated test project for destructive, security-sensitive, billing-sensitive, migration, backup/restore, or write-heavy exercises unless the lesson explicitly marks managed verification as required. Treat shown output as expected evidence unless it is explicitly identified as captured output, and re-check current Firebase/Google Cloud edition, mode, quota, pricing, and security documentation before production execution.
The seller dashboard query is correct both with and without an index. That is exactly why Enterprise requires a new operational habit: query success is insufficient. AtlasMart needs a repeatable evidence loop that compares scan bytes, latency, memory and Read Units before deciding whether an index is worth its write/storage cost.
AtlasMart retains the course-wide project identity
demo-atlasmart-firestore, Node.js 22+, Firebase
CLI 15.30.0, Firebase JavaScript SDK
12.19.0, Firebase Admin Node.js SDK
14.4.0, and the Admin-bundled
@google-cloud/firestore 9.1.0.
Chapters 01–17 used Standard edition / Native mode /
(default) as the canonical managed model. Chapter
18 adds an isolated Enterprise-edition emulator profile on
Firestore 127.0.0.1:8180 with Emulator UI
127.0.0.1:4100, so it cannot accidentally share
state with the Standard lab on port 8080. Mandatory exercises
are local/no-cost.
Current Local Emulator Suite documentation allows the
Firestore emulator to be configured with
edition: "enterprise". That proves local
Enterprise-edition configuration and lets us exercise ordinary
document/Core behavior. It does not establish
production latency, byte-based billing, index-build state,
Query Explain statistics, or complete Pipeline/search/DML
parity. Where the current production service is required, the
lesson uses a deterministic query-plan/byte-scan simulator and
marks the real managed command as optional. DML pipeline
stages and Pipeline text/geospatial search are explicitly
labeled Preview.
Learning outcomes
Use Query Explain modes—explain, analyze, stats—for distinct evidence needs.
Interpret data bytes read, result count and execution tree without inventing optimizer guarantees.
Choose between no index, sparse/non-sparse index, and unique index from access patterns.
Account for index write/storage amplification under Enterprise unit billing.
Run safe bounded index add/remove experiments with rollback.
2. Query Explain modes
| Mode | What it does | Use |
|---|---|---|
explain |
Plans without executing | Inspect planner behavior without full query execution |
analyze |
Plans + executes + returns results | Validate rows and execution statistics together |
stats |
Plans + executes, no result payload | Collect runtime evidence when rows are not needed |
Even plan-only Explain has a minimum Enterprise read-unit charge in the managed service. Mandatory local exercises therefore use saved/synthetic Explain-shaped evidence, not a claim of zero-cost cloud profiling.
3. Optional managed Query Explain example
import { field } from "@google-cloud/firestore/pipelines";const q = db.pipeline() .collection("catalogItems") .where(field("sellerId").equal("seller-a")) .sort(field("stock").ascending()) .limit(20);const result = await q.execute({ explainOptions: { mode: "analyze", outputFormat: "text" }});console.log(result.explainStats?.text);// Archive output with database ID, query hash, index state and dataset version.4. Deterministic saved-evidence format
{ "databaseEdition": "enterprise", "mode": "native", "queryId": "seller-a-low-stock-v1", "datasetVersion": "atlasmart-ch18-six-products", "indexState": "none|seller-stock-index", "resultsReturned": 0, "dataBytesRead": 0, "readUnitsObserved": 0, "latencyMs": 0, "queryExplainMode": "analyze", "capturedAt": "ISO-8601", "notes": "managed evidence only; never fill with invented numbers"}5. Enterprise index types change the design space
Enterprise creates no indexes by default. A conventional non-sparse index includes collection documents even when indexed fields are absent (missing values are handled as null in index generation). A sparse index includes only documents that contain at least one indexed field value and can reduce index size. A unique index can enforce uniqueness of indexed field combinations. These are database constraints/performance structures, not substitutes for Security Rules or tenant authorization.
| Choice | Benefit | Cost/risk |
|---|---|---|
| No index | No index write/storage amplification | Collection scan latency/read units can grow |
| Non-sparse | Broad predictable indexed access | More index entries/writes/storage |
| Sparse | Skip documents missing indexed fields | Different membership semantics; query contract must match |
| Unique | Database-enforced uniqueness | Write conflicts/constraint ownership must be designed |
6. Why indexes can make writes more expensive
Enterprise writes are charged in 1 KiB tranches for document/index data processed. Adding indexes reduces read scan work but increases index entries that must be updated/deleted when indexed fields change. The correct objective is not “index everything” or “index nothing”; it is minimum total cost and latency for the observed read/write workload plus correctness constraints.
{ "queryPerMinute": 1200, "writesPerMinute": 80, "fieldsUpdatedFrequently": ["stock", "price"], "mustHaveUniqueConstraint": [], "managedEvidenceRequired": ["dataBytesRead", "readUnits", "writeUnits", "p95Ms", "p99Ms"], "note": "illustrative workload counts; replace with measured AtlasMart traffic"}7. Index experiment protocol
- Freeze a dataset snapshot/version and expected result set.
- Capture plan/runtime evidence without the candidate index on a bounded test database.
- Create the candidate index and wait for ready state.
- Repeat the identical query/load and compare result correctness, bytes read, Read Units and latency distribution.
- Measure write amplification for representative document updates.
- Keep ownership metadata: query IDs served, reason, rollback command, last validation date.
8. Failure injection: an index disappears
In Standard, a query might fail for missing required indexes. In Enterprise, the more subtle failure is performance/cost degradation while correctness remains intact. Alerting therefore needs query latency/Read Unit signals, not only error rate. A “no errors” dashboard can miss an expensive scan regression.
Disable an index globally because one benchmark showed no latency change on six documents. Repair it with production-sized or scaled test data, realistic selectivity, p95/p99, read/write units, and explicit query ownership.
9. Mandatory local comparison
import fs from "node:fs/promises";const products = JSON.parse(await fs.readFile("atlasmart-enterprise-fixture.json", "utf8"));const bytes = x => Buffer.byteLength(JSON.stringify(x), "utf8");const readUnits = n => Math.max(1, Math.ceil(n / 4096));function simulateQuery({sellerId, minStock, indexed}) { const matched = products.filter(p => p.sellerId === sellerId && p.stock >= minStock); const scanned = indexed ? products.filter(p => p.sellerId === sellerId) : products; const scannedBytes = scanned.reduce((s,p)=>s+bytes(p),0); return { indexed, scannedDocs: scanned.length, returnedDocs: matched.length, simulatedScannedBytes: scannedBytes, simulatedReadUnits: readUnits(scannedBytes), ids: matched.map(x=>x.id) };}console.log(simulateQuery({sellerId:"seller-a", minStock:5, indexed:false}));console.log(simulateQuery({sellerId:"seller-a", minStock:5, indexed:true}));// This is a deterministic teaching model, NOT Firestore Query Explain output or a bill.The simulator makes only one claim: an index can reduce the candidate bytes a conceptual query must inspect. It does not model Firestore's optimizer, index encoding, storage overhead, cache, network, concurrency or billing exactly.
Verification checklist
- Every managed Explain record includes edition, database ID, dataset/query hash and index state.
- Result correctness is checked before performance comparison.
- p95/p99 are measured from repeated managed requests, never fabricated from the local simulator.
- Index write/storage cost is evaluated alongside read savings.
- Index removal has a tested rollback/recreate path.
Production judgment and bridge to Lesson 5
Enterprise's index-optional model is powerful because it decouples query validity from index existence, but that moves responsibility to observability. Lesson 5 uses one real access pattern to decide whether Standard Core, Enterprise Core or Enterprise Pipeline is the cleanest implementation.
Knowledge check
- What is the key Enterprise risk of removing an index?
- What is a sparse Enterprise index?
- Why not index every field?
- What does Query Explain analyze mode do?
- What metric should accompany error rate for detecting scan regressions?
Review the answers
1. The query can remain correct while scans, latency and Read Units increase.
2. An index that includes only documents containing a value, including null, for at least one indexed field.
3. Indexes add storage and write work/cost; choose them from measured access patterns.
4. It plans and executes the query and returns runtime statistics plus normal results.
5. Latency and Read Unit/bytes-read evidence, because scans can degrade without producing errors.
Summary and next step
This lesson established the working contract for Index-Optional Querying in Enterprise: Query Explain, Performance, and When to Add Indexes. Keep its edition/mode assumptions, trust boundary, verification evidence, and operational constraints explicit when reusing the pattern.
Next, continue to Translate an Access Pattern Between Standard Core Queries and Enterprise Pipeline Operations and Compare Tradeoffs.
Authoritative references
- Firebase · Overview of Firestore in Native mode (Core and Pipeline operations)
- Firebase · Standard vs Enterprise Native mode support
- Firebase · Get data with Pipeline operations
- Firebase · Perform joins with sub-pipelines
- Firebase · Query Explain for Enterprise
- Firebase · Enterprise Native index overview
- Firebase · Pipeline DML stages (Preview)
- Firebase · Pipeline search stage (Preview)
- Google Cloud · Firestore Enterprise pricing
- Firebase · Connect to the Firestore emulator / Enterprise edition configuration