Index Exemptions for Large Strings / Arrays / Maps, Sequential Fields, TTL Fields, and Write / Storage Cost

Learning outcomes

AtlasMart has a 5–20 KiB product description, nested vendor metadata, user-generated tag arrays, event timestamps, and future TTL fields. Default indexing every surface would be easy—but it can waste storage, amplify writes, approach per-document index-entry limits, and create a hotspot on sequential fields that the application never queries.

1. Exemption means “this field is not a query key here”

An automatic-index exemption is a schema contract: AtlasMart agrees that a field will not participate in the corresponding automatic query/order modes. The benefit is reduced index storage and write-side fan-out. The cost is capability: a future screen cannot suddenly filter/order on the exempted field without revisiting the index design.

2. Hard index limits make unbounded maps/arrays a correctness problem

In Standard, one document may have at most 40,000 total index entries across automatic and manual indexes; one index entry is limited to 7.5 KiB; the combined index-entry size per document is limited to 8 MiB; and indexed field values beyond 1,500 bytes are truncated for index purposes. These are not merely cost warnings: crossing index limits can make a write or index build fail.

Arrays are especially easy to underestimate. An array membership index creates entries per array value, and a manual index containing an array field can produce an entry for each member combined with the other indexed fields. A growing product field with thousands of tags is therefore not just bad taxonomy—it is an index-shape risk.

3. Express exemptions and TTL policy together

{  "indexes": [    {      "collectionGroup": "catalogItems",      "queryScope": "COLLECTION",      "fields": [        { "fieldPath": "category", "order": "ASCENDING" },        { "fieldPath": "price", "order": "ASCENDING" }      ]    },    {      "collectionGroup": "orders",      "queryScope": "COLLECTION_GROUP",      "fields": [        { "fieldPath": "tenantId", "order": "ASCENDING" },        { "fieldPath": "createdAt", "order": "DESCENDING" }      ]    },    {      "collectionGroup": "catalogItems",      "queryScope": "COLLECTION",      "fields": [        { "fieldPath": "tags", "arrayConfig": "CONTAINS" },        { "fieldPath": "rating", "order": "DESCENDING" }      ]    }  ],  "fieldOverrides": [    { "collectionGroup": "catalogItems", "fieldPath": "description", "indexes": [] },    { "collectionGroup": "catalogItems", "fieldPath": "attributes", "indexes": [] },    { "collectionGroup": "catalogEvents", "fieldPath": "expiresAt", "ttl": true, "indexes": [] }  ]}

The expiresAt example intentionally combines ttl: true with an empty indexes array. Current index-definition syntax permits a field-level configuration to carry TTL plus indexing configuration. The mandatory local lab does not rely on TTL deletion—the emulator is not a proof of managed asynchronous TTL behavior. Chapter 21 will treat TTL timing, billing, events, and legal/lifecycle consequences in depth.

4. Sequential fields: understand the specific 500-write/s documented case

Firestore documentation calls out a Standard pattern where an indexed field that increases or decreases sequentially across documents, such as a timestamp in a high-write collection, can cap writes to that collection at 500 writes per second. Exempting the field can bypass that particular index constraint when you do not query it.

This is not a blanket statement that “Firestore only handles 500 writes/s.” Document IDs, key ranges, other indexes, contention, traffic ramp, and schema distribution all matter. Chapter 15 will cover hotspot physics and the 500/50/5 ramp concept. Here, the actionable rule is narrower: do not maintain a sequential index you have no query owner for.

5. Compare index surfaces without inventing backend cost

const products = [  {id:"p-1001", tags:["outdoor","camera"], descriptionBytes:9000, attributeLeaves:4},  {id:"p-1002", tags:["studio","camera"], descriptionBytes:12000, attributeLeaves:6},  {id:"p-1003", tags:["outdoor","iot"], descriptionBytes:7000, attributeLeaves:5}];function relativeSurface(p,{indexDescription,indexAttributes}){  // Relative design metric only: scalar directions + array members + optional nested leaves.  return 2*5 + p.tags.length + (indexDescription?2:0) + (indexAttributes?2*p.attributeLeaves:0);}for (const p of products) console.log(p.id, {  defaultLike: relativeSurface(p,{indexDescription:true,indexAttributes:true}),  exempted: relativeSurface(p,{indexDescription:false,indexAttributes:false})});console.log("Do not convert this model directly into Firestore billing or latency.");

The model demonstrates direction: exempting large, unqueried surfaces lowers index maintenance. It deliberately does not reproduce Firestore's exact internal entry-size accounting. For exact index-entry size, quotas, storage, or billable reads, use the official sizing/pricing formulas and managed metrics for your actual schema.

6. Standard cost semantics: separate storage, writes, and query index reads

In Standard edition, a document set/update is billed as a document write; automatic index-entry writes are not billed as separate document writes, but indexes consume storage and influence write work/latency. Query billing can also include index entries read: current pricing exempts queries with up to one range field from index-entry read charges, while queries with multiple range fields can incur index-entry read charges. KNN vector queries have a different batching rule for vector index entries.

Do not copy those semantics into Enterprise. Enterprise uses read/write units based on byte tranches and index-entry writes can consume write units. Lesson 4 isolates that model explicitly.

7. Vector indexes are manual, dimensional contracts

A vector embedding may look like an array of numbers, but Firestore vector search uses a dedicated vectorConfig index mode. Current vector indexes use the flat index type and support dimensions up to 2048. AtlasMart will not generate embeddings in this chapter; Chapter 17 owns embedding model/version/quality. Here, the goal is to recognize that a vector index has explicit dimensions and a nearest-neighbor query owner.

{  "indexes": [    {      "collectionGroup": "catalogItems",      "queryScope": "COLLECTION",      "fields": [        { "fieldPath": "embedding", "vectorConfig": { "dimension": 8, "flat": {} } }      ]    }  ],  "fieldOverrides": []}

Hands-on lab and verification checklist

1. Extend the Chapter 05 seed with large description and nested attributes exactly as shown in Lesson 1.
2. Run the Chapter 05 query contract suite before and after adding exemptions; expected result IDs must remain identical because those fields are intentionally not query keys.
3. Run fanout-model.mjs and record the relative surface delta for the fixed fixtures.
4. Add a deliberately huge synthetic searchTokens array in a throwaway in-memory object and show the model growing; do not write an unbounded stress object into production.
5. Keep the vector index example out of the mandatory deployment. Record it as “unowned / Chapter 17 pending.”
6. Optional managed project: compare index storage/Query Explain/usage evidence before and after an exemption only with bounded synthetic data and a rollback copy of the old config.

Production judgment

Index exemptions are safest when they encode a stable product decision: “this field is display/lifecycle payload, not a query key.” If a future feature changes that decision, treat re-indexing as a migration with build time and cost, not a one-line code edit. Never use emulator timings to claim write-throughput improvements.

Knowledge check

1. Why can a large map create unexpected index work?
2. What does the 1,500-byte indexed-field limit imply for long text?
3. Why might you exempt a TTL timestamp?
4. Does exempting an indexed sequential timestamp prove unlimited write throughput?
5. Why is a Firestore vector index not equivalent to automatic indexing of a number array?

Summary and next step

Exemptions reduce work by declaring what is not queryable. Lesson 4 flips the edition: Enterprise Native makes indexes optional by default, so the engineering question becomes when a successful table scan must be replaced by an explicit index.

Authoritative references

• Index types in Cloud Firestore — Standard automatic/manual index modes, scopes, entry limits, exemptions, and index fan-out guidance.
• Manage indexes in Cloud Firestore — Missing-index workflow, roles, build state, CLI/console management, and vector indexes.
• Cloud Firestore Index Definition Reference — Current firestore.indexes.json schema, vector configuration, field overrides, and TTL configuration.
• Best practices for Cloud Firestore — Index fan-out, sequential-field, TTL, large string/array/map exemption guidance.
• Understand query performance using Query Explain — Planner versus analyze evidence and billing/scan statistics for managed Firestore.
• Enterprise edition index overview — Optional indexing, sparse/dense behavior, and query-performance reasoning in Enterprise Native mode.
• Firestore Native mode Core/Pipeline overview — Standard/Enterprise indexing requirements and interface differences.
• Search with vector embeddings — Vector index management, flat index type, supported dimensions, and vector-search limitations.
• Firestore pricing — Current document/index-entry billing semantics; re-check region and edition before budgeting.
• Firebase release notes — Current CLI/SDK versions used by the pinned lab.
• Firestore release notes — Enterprise Native/Pipeline launch-stage changes and emulator support.