Chapter 26 · Testing, Emulator Suite, CI/CD, Index/Rules Deployment, and Schema Migration
Create a Production Deployment Checklist Covering Rules, Indexes, Billing, Backups, Observability, and Rollback
Assemble an executable production release gate covering Rules, indexes, schema migration, billing, recovery, observability, canaries, and rollback.
1. AtlasMart go-live: a checklist must be executable evidence
A release checklist that says “security tested,” “indexes okay,” and “backups enabled” is not operational control. Each item needs an owner, command/evidence, environment, pass condition, rollback action, and timestamp. Chapter 26 ends by converting the previous 25 chapters into a release gate that distinguishes local proof from managed production proof.
- Build a production checklist spanning Rules, indexes, schemas, billing, recovery, observability and rollback.
- Attach every gate to evidence rather than subjective status.
- Order deployment to preserve backward compatibility and minimize blast radius.
- Define canary/rollback thresholds before release.
- Keep Standard/Enterprise, client/server and emulator/production assumptions explicit.
Use the Emulator Suite, a Firebase demo project, or an isolated test project for destructive, security-sensitive, billing-sensitive, migration, backup/restore, or write-heavy exercises unless the lesson explicitly marks managed verification as required. Treat shown output as expected evidence unless it is explicitly identified as captured output, and re-check current Firebase/Google Cloud edition, mode, quota, pricing, and security documentation before production execution.
AtlasMart keeps project ID
demo-atlasmart-firestore, Standard-edition Native
mode, database (default), Node.js 22+, Firebase
CLI course baseline 15.30.0, Firebase JavaScript
SDK 12.19.0, Firebase Admin Node SDK
14.4.0,
@firebase/rules-unit-testing 5.0.2, Firestore
emulator 127.0.0.1:8080, Auth emulator
127.0.0.1:9099, and Emulator UI
127.0.0.1:4000. Mandatory work uses a
demo- project and local emulators only. Cloud
deployment, IAM, App Check enforcement, billing, production
indexes, quotas, contention, Query Insights/Key Visualizer,
backup/PITR, and Enterprise/MongoDB-compatibility canaries are
explicitly optional managed checks, never silently inferred
from emulator success.
2. Release evidence record
{ "release": "atlasmart-2026-09-17.1", "commit": "<git-sha>", "firebaseCli": "15.30.0", "runtime": "node-22", "database": { "edition":"standard", "mode":"native", "id":"(default)", "location":"<verify-managed>" }, "local": { "project":"demo-atlasmart-firestore", "rulesTests":"PASS", "integrationTests":"PASS", "migrationAudit":{"mismatches":0,"unreadable":0} }, "managed": { "stagingIndexCanary":"PENDING", "iamCanary":"PENDING", "billingBudgetAlerts":"VERIFY", "backupPitrDrill":"VERIFY", "observabilityDashboards":"VERIFY" }, "rollback": { "featureFlag":"documented", "previousRulesSha":"<sha>", "owner":"release-oncall" }}
3. Gate 1 — repository and environment identity
- Commit SHA/tag is immutable and matches deployed artifact.
- Firebase CLI/SDK/runtime versions are recorded.
- Target project/database ID/edition/mode/location are explicit; no reliance on implicit active CLI project.
- Production does not contain emulator environment variables.
- Secrets/service credentials are not stored in repository or logs.
4. Gate 2 — Rules and trust boundaries
- Mobile/web paths have allow + deny regression tests, including tenant isolation, forged fields and query broadening.
- Rules file hash is recorded and repository copy matches intended deployment.
- Admin/server paths have application authorization tests because they bypass Rules.
- IAM role set is least-privilege and tested with the actual runtime identity.
- App Check enforcement is verified separately where used; it is not treated as user authorization.
5. Gate 3 — indexes and query contracts
- Every new Standard composite/vector query has source-controlled index definition.
- Staging/production-only canary proves required indexes exist and are ready; emulator success is not accepted as evidence.
- Enterprise optional-index query plans are measured when Enterprise is the target.
- Query Explain/Insights evidence exists for new high-frequency/high-cost paths where available.
- Index removal has no active caller and has an application-first rollback path.
6. Gate 4 — schema/migration/backfill
- Old and new readers/writers coexist for the planned compatibility window.
- Backfill is idempotent, resumable, throttled and emits progress/error evidence.
- Failure injection and resume have passed locally.
- Dual-read mismatch and unreadable-document counts are within explicit threshold (normally zero for money-like fields).
- Contract/destructive cleanup is a later release, not bundled with initial expansion.
7. Gate 5 — billing, quota, abuse and capacity
- Chapter 25 workload model is refreshed with current traffic, document/index sizes, retry/listener rates and current dated prices.
- Billing budgets/alerts are configured as notifications; they are not mistaken for hard spend caps.
- Security/backup controls were not weakened to hit a cost target.
- Abuse controls, Rules and App Check strategy are reviewed for worst-case listener/query amplification.
- Bounded load/cost canary has stop conditions; emulator throughput is not used as capacity proof.
8. Gate 6 — backup, PITR, retention and recovery
- Required backup schedule and PITR state are verified in managed configuration.
- Last recovery drill measured RPO/RTO and validated data + index/rule/config dependencies.
- TTL/retention policy excludes legal-hold/audit records as designed.
- Rollback/runbook knows whether recent writes after cutover need reconciliation.
- Operator IAM for backup/restore is least-privilege and auditable.
9. Gate 7 — observability and incident response
- Dashboard/alerts cover request counts, p95/p99 latency, error codes, quota signals and cost/usage.
- Correlation IDs are logged without PII/tokens.
- New query/model has baseline before/after evidence.
- Hotspot diagnostics / Query Insights / Key Visualizer are available where target supports them and thresholds justify use.
- Runbook distinguishes permission, missing-index, hotspot, quota, latency and cost incidents.
10. Deployment order: expand before contract
| Phase | Action | Rollback posture |
|---|---|---|
| 0. Local gate | Rules/integration/migration failure-injection tests. | No production effect. |
| 1. Managed prerequisites | Create/build indexes, verify IAM/backup/observability. | Do not release callers until ready. |
| 2. Compatible config | Deploy Rules that allow old + new schema. | Restore previous reviewed Rules if deny/error threshold trips. |
| 3. App canary | Release dual reader/writer behind feature flag. | Disable flag immediately. |
| 4. Backfill | Run bounded resumable job; monitor progress/mismatch/errors. | Pause job; application still reads old/new. |
| 5. Broaden rollout | Increase cohort after SLO/cost/security evidence passes. | Reduce cohort/flag off. |
| 6. Contract later | Remove legacy fields/rules/indexes only after rollback window closes. | Separate approval; destructive work is not emergency rollback. |
11. Canary thresholds must exist before traffic
{ "windowMinutes": 30, "rollbackIf": { "permissionDeniedRate": "> baseline + agreed delta", "dualReadMismatchRate": "> 0 for price fields", "unreadableDocumentCount": "> 0", "p95Latency": "> release SLO budget", "errorRate": "> release SLO budget", "unexpectedCostMultiplier": "> 1.25x modeled canary envelope" }, "note": "Thresholds are product/SLO decisions; these labels are not universal Firestore values."}
12. Controlled failure drill: rollback under a mixed-version dataset
Inject a canary failure after 40% of documents are migrated to v2. The correct rollback is:
- Stop rollout / disable v2-preferred feature flag.
- Pause backfill.
- Confirm v1/dual reader serves both migrated and unmigrated documents.
- Restore prior Rules only if compatible with the mixed dataset; otherwise keep the expanded Rules while application is rolled back.
- Preserve v2 fields; do not run destructive reverse migration during the incident.
- Record decision, timestamps, mismatch/error counts and recovery of SLO.
“Rollback” is not always “restore every file to yesterday.” Database releases create persistent data/config state. The rollback unit is the business behavior, with reconciliation where needed.
13. Edition/mode matrix for the final gate
| Target | Local coverage | Mandatory managed verification |
|---|---|---|
| Standard Native Core | Firestore/Auth emulator, Rules, client/server behavior, migration logic. | Composite/vector indexes, limits/contention, IAM, billing, backups/PITR, regional latency/observability. |
| Enterprise Native Core | Enterprise edition emulator where supported for compatible API behavior. | RU/WU/realtime billing, optional-index plans, Enterprise limits, IAM/network/recovery. |
| Enterprise Pipeline | Deterministic pipeline/query-plan simulation for features the local stack cannot faithfully execute. | Server SDK support, launch-stage feature behavior, Pipeline query plans/cost/security. |
| MongoDB compatibility | Compatibility fixtures/local MongoDB source tests where useful. | Real compatible endpoint/auth/driver/tool matrix, index/transaction/change semantics, Enterprise billing. |
14. Mandatory local deployment rehearsal
- Run the full demo-project CI from clean seed.
- Hash Rules/index files and write release evidence JSON.
- Run schema-v2 failure injection, resume and audit.
-
Simulate index readiness state and refuse to enable the
feature until it reads
READY. - Enable a deterministic 25% canary cohort; inject a mismatch and assert rollback policy disables it.
-
Produce final checklist with every cloud-only item marked
VERIFY_MANAGED—never fakePASS.
The release artifact is reproducible, rollback is executable, local evidence is complete, and every emulator gap has a named managed verification owner before production approval.
Production judgment and bridge to Chapter 27
The checklist is not bureaucracy; it is the boundary between a collection of Firestore features and an operable system. Chapter 27 uses this release discipline in the capstone: design the model, security, scaling, retrieval, recovery and operations together, then defend the decisions with measured evidence.
Knowledge check
- Why must a checklist item contain evidence and a rollback action?
- What should happen before releasing code that needs a new composite index?
- Why can Rules rollback be unsafe after a schema migration begins?
- What is the correct response to a canary failure in a mixed-version dataset?
-
Why must cloud-only checklist items remain
VERIFY_MANAGEDlocally?
Review the answers
1. A subjective “done” cannot be audited or executed during an incident; evidence proves the state and rollback bounds the blast radius.
2. Deploy/build the index and prove readiness in a managed staging/target environment.
3. The previous Rules may reject documents/clients in the new mixed schema; behavior rollback and data/config rollback are separate.
4. Disable the feature/canary, pause backfill, serve through compatible readers, then investigate/reconcile without destructive reverse migration.
5. The emulator cannot prove IAM, production indexes/limits, billing, recovery, real latency and other managed-service properties.
Summary and next step
This lesson established the working contract for Create a Production Deployment Checklist Covering Rules, Indexes, Billing, Backups, Observability, and Rollback. Keep its edition/mode assumptions, trust boundary, verification evidence, and operational constraints explicit when reusing the pattern.
Next, continue to Define User Journeys, Data Model, Queries, Realtime/Offline Needs, Atomicity, Retention, Security, SLOs, and Cost Budget.
Authoritative references
- Firebase: Connect your app to the Cloud Firestore Emulator — demo projects, edition selection, Admin SDK environment variables, reset/import/export, Rules reports, and production differences.
-
Firebase: Install, configure and integrate Local Emulator
Suite
—
emulators:exec, import/export and CI workflow. - Firebase: Test Cloud Firestore Security Rules — emulator rules testing and CI execution.
-
Firebase: Build Security Rules unit tests
—
@firebase/rules-unit-testing, mocked auth, clearing state and disabled-rules fixture setup. - Firebase: Manage and deploy Security Rules — source control, local testing and selective deployment.
- Firebase: Manage indexes in Cloud Firestore — source-controlled index definitions and CLI deployment.
- Firebase: Cloud Firestore index definition reference — composite, field override and vector index JSON shapes.
- Firebase CLI reference — configured multi-database rules/indexes and deployment selectors.