Build a Troubleshooting Runbook for Permission Errors, Missing Indexes, Hotspots, Latency, Quota, and Cost Spikes

1. AtlasMart problem: stop debugging by reflex

AtlasMart has accumulated “fixes” from past incidents: retry everything, add indexes whenever a query fails, raise quotas on spikes, and restart listeners when users complain. Those actions can amplify outages. The final lesson converts Chapters 05–24 into an evidence-first runbook: classify the failure, collect the minimum safe evidence, test a hypothesis, change one control, verify, and preserve rollback.

2. Triage matrix: symptom → evidence → action

3. Branch 1: permission errors—identify the authorization plane

1. Is this mobile/web client or trusted server?
2. Client: verify Firebase Auth identity/claims, exact query shape, Rules test, Rules evaluation ALLOW/DENY/ERROR trend.
3. Server/Admin: Rules are bypassed; inspect ADC/workload identity, IAM, application authorization, VPC-SC/perimeter policy and audit log principal.
4. MongoDB compatibility: inspect SCRAM/OIDC/IAM auth and Mongo-compatible command support rather than Firebase client Rules assumptions.

4. Branch 2: missing index or high scan work

In Standard, missing required indexes can fail a query. In Enterprise, the same conceptual access pattern may run by scanning because indexes are optional. Therefore the runbook has two different branches:

5. Branch 3: hotspot/contention

Look for hot-document updates, sequential document IDs, indexed sequential fields, sudden traffic ramps and transaction retry pressure. For Standard Native, eligible Key Visualizer heatmaps can corroborate document/index key concentration. For all modes, combine request status/latency with data-model knowledge. Do not shard a strict invariant such as inventory merely to reduce contention without redesigning correctness.

6. Branch 4: quota and cost spikes

Quota exhaustion and cost spikes can share a root cause—accidental fan-out, listener leaks, abuse, batch jobs—but they are not the same. A quota is a service limit/control; billing is usage multiplied by SKU/pricing. First isolate operation/caller/query family and determine whether work is legitimate. Add backpressure, bounded concurrency, cache/materialization, query/index redesign or abuse protection before requesting a higher ceiling.

7. Mandatory incident drill: saved evidence bundle

{  "incident":"atlasmart-obs-001",  "window":"2026-09-17T00:00:00Z/2026-09-17T00:30:00Z",  "edition":"Standard",  "mode":"Native/Core",  "symptom":"p99 checkout latency + read spike",  "evidence":[    "app-latency.json",    "firestore-latency.json",    "query-insights.json",    "explain-before.json",    "listener-gauge.json",    "billing-note.txt"  ],  "sensitivePayloadsIncluded":false}

import fs from "node:fs";const app=JSON.parse(fs.readFileSync("incident/app-latency.json"));const db=JSON.parse(fs.readFileSync("incident/firestore-latency.json"));const listeners=JSON.parse(fs.readFileSync("incident/listener-gauge.json"));const insights=JSON.parse(fs.readFileSync("incident/query-insights.json"));const findings=[];if (app.p99_ms > db.p99_ms*2) findings.push("client/network/application-overhead-suspect");if (listeners.after > listeners.before*1.5) findings.push("listener-growth-suspect");const top=[...insights].sort((a,b)=>b.read_ops-a.read_ops)[0];if (top && top.avg_docs_scanned > top.avg_results*20) findings.push(`scan-amplification:${top.id}`);console.log({findings});if (!findings.length) process.exitCode=2;

Run the drill twice. First with a listener-growth fixture; second with a scan-amplification fixture. The diagnostic must choose different hypotheses. That proves the runbook is evidence-driven rather than hard-coded to the last incident.

8. Change gate: one hypothesis, one reversible change

9. Machine-check the runbook

{  "branches":[    {"id":"permission","requires":["principal","auth_plane","error_code","policy_evidence"],"rollback":true},    {"id":"index-scan","requires":["query_shape","edition_mode","index_state","explain_or_fixture"],"rollback":true},    {"id":"hotspot","requires":["key_shape","contention_errors","latency_distribution"],"rollback":true},    {"id":"quota-cost","requires":["operation_mix","caller","quota_or_billing_evidence"],"rollback":true}  ]}

import fs from "node:fs";const r=JSON.parse(fs.readFileSync("runbook.json"));for(const b of r.branches){  if(!b.requires?.length) throw new Error(`${b.id}:no-evidence-contract`);  if(b.rollback !== true) throw new Error(`${b.id}:no-rollback-gate`);}console.log(`RUNBOOK_OK branches=${r.branches.length}`);

10. Operational non-guarantees

• A fast Explain sample does not guarantee fleet p99.
• A clean Key Visualizer scan does not exclude network/client bottlenecks.
• Query Insights is delayed and aggregated.
• Monitoring usage metrics are sampled/delayed and not final billing records.
• Emulator timings are not production performance.
• Retries can increase load and cost; retry only documented transient/idempotent operations with bounds.
• Backups/PITR protect recoverability, not live latency or authorization correctness.

11. Bridge to Chapter 25

Chapter 24 established measured work: requests, scans, reads, writes, listeners, storage, errors and latency. Chapter 25 converts those measured dimensions into cost engineering, quotas, forecasts and capacity/usage budgets without mistaking “cheap per operation” for “cheap at fleet scale.”

Knowledge check

1. Why must the runbook identify the authorization plane for a permission error?
2. How does Standard missing-index behavior differ from Enterprise index-optional behavior?
3. What is the first response to a quota spike?
4. Why should each tuning change have a rollback trigger?
5. Which source is financial truth when the usage dashboard disagrees with billing?

Authoritative references

• Firebase: Monitor Cloud Firestore activity — usage dashboards, Monitoring metrics, Security Rules metrics, listener/connections, and dashboard-versus-billing caveats.
• Google Cloud Monitoring metrics reference — current firestore.googleapis.com/* metric types, labels, sampling and launch stages.
• Firebase: Query Explain for Standard/Core queries — planner vs analyze behavior, scan statistics, IAM authentication, and billing semantics.
• Firebase: Enterprise Native Pipeline Query Explain — explain/analyze/stats modes and execution-tree evidence.
• Firebase: Query Insights — normalized queries, latency/read-load statistics, retention, delay and IAM.
• Firebase: Key Visualizer overview — Standard Native hotspot heatmaps, eligibility, metrics, limits, and data retention.
• Firebase: Cloud Firestore audit logging — Data Access/Admin Activity audit evidence and processing_duration.
• Firebase: Firestore editions overview — current observability availability and Standard/Enterprise distinctions.