Cloud Monitoring Metrics for Reads / Writes / Deletes, Latency, Errors, Quotas, Storage, and Listener Behavior

1. AtlasMart incident: “Firestore is slow” is not a diagnosis

At 10:15 UTC the AtlasMart operations channel reports three symptoms at once: checkout p99 latency increased, document reads doubled, and seller dashboards show delayed low-stock updates. One engineer blames Firestore capacity, another blames the network, and a third proposes raising quotas. None of those actions is justified yet. Chapter 24 starts from an operational rule: measure the layer that owns the symptom before changing the system.

2. Four clocks answer four different questions

If client p99 rises while backend p99 stays stable, investigate the path before Firestore: network, connection churn, client CPU, proxy, retries, serialization, or application queueing. If both rise, continue into query/index/hotspot evidence.

3. Cloud Monitoring metrics: build a database-level signal map

4. Realtime listeners: connection count is not listener count

A mobile/web SDK normally maintains one active connection that can multiplex several snapshot listeners. Server client libraries create a connection per snapshot listener. Therefore a “connections doubled” alert and a “listeners doubled” alert imply different things. A screen accidentally mounting ten listeners per user can inflate network/snapshot_listeners without multiplying mobile connections tenfold.

const active = new Map();export function trackedListen(key, subscribe) {  if (active.has(key)) throw new Error(`duplicate-listener:${key}`);  const unsubscribe = subscribe();  active.set(key, { startedAt: Date.now(), unsubscribe });  return () => {    active.get(key)?.unsubscribe();    active.delete(key);  };}export function listenerGauge() { return active.size; }

Local listener registries provide immediate application evidence; Cloud Monitoring gives managed aggregate evidence later. Use both rather than treating either as the whole truth.

5. Mandatory local lab: timing, percentiles, error codes, and correlation IDs

import crypto from "node:crypto";import { performance } from "node:perf_hooks";const samples = [];export async function observed(name, fn) {  const correlationId = crypto.randomUUID();  const start = performance.now();  try {    const value = await fn(correlationId);    samples.push({name, correlationId, ok:true, ms:performance.now()-start});    return value;  } catch (error) {    samples.push({name, correlationId, ok:false, code:error.code ?? "UNKNOWN", ms:performance.now()-start});    throw error;  }}function pct(values, p) {  const a=[...values].sort((x,y)=>x-y);  const i=Math.min(a.length-1, Math.ceil((p/100)*a.length)-1);  return a[Math.max(0,i)] ?? 0;}export function report() {  const ok=samples.filter(x=>x.ok).map(x=>x.ms);  return {count:samples.length, errors:samples.length-ok.length,    p50_ms:pct(ok,50), p95_ms:pct(ok,95), p99_ms:pct(ok,99)};}

1. Seed a bounded AtlasMart fixture in the emulator: 200 catalogItems, 40 sellers, deterministic prices/categories.
2. Run 100 bounded queries through observed(); save raw per-request JSON and a percentile summary.
3. Inject ten deterministic denied client reads through Rules and record the error code separately from latency.
4. Mount and unmount the same listener 100 times; assert the local registry returns to zero.
5. Do not infer production p99 or capacity from these numbers. They prove instrumentation and regression logic only.

6. Controlled failure: averages hide the tail

Create 95 synthetic samples at 10 ms and five at 500 ms. The average looks modest compared with the user-visible tail. A production SLO expressed only as mean latency can therefore remain “healthy” while a small but important cohort is badly affected.

const x=[...Array(95).fill(10), ...Array(5).fill(500)];const mean=x.reduce((a,b)=>a+b,0)/x.length;const sorted=[...x].sort((a,b)=>a-b);const p95=sorted[Math.ceil(.95*x.length)-1];const p99=sorted[Math.ceil(.99*x.length)-1];console.log({mean,p95,p99});// Expected shape: mean is far below p99; investigate tail cohorts.

7. Optional managed verification: Monitoring and audit evidence

In a disposable project, filter api/request_latencies by database, API method and response code; compare p50/p95/p99 with application timing over the same interval. Data Access audit logs can provide processing_duration for audited reads/writes, but enabling and retaining those logs has cost/privacy implications. Never enable verbose payload logging as an incident shortcut.

Production judgment: alert on decisions, not curiosity

Useful alerts have an owner and a decision: sustained backend p99 degradation, rising error-code rate, unexpected listener growth, Rules denials after a deploy, or storage/index growth beyond a forecast envelope. Avoid alerting on every metric independently. Quota increases are not a first response to abuse, retry storms, accidental listeners, or an inefficient query shape.

Knowledge check

1. Why can application p99 be higher than Firestore api/request_latencies?
2. Why is the usage dashboard not financial truth?
3. What is the relationship between mobile connections and snapshot listeners?
4. Why is p99 often more actionable than a mean?
5. What does emulator latency prove?

Authoritative references

• Firebase: Monitor Cloud Firestore activity — usage dashboards, Monitoring metrics, Security Rules metrics, listener/connections, and dashboard-versus-billing caveats.
• Google Cloud Monitoring metrics reference — current firestore.googleapis.com/* metric types, labels, sampling and launch stages.
• Firebase: Query Explain for Standard/Core queries — planner vs analyze behavior, scan statistics, IAM authentication, and billing semantics.
• Firebase: Enterprise Native Pipeline Query Explain — explain/analyze/stats modes and execution-tree evidence.
• Firebase: Query Insights — normalized queries, latency/read-load statistics, retention, delay and IAM.
• Firebase: Key Visualizer overview — Standard Native hotspot heatmaps, eligibility, metrics, limits, and data retention.
• Firebase: Cloud Firestore audit logging — Data Access/Admin Activity audit evidence and processing_duration.
• Firebase: Firestore editions overview — current observability availability and Standard/Enterprise distinctions.