Key Visualizer / Hotspot Diagnostics Where Available, Application Tracing, Logs, and Correlation IDs

1. AtlasMart problem: the hotspot is visible only when evidence is correlated

AtlasMart receives a flash-sale burst. Client p99 rises, Firestore backend write latency rises, and a few request IDs dominate error logs. The data team suspects sequential event IDs, while the platform team suspects an external API. Key Visualizer, application correlation IDs and structured logs answer different parts: where key-range/index heat concentrates, which user action generated the requests, and whether time was spent before or inside Firestore.

2. Key Visualizer: what it can and cannot show

A Key Visualizer scan covers a two-hour period divided into 10-second segments. Current scan eligibility requires traffic exceeding 3,000 document operations in any minute of that period. Scan data is retained for 14 days. Heatmaps bucket contiguous document or index keys; they do not show every individual request and are not available simply because a database contains lots of data.

3. Key Visualizer metrics are storage-layer evidence

Document-key scans include Ops/s, write/lookup/query rates, average latency and tail latency metrics. Index-key scans expose Index Write Ops/s. Because these are storage-layer measurements, they can be lower than total API-call latency. A clean heatmap does not prove the client network is healthy; an ugly heatmap does not identify the original product action by itself.

4. Correlation IDs: connect product action → backend → Firestore

import crypto from "node:crypto";export function makeContext(incoming={}) {  return {    correlationId: incoming.correlationId || crypto.randomUUID(),    operation: incoming.operation || "unknown",    tenantHash: incoming.tenantId ? crypto.createHash("sha256").update(incoming.tenantId).digest("hex").slice(0,12) : undefined  };}export function safeLog(level, ctx, fields={}) {  const forbidden=["authorization","idToken","refreshToken","email","phone","address"];  for (const k of forbidden) if (k in fields) throw new Error(`forbidden-log-field:${k}`);  console.log(JSON.stringify({severity:level, ...ctx, ...fields, ts:new Date().toISOString()}));}

Hashing a tenant identifier is not automatically sufficient anonymization; the point is to avoid raw business identifiers unless policy explicitly allows them. Never log Firebase ID tokens, refresh tokens, service-account credentials, App Check tokens or full user payloads for convenience.

5. Instrument one AtlasMart request path

import { performance } from "node:perf_hooks";import { makeContext, safeLog } from "./correlation.mjs";export async function lowStock(req, db) {  const ctx=makeContext({correlationId:req.headers["x-correlation-id"], operation:"seller-low-stock", tenantId:req.tenantId});  const t0=performance.now();  safeLog("INFO",ctx,{phase:"start"});  try {    const snap=await db.collection("catalogItems")      .where("sellerId","==",req.sellerId)      .where("stock","<=",5)      .limit(50).get();    safeLog("INFO",ctx,{phase:"firestore-done",ms:performance.now()-t0,results:snap.size});    return snap.docs.map(d=>({id:d.id,...d.data()}));  } catch (e) {    safeLog("ERROR",ctx,{phase:"firestore-error",ms:performance.now()-t0,code:e.code ?? "UNKNOWN"});    throw e;  }}

6. Mandatory local hotspot simulation: deterministic, not performance theater

The emulator cannot reproduce managed key-range splitting or Key Visualizer, so the mandatory exercise simulates key-distribution evidence rather than claiming a real hotspot benchmark.

const sequential=Array.from({length:1000},(_,i)=>`event-${String(i).padStart(8,"0")}`);const scattered=Array.from({length:1000},(_,i)=>`s${(i*7919)%9973}-${i}`);function prefixHistogram(ids,n=2){  const m=new Map(); for(const id of ids){const p=id.slice(0,n);m.set(p,(m.get(p)||0)+1);} return [...m.entries()].sort((a,b)=>b[1]-a[1]);}console.log("sequential top",prefixHistogram(sequential,3).slice(0,5));console.log("scattered top",prefixHistogram(scattered,3).slice(0,5));// This illustrates concentration only; it is NOT a Firestore throughput benchmark.

Then compare this static distribution with the Chapter 15 ramp/hot-document reasoning. Optional production verification can use Key Visualizer only on an isolated workload that naturally meets scan eligibility; do not generate billable traffic solely to unlock a heatmap.

7. Tracing boundaries

OpenTelemetry/Cloud Trace or another tracing system can connect backend spans, HTTP dependencies and Firestore call wrappers, but the Firestore service’s own backend execution evidence still comes from Monitoring/Explain/Insights. Keep a correlation ID in logs and attach trace/span IDs where available. Do not claim a trace span equals Firestore internal processing time unless the instrumentation specifically measures that layer.

8. Controlled failure: verbose logging leaks the incident

Inject an authorization field into safeLog(). The logger must reject it. A common incident anti-pattern is enabling raw request logging to diagnose latency and accidentally persisting credentials or PII. The repair is field allowlisting/redaction, not “remember to delete logs later.”

Production judgment

Use Key Visualizer when Standard Native key/index hotspotting is plausible and scan eligibility exists. Use Query Explain/Insights for query work, Monitoring for fleet-level latency/errors, and app tracing/logging for request causality. A hotspot fix—random IDs, sharding, index exemption, ramp control—must preserve query requirements and correctness.

Knowledge check

1. What current traffic threshold makes a two-hour Key Visualizer scan eligible?
2. What does a diagonal bright band commonly suggest?
3. Why can Key Visualizer latency be lower than user-observed latency?
4. Should you generate production traffic solely to unlock Key Visualizer?
5. What fields should never be casually logged?

Authoritative references

• Firebase: Monitor Cloud Firestore activity — usage dashboards, Monitoring metrics, Security Rules metrics, listener/connections, and dashboard-versus-billing caveats.
• Google Cloud Monitoring metrics reference — current firestore.googleapis.com/* metric types, labels, sampling and launch stages.
• Firebase: Query Explain for Standard/Core queries — planner vs analyze behavior, scan statistics, IAM authentication, and billing semantics.
• Firebase: Enterprise Native Pipeline Query Explain — explain/analyze/stats modes and execution-tree evidence.
• Firebase: Query Insights — normalized queries, latency/read-load statistics, retention, delay and IAM.
• Firebase: Key Visualizer overview — Standard Native hotspot heatmaps, eligibility, metrics, limits, and data retention.
• Firebase: Cloud Firestore audit logging — Data Access/Admin Activity audit evidence and processing_duration.
• Firebase: Firestore editions overview — current observability availability and Standard/Enterprise distinctions.