Chapter 16 · Aggregation Queries, Server-Side Count/Sum/Avg, Materialized Aggregates, and Analytics Boundaries
When to Maintain Write-Time Aggregates for Realtime Display vs Run Read-Time Aggregations
Compare read-time aggregation with transaction-maintained and event-maintained summary documents, including lag, idempotency, contention, repair, and security tradeoffs.
1. AtlasMart problem: rating average appears on every product card
Use the Emulator Suite, a Firebase demo project, or an isolated test project for destructive, security-sensitive, billing-sensitive, migration, backup/restore, or write-heavy exercises unless the lesson explicitly marks managed verification as required. Treat shown output as expected evidence unless it is explicitly identified as captured output, and re-check current Firebase/Google Cloud edition, mode, quota, pricing, and security documentation before production execution.
A query-time average is elegant for occasional detail views. It is less attractive when the same value is rendered on hundreds of cards, refreshed often, expected to update through a realtime listener, or computed across a growing review set. A materialized aggregate stores a derived summary as a normal Firestore document. Reads become cheap and listenable, but every source mutation now has to maintain derived state correctly.
AtlasMart continues the same mandatory environment used in
Chapters 01–15: project ID
demo-atlasmart-firestore, Standard edition /
Native mode / (default) database, Firestore
emulator 127.0.0.1:8080, Authentication emulator
127.0.0.1:9099, Emulator UI
127.0.0.1:4000, Firebase JavaScript SDK
12.19.0, Firebase Admin Node.js SDK
14.4.0 carrying
@google-cloud/firestore 9.1.0,
@firebase/rules-unit-testing 5.0.2,
and Node.js 22+. For continuity with Chapters 01–15 the lab
remains pinned to Firebase CLI 15.30.0; CLI
15.30.1 is now available, but its patch notes do
not change the aggregation semantics taught here. Mandatory
work remains local/no-cost. The emulator is useful for
deterministic behavior and rules tests, but it is not billing
evidence, production latency evidence, Enterprise scan-cost
evidence, or a warehouse.
Core aggregation queries support count(),
sum() and average()/avg()
naming depending on SDK. They execute on the backend, skip
local cache and pending local writes, do not support realtime
listeners or offline queries, and can return
DEADLINE_EXCEEDED when an aggregation cannot
complete within 60 seconds. In Standard Native, aggregation
pricing is based on index entries read, billed as one read for
each batch of up to 1,000 index entries with a minimum of one
document read. Enterprise billing uses read units based on
data processed rather than this Standard formula. Enterprise
Pipeline operations have a distinct
aggregate(...) stage with grouping and
accumulator capabilities. Firestore with MongoDB compatibility
has its own MongoDB aggregation surface and must not be
treated as the Native SDK API.
Learning outcomes
Choose synchronous transaction materialization when the summary and source write fit one safe atomic boundary.
Design asynchronous materialization as an idempotent workflow that tolerates duplicate delivery and measurable lag.
Recognize when one summary document becomes a write hotspot and when sharded counters are appropriate.
Build reconciliation as a first-class repair path rather than trusting derived counters forever.
Compare read-time and write-time cost/freshness without claiming asynchronous materialization is realtime.
2. Same source truth, four read models
| Pattern | Freshness/UX | Write amplification | Read work | Correctness/repair | Best fit |
|---|---|---|---|---|---|
| Query-time aggregation | Fresh server answer when requested; not realtime/offline | None beyond source writes | Scales with index/data scanned; Standard bills index entries | No derived state to drift; underlying query/rules must be valid | Occasional counts/sums/averages on bounded query shapes |
| Materialized summary document | Realtime/listenable if clients listen to summary | Extra write(s) per source mutation | One/few document reads | Can drift if async; needs idempotency/reconciliation | Frequently displayed dashboard value with bounded writer contention |
| Distributed counter/shards | Near-realtime after shard writes; exact read requires all shards or roll-up | One shard write per event plus optional roll-up | Reads grow with shard count unless roll-up used | Shard total is deterministic; roll-up may lag and needs repair | High-frequency additive counters |
| External warehouse | Usually delayed relative to operational DB | Replication/export pipeline | Analytical scans/SQL in warehouse | Must track lag, duplicates, backfills, reconciliation | Historical, multidimensional, large-scale analytics |
No row wins universally. The correct choice depends on how often the value changes, how often it is read, freshness requirements, contention, cost model, and operational appetite for repair.
3. Pattern A: synchronous transaction-maintained summary
import { FieldValue } from "firebase-admin/firestore";export async function createPublishedReviewWithSummary(db, productId, reviewId, review) { const reviewRef = db.doc(`products/${productId}/reviews/${reviewId}`); const summaryRef = db.doc(`products/${productId}/metrics/ratings`); await db.runTransaction(async (tx) => { const current = await tx.get(summaryRef); const s = current.exists ? current.data() : { numericCount: 0, ratingSum: 0 }; const isNumeric = Number.isFinite(review.rating); const numericCount = s.numericCount + (isNumeric ? 1 : 0); const ratingSum = s.ratingSum + (isNumeric ? review.rating : 0); tx.create(reviewRef, review); tx.set(summaryRef, { public: true, publishedCount: FieldValue.increment(1), numericCount, ratingSum, ratingAverage: numericCount ? ratingSum / numericCount : null, updatedAt: FieldValue.serverTimestamp(), source: "transaction" }, { merge: true }); });}// The review and summary are atomic here, but the summary document can become a hotspot.// External side effects still do not belong inside the retryable transaction callback.
This pattern makes creation of a published review and update of
its rating summary atomic. It is compelling when the source
write path is centralized and write rate is comfortably below a
hot-document contention threshold. The transaction callback may
retry, so external side effects still stay outside it. If many
independent writers all update
products/p-1001/metrics/ratings, that summary
document becomes a shared contention point.
4. Pattern B: asynchronous idempotent materializer
import { FieldValue } from "firebase-admin/firestore";export async function applyReviewEvent(db, event) { const eventRef = db.doc(`aggregateEvents/${event.id}`); const summaryRef = db.doc(`products/${event.productId}/metrics/ratings`); await db.runTransaction(async (tx) => { if ((await tx.get(eventRef)).exists) return; // deduplicate replay const summary = await tx.get(summaryRef); const s = summary.exists ? summary.data() : { numericCount: 0, ratingSum: 0 }; const deltaCount = Number.isFinite(event.rating) ? event.direction : 0; const deltaSum = Number.isFinite(event.rating) ? event.direction * event.rating : 0; const numericCount = s.numericCount + deltaCount; const ratingSum = s.ratingSum + deltaSum; tx.set(eventRef, { appliedAt: FieldValue.serverTimestamp(), productId: event.productId, direction: event.direction }); tx.set(summaryRef, { public: true, numericCount, ratingSum, ratingAverage: numericCount ? ratingSum / numericCount : null, source: "event-materializer", updatedAt: FieldValue.serverTimestamp() }, { merge: true }); });}
Asynchronous processing breaks the atomic boundary: the review
can commit before the summary changes. That creates an explicit
consistency window. The durable
aggregateEvents/{eventId} record makes duplicate
delivery harmless at the database layer. This mirrors the
at-least-once event reality established in Chapter 10. The UI
may listen to the summary, but the summary itself can still lag
the source event.
A listener can deliver the summary update quickly after it is written; it cannot make an event handler execute exactly once or instantly. Measure source-event-to-summary lag separately.
5. Pattern C: distributed count/sum shards
import { FieldValue } from "firebase-admin/firestore";import { randomInt } from "node:crypto";const SHARDS = 8;export async function addRatingToShard(db, productId, numericRating) { const shard = String(randomInt(0, SHARDS)).padStart(2, "0"); await db.doc(`products/${productId}/ratingCounterShards/${shard}`).set({ numericCount: FieldValue.increment(1), ratingSum: FieldValue.increment(numericRating) }, { merge: true });}export async function readRatingShards(db, productId) { const snap = await db.collection(`products/${productId}/ratingCounterShards`).get(); let numericCount = 0, ratingSum = 0; for (const doc of snap.docs) { numericCount += doc.get("numericCount") ?? 0; ratingSum += doc.get("ratingSum") ?? 0; } return { numericCount, ratingSum, ratingAverage: numericCount ? ratingSum / numericCount : null, shardReads: snap.size };}
For additive metrics, shards distribute write pressure. Average
is derived as sum / count. The tradeoff moves to
reads: exact shard aggregation requires reading each shard, and
more shards mean more read work. A slower roll-up document can
reduce client reads at the cost of bounded staleness. This is a
good example of intentionally exchanging write distribution for
read complexity.
6. Repair is part of the design
import { AggregateField, FieldValue } from "firebase-admin/firestore";export async function reconcileRatings(db, productId) { const q = db.collection(`products/${productId}/reviews`).where("published", "==", true); const truth = (await q.aggregate({ publishedCount: AggregateField.count(), ratingSum: AggregateField.sum("rating"), ratingAverage: AggregateField.average("rating") }).get()).data(); await db.doc(`products/${productId}/metrics/ratings`).set({ public: true, ...truth, source: "reconciliation", reconciledAt: FieldValue.serverTimestamp() }, { merge: true }); return truth;}
Derived state should be treated as rebuildable. Reconciliation
compares the materialized value against source truth and records
when the repair happened. In the local fixture, the expected
published review truth is count=5, numeric
sum=12, and average=4. A repair job
that produces a different answer is either exposing data drift
or using a different query population—both deserve
investigation.
7. Deliberately wrong approach: non-idempotent trigger increments forever
A function receives a review-created event and executes
FieldValue.increment(1) without deduplicating the
event. If the event is delivered twice, the counter is wrong.
Retrying the trigger makes the damage worse, not better. Repair
it by giving every logical change a durable event identity,
applying it transactionally once, recording application
evidence, and retaining a reconciliation path that can rebuild
the summary.
8. Cost and correctness surface
| Question | Read-time aggregate | Materialized summary | Distributed counter |
|---|---|---|---|
| Source write cost | No extra derived write | At least one extra summary/event write | Shard write; optional roll-up writes |
| Display read | Aggregation scan/index work per refresh | One summary document read/listener | All shards for exact total, or one roll-up read |
| Realtime/offline | No listener/offline aggregation | Normal document can be cached/listened to | Shard/roll-up documents can be cached/listened to |
| Drift risk | No persisted derived state | Yes if maintenance fails/duplicates | Shard truth additive; roll-up can lag/drift |
| Repair | Rerun query | Recompute from source and overwrite | Sum shards and/or rebuild roll-up |
9. Security boundary for derived documents
Clients that may read a product summary do not automatically need permission to write it. A safe pattern is public/readable metrics with writes denied to untrusted clients; a trusted backend or tightly validated transaction updates the summary. If client-side transactions maintain the aggregate, Rules must validate both the source document and aggregate invariants across the write set, which adds complexity and access-call pressure. Prefer the simplest trust path that meets the product requirement.
Production judgment and bridge to Lesson 4
Materialization is an operational feature, not a cache checkbox. It introduces lag targets, deduplication, contention, repair and monitoring. When the question expands from “current rating average” to “monthly revenue by seller, region, category, cohort and campaign over years,” even a well-designed materialized Firestore document is the wrong analytical substrate. Lesson 4 draws that boundary.
Knowledge check
- What new failure mode appears when an aggregate is materialized?
- Why can a single materialized summary become a hotspot?
- What does a distributed counter trade?
- Can a listener make an asynchronous summary exactly realtime?
- What is the safest mindset for derived aggregate data?
Review the answers
1. Derived state can lag or drift from source truth, so idempotency and reconciliation become required.
2. All writers may contend on the same document even if source documents are distributed.
3. It improves write distribution but increases exact-read work with the number of shards, unless a lagging roll-up is used.
4. No. It can deliver the summary after it changes; upstream event processing can still be delayed or duplicated.
5. Treat it as rebuildable state with a recorded source query/contract and a deterministic reconciliation path.
Summary
AtlasMart can now choose between no-derived-state query-time arithmetic, a low-read materialized summary, or distributed additive shards, with lag, contention and repair made explicit.
Authoritative references
- Summarize data with aggregation queries
- Understand Cloud Firestore billing
- Write-time aggregations
- Distributed counters
- Securely query data
- Connect to the Firestore Emulator and emulator differences
- Firestore Standard Core operations overview
- Firestore Enterprise Native Core/Pipeline overview
- Enterprise Pipeline aggregate stage
- Enterprise pricing examples
- MongoDB compatibility behavior differences
- Managed Firestore export/import
- Firebase Extensions migration guidance, including Stream Firestore to BigQuery
- Firebase JavaScript SDK release notes
- Firebase Admin Node.js SDK release notes