Chapter 10 · Atomicity Across Business Workflows: Counters, Reservations, Idempotency, and Event-Driven Consistency
Define the True Atomic Boundary: One Document, Transaction Set, or Eventual Multi-Step Workflow
Choose AtlasMart atomic boundaries deliberately and connect Firestore transactions to idempotent eventual workflows with durable outbox/inbox evidence.
Learning outcomes
Identify the smallest atomic boundary that actually protects an AtlasMart business invariant instead of trying to make an entire checkout globally atomic.
Distinguish one-document transforms, multi-document transactions and eventual multi-step workflows with compensation.
Use durable workflow state, idempotency keys, outbox/inbox evidence and correlation IDs so asynchronous steps can be replayed safely.
Explain why external payments, emails and event delivery cannot be made atomic merely by wrapping Firestore writes in a transaction.
Use the Emulator Suite, a Firebase demo project, or an isolated test project for destructive, security-sensitive, billing-sensitive, migration, backup/restore, or write-heavy exercises unless the lesson explicitly marks managed verification as required. Treat shown output as expected evidence unless it is explicitly identified as captured output, and re-check current Firebase/Google Cloud edition, mode, quota, pricing, and security documentation before production execution.
AtlasMart continues the same environment used in Chapters
01–09: project ID demo-atlasmart-firestore,
Standard edition / Native mode /
(default) database for mandatory labs, Firestore
emulator 127.0.0.1:8080, Authentication emulator
127.0.0.1:9099, Emulator UI
127.0.0.1:4000, Firebase CLI
15.30.0, Firebase JavaScript SDK
12.19.0, Firebase Admin Node SDK
14.4.0 with
@google-cloud/firestore 9.1.0, and Node.js 22+.
Mandatory work remains local/no-cost. Cloud Functions,
Eventarc, managed TTL deletion, production IAM, billing,
regional delivery latency and external payment systems are
discussed accurately but are not falsely claimed to have run
in the local emulator.
The local lab simulates duplicate and reordered events deterministically with ordinary Node code so the learner can prove idempotency, compensation and repair behavior without deploying cloud infrastructure. Firestore-triggered Cloud Functions and Eventarc Standard can deliver events at least once; Firestore event ordering is not guaranteed. Firestore TTL deletion is asynchronous and documents are typically removed within about 24 hours after expiration, so TTL is a retention mechanism—not an exact reservation scheduler. Any production p95/p99, event-delivery delay, TTL cleanup delay, trigger retry count or cost must be measured in the actual edition/region/billing configuration rather than inferred from emulator timing.
1. The AtlasMart problem: checkout crosses more systems than Firestore can atomically own
An AtlasMart checkout begins with one SKU in
catalogItems/p-1001, but the complete business
outcome spans inventory, an order, a payment provider, email,
analytics and perhaps fulfillment. Firestore can atomically
update one document, a write batch, or a transaction set inside
Firestore. It cannot commit an external card charge, an Eventarc
delivery and an email atomically with that transaction. The
first design task is therefore not “which SDK method?” but
where is the true invariant boundary?
For this chapter, an atomic boundary is the set of Firestore state that must change all-or-nothing at one instant. An eventual workflow is a sequence of durable steps that may be separated by seconds, retries or failures. A compensation is a deliberate inverse business action, such as releasing a reservation after payment failure; it is not database rollback. An idempotency key identifies one logical request so duplicate deliveries or retries can converge on one outcome. An outbox is durable evidence that an event still needs publishing/processing. An inbox/deduplication record is durable evidence that a delivery has already been applied.
| Business decision | Smallest useful boundary | Why |
|---|---|---|
| Update a last-seen timestamp | One document / serverTimestamp transform | No cross-document read-dependent invariant |
| Reserve one SKU if stock remains | Transaction over item + reservation/order evidence | Decision depends on current stock and must not oversell |
| Create order + immutable audit projection after decision | Batched write | Known values; no fresh read dependency |
| Reserve inventory, charge payment, notify customer | Eventual workflow with durable state and compensation | External systems cannot join a Firestore transaction |
| High-rate approximate telemetry count | Sharded counter | Commutative increments can be distributed; exact read is more expensive |
2. Model the workflow before writing handlers
| State | Meaning | Allowed next states | Durable evidence |
|---|---|---|---|
| NEW | Idempotency command accepted; no inventory reserved yet | RESERVED, REJECTED | workflowCommands/{key} |
| RESERVED | Inventory is held by a reservation document | PAYMENT_PENDING, COMPENSATING | reservations/{orderId} + order state |
| PAYMENT_PENDING | External payment placeholder/event expected | COMPLETED, COMPENSATING, MANUAL_REVIEW | workflowOutbox + workflowEvents |
| COMPLETED | Order is terminally successful | none except explicit administrative correction | orders/{id} terminal state |
| COMPENSATING | Workflow is releasing inventory or reversing downstream effects | CANCELLED, MANUAL_REVIEW | compensationAttempt, reason, correlationId |
| CANCELLED | Compensation completed | none | terminal order + released reservation |
| MANUAL_REVIEW | Automatic progress is unsafe or repeatedly failed | operator-defined repair transition | dead-letter/repair document |
The state machine is intentionally explicit. It prevents a
handler from inventing transitions based only on the event that
happened to arrive. Every transition records a
correlationId, idempotencyKey,
updatedAt and optional lastEventId.
Terminal states reject ordinary replay. A
MANUAL_REVIEW state preserves evidence rather than
deleting the failure that an operator needs to understand.
{ "orderId": "ord-ch10-001", "customerId": "user-atlas-001", "sku": "P-1001", "qty": 1, "state": "PAYMENT_PENDING", "idempotencyKey": "checkout-user-atlas-001-cart-042", "correlationId": "corr-ch10-001", "reservationId": "ord-ch10-001", "paymentAttempt": 1, "lastEventId": "evt-payment-requested-001", "schemaVersion": 3}
3. Transaction the strict inventory boundary, then emit durable work
process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { initializeApp } from "firebase-admin/app";import { getFirestore, FieldValue, Timestamp } from "firebase-admin/firestore";initializeApp({ projectId: "demo-atlasmart-firestore" });const db = getFirestore();
async function reserveOrder(command) { const commandRef = db.doc(`workflowCommands/${command.idempotencyKey}`); const itemRef = db.doc(`catalogItems/${command.sku}`); const orderRef = db.doc(`orders/${command.orderId}`); const reservationRef = db.doc(`reservations/${command.orderId}`); const outboxRef = db.doc(`workflowOutbox/payment-${command.orderId}`); return db.runTransaction(async tx => { const [existing, item] = await Promise.all([tx.get(commandRef), tx.get(itemRef)]); if (existing.exists) return existing.data(); // duplicate logical request converges if (!item.exists || item.get("stock") < command.qty) { const rejected = { ...command, state:"REJECTED", reason:"INSUFFICIENT_STOCK", schemaVersion:3 }; tx.create(commandRef, rejected); tx.create(orderRef, rejected); return rejected; } const now = FieldValue.serverTimestamp(); tx.update(itemRef, { stock: FieldValue.increment(-command.qty), updatedAt: now }); tx.create(reservationRef, { orderId: command.orderId, sku: command.sku, qty: command.qty, state:"HELD", expiresAt: Timestamp.fromMillis(Date.now()+15*60_000), correlationId: command.correlationId, schemaVersion:3, createdAt: now }); tx.create(orderRef, { ...command, state:"RESERVED", schemaVersion:3, createdAt: now, updatedAt: now }); tx.create(outboxRef, { type:"PAYMENT_REQUESTED", orderId:command.orderId, eventId:`evt-payment-requested-${command.orderId}`, correlationId:command.correlationId, state:"READY", createdAt:now }); tx.create(commandRef, { ...command, state:"RESERVED", orderId:command.orderId, createdAt:now }); return { ...command, state:"RESERVED" }; });}
The outbox write is inside the same Firestore transaction as the reservation. That means “reservation committed but no durable payment work exists” is not a possible committed Firestore state. It does not mean the payment itself is atomic with the reservation. A separate worker consumes the outbox and must be idempotent.
4. Wrong approach: pretend the whole checkout is exactly once
A common design charges the payment gateway immediately inside the transaction or trigger and then marks the order completed. That fails under retry: a transaction callback can rerun, a Firestore trigger can invoke more than once, and Eventarc may redeliver. Another common error deletes an outbox/dead-letter record after a failure, removing the only evidence needed for repair.
Keep irreversible external effects outside retryable Firestore transactions. Give the external call its own idempotency key where supported. Persist before/after evidence. Make state transitions conditional. Retain failure records until a documented repair/retention policy says otherwise.
5. Edition, mode and trust boundaries
The mandatory lab uses Standard Native Core operations through the Admin SDK. Client Security Rules deny writes to workflow-owned collections; a trusted server path is responsible for them. Enterprise Native Core can implement the same state-machine ideas, but server concurrency defaults and billing differ, so operational measurements must be repeated. Enterprise Pipeline operations are not a magic multi-system transaction layer. Firestore with MongoDB compatibility uses its own driver/transaction/event surfaces and must be validated separately rather than assuming Native SDK semantics.
6. Reproducible AtlasMart lab
{ "name": "atlasmart-firestore-ch10", "private": true, "type": "module", "engines": { "node": ">=22" }, "dependencies": { "firebase-admin": "14.4.0" }, "devDependencies": { "firebase-tools": "15.30.0" }}
{ "firestore": { "rules": "firestore.rules", "indexes": "firestore.indexes.json" }, "emulators": { "firestore": { "port": 8080 }, "auth": { "port": 9099 }, "ui": { "enabled": true, "port": 4000 } }}
rules_version = '2';service cloud.firestore { match /databases/{database}/documents { match /catalogItems/{productId} { allow read: if true; allow write: if false; } match /profiles/{uid} { allow read, write: if request.auth != null && request.auth.uid == uid; } match /orders/{orderId} { allow read: if request.auth != null && resource.data.customerId == request.auth.uid; allow write: if false; } // Workflow state, reservations, outbox/inbox, dedupe and repair evidence are server-owned. match /workflowCommands/{id} { allow read, write: if false; } match /reservations/{id} { allow read, write: if false; } match /workflowEvents/{id} { allow read, write: if false; } match /workflowOutbox/{id} { allow read, write: if false; } match /workflowDeadLetters/{id} { allow read, write: if false; } match /counters/{counterId}/{document=**} { allow read, write: if false; } match /{document=**} { allow read, write: if false; } }}
mkdir atlasmart-firestore-ch10 && cd atlasmart-firestore-ch10npm init -ynpm install firebase-admin@14.4.0npm install --save-dev firebase-tools@15.30.0# Save firebase.json, firestore.rules and firestore.indexes.json from this lesson.printf '{"indexes":[],"fieldOverrides":[]}' > firestore.indexes.jsonnpx firebase-tools@15.30.0 emulators:start --project demo-atlasmart-firestore --only firestore,auth
Seed catalogItems/p-1001 with stock 2. Run
reserveOrder twice with the same idempotency key
and once with a different key. Assert that the duplicate returns
the existing logical outcome without decrementing stock twice,
while a distinct accepted order consumes one additional unit.
Inspect orders, reservations,
workflowCommands and workflowOutbox in
the Emulator UI.
await db.doc("catalogItems/p-1001").set({sku:"P-1001",stock:2,schemaVersion:3});const command={ orderId:"ord-ch10-001", customerId:"user-atlas-001", sku:"P-1001", qty:1, idempotencyKey:"checkout-user-atlas-001-cart-042", correlationId:"corr-ch10-001"};const first=await reserveOrder(command);const duplicate=await reserveOrder(command);const stock=(await db.doc("catalogItems/p-1001").get()).get("stock");console.log({first,duplicate,stock});if (first.state!=="RESERVED" || duplicate.state!=="RESERVED" || stock!==1) process.exitCode=1;
Production judgment
Choose the narrowest atomic unit that protects the invariant, then accept that the rest of the workflow is a state machine. A smaller atomic boundary usually reduces contention but increases the importance of idempotency, compensation and observability. A larger transaction can simplify local reasoning but cannot absorb external systems. Record every asynchronous step with durable correlation identifiers and explicit terminal/manual-repair states. Lesson 2 applies the same thinking to high-rate counters, where the tradeoff is write distribution versus exact read cost.
Knowledge check
- Why is an external payment not part of a Firestore transaction?
- What does an outbox document prove?
- Why keep an idempotency record?
- Is compensation the same as rollback?
- What should happen when automated repair is unsafe?
Review the answers
1. The external provider does not participate in Firestore commit/rollback; retries can repeat the side effect unless it is separately idempotent.
2. That durable work was committed together with database state and still needs processing; it does not prove the external effect has completed.
3. It lets duplicate logical requests converge on the already-recorded outcome instead of repeating the business action.
4. No. Compensation is a later business action that moves the workflow to a safe state after earlier committed steps.
5. Move the workflow to a durable manual-review state with evidence and correlation IDs rather than silently deleting or guessing.
Summary and next step
Firestore atomicity is powerful but bounded. Reliable business workflows combine a deliberately small atomic core with idempotent asynchronous processing, durable evidence, compensation and human recovery. Next we examine distributed counters as a concrete write-distribution tradeoff.
Authoritative references
- Transactions and batched writes — atomic transaction/batch boundaries and retry behavior.
- Distributed counters — shard-based write distribution and read aggregation tradeoffs.
- Manage data retention with TTL policies — asynchronous deletion behavior, limits, pricing and monitoring.
- Cloud Firestore triggers — at-least-once delivery, non-guaranteed ordering, trigger scope and idempotency requirement.
- Eventarc Standard retry events — at-least-once delivery, duplicate handling, idempotency and dead-letter guidance.
- Firestore best practices — hotspot and scaling guidance.
- Firestore Enterprise overview — edition/mode boundaries to re-check before porting workflow assumptions.
- Firebase release notes — current SDK/tool versions.
- Admin Node.js release notes — 14.4.0 and Firestore client dependency baseline.
- Firebase CLI release notes — 15.30.0 baseline.