Design a Saga-Like Firestore Workflow with Recovery, Replay, Monitoring, and Manual Repair Paths

Learning outcomes

1. The AtlasMart problem: failures happen between valid commits

At the end of Chapter 9, AtlasMart could protect one transactional invariant. By the end of Chapter 10 it needs to survive a crash after inventory reservation but before payment processing, a duplicate payment event, an expired reservation, a compensation failure, and an operator replay. These are not exceptional edge cases; they are the normal failure surface of a multi-step workflow.

A saga-like workflow decomposes one business process into committed steps with compensating actions for steps that must be reversed. Firestore does not provide a built-in saga coordinator; AtlasMart stores its own durable state machine and evidence. The design goal is convergence to a valid terminal state, not pretending all steps were one global transaction.

2. Persist the workflow timeline

process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { initializeApp } from "firebase-admin/app";import { getFirestore, FieldValue, Timestamp } from "firebase-admin/firestore";initializeApp({ projectId: "demo-atlasmart-firestore" });const db = getFirestore();

async function appendWorkflowEvent({orderId,eventId,type,fromState,toState,correlationId,details={}}) {  await db.doc(`workflowEvents/${eventId}`).create({    orderId,eventId,type,fromState,toState,correlationId,details,    schemaVersion:3,recordedAt:FieldValue.serverTimestamp()  });}

For replay and incident response, do not rely only on the current order document. The current state answers “where are we?”; the event/attempt evidence answers “how did we get here, what already ran, and what can be retried safely?”

3. One transition function owns allowed state movement

const allowed={  NEW:new Set(["RESERVED","REJECTED"]),  RESERVED:new Set(["PAYMENT_PENDING","COMPENSATING"]),  PAYMENT_PENDING:new Set(["COMPLETED","COMPENSATING","MANUAL_REVIEW"]),  COMPENSATING:new Set(["CANCELLED","MANUAL_REVIEW"]),  COMPLETED:new Set(), CANCELLED:new Set(), REJECTED:new Set(), MANUAL_REVIEW:new Set()};async function transitionOrder(orderId,toState,{eventId,correlationId,details={}}) {  const orderRef=db.doc(`orders/${orderId}`);  const eventRef=db.doc(`workflowEvents/${eventId}`);  return db.runTransaction(async tx=>{    const [order,event]=await Promise.all([tx.get(orderRef),tx.get(eventRef)]);    if (event.exists) return {action:"duplicate",state:order.get("state")};    if (!order.exists) throw new Error("ORDER_NOT_FOUND");    const from=order.get("state");    if (!allowed[from]?.has(toState)) throw new Error(`INVALID_TRANSITION_${from}_TO_${toState}`);    tx.update(orderRef,{state:toState,lastEventId:eventId,updatedAt:FieldValue.serverTimestamp()});    tx.create(eventRef,{orderId,eventId,type:"STATE_TRANSITION",fromState:from,toState,correlationId,details,schemaVersion:3,recordedAt:FieldValue.serverTimestamp()});    return {action:"transitioned",from,to:toState};  });}

The same event cannot move state twice because its event record is created atomically with the transition. A different event that proposes an invalid movement fails and is parked for review.

4. Recovery scanner: find workflows that stopped progressing

async function findStalledOrders(cutoff) {  // Production requires the appropriate index; keep the lab fixture bounded.  const snap=await db.collection("orders")    .where("state","in",["RESERVED","PAYMENT_PENDING","COMPENSATING"])    .where("updatedAt","<",cutoff)    .limit(100)    .get();  return snap.docs.map(d=>({id:d.id,...d.data()}));}

A scanner is not itself a repair. Each candidate goes through state-specific logic: requeue an outbox event, expire a reservation, retry compensation, or escalate to manual review. The scanner records a repair attempt ID so repeated scans are observable and idempotent.

5. Replay only from durable evidence

async function requestReplay({deadLetterId,operator,reason}) {  const dlRef=db.doc(`workflowDeadLetters/${deadLetterId}`);  const replayRef=db.doc(`workflowOutbox/replay-${deadLetterId}`);  await db.runTransaction(async tx=>{    const dl=await tx.get(dlRef);    if (!dl.exists) throw new Error("DEAD_LETTER_NOT_FOUND");    if (dl.get("state")!=="NEEDS_REVIEW") return;    tx.set(replayRef,{      type:"REPLAY",sourceDeadLetter:deadLetterId,      originalEventId:dl.get("id"),orderId:dl.get("orderId"),      state:"READY",operator,reason,correlationId:dl.get("correlationId"),      createdAt:FieldValue.serverTimestamp(),schemaVersion:3    },{merge:false});    tx.update(dlRef,{state:"REPLAY_REQUESTED",replayRequestedBy:operator,replayRequestedAt:FieldValue.serverTimestamp()});  });}

Manual repair should never mean “edit the order in the console until it looks right.” A replay request is itself durable, attributable and subject to the same transition rules. For truly exceptional corrections, preserve the before/after state, operator identity and reason.

6. Failure injection matrix

7. Observability: monitor the workflow, not only Firestore API errors

Useful workflow signals include counts by state, age of the oldest non-terminal order, READY/PROCESSING outbox backlog, duplicate-event count, compensation attempts, dead-letter backlog, replay success/failure and reservation age past expiresAt. Add correlationId, orderId, eventId, idempotencyKey, attempt number and state transition to structured logs. These application metrics explain business stuckness that raw Firestore request latency alone cannot.

8. Reproducible end-to-end lab

{  "name": "atlasmart-firestore-ch10",  "private": true,  "type": "module",  "engines": { "node": ">=22" },  "dependencies": {    "firebase-admin": "14.4.0"  },  "devDependencies": {    "firebase-tools": "15.30.0"  }}

{  "firestore": {    "rules": "firestore.rules",    "indexes": "firestore.indexes.json"  },  "emulators": {    "firestore": { "port": 8080 },    "auth": { "port": 9099 },    "ui": { "enabled": true, "port": 4000 }  }}

rules_version = '2';service cloud.firestore {  match /databases/{database}/documents {    match /catalogItems/{productId} { allow read: if true; allow write: if false; }    match /profiles/{uid} { allow read, write: if request.auth != null && request.auth.uid == uid; }    match /orders/{orderId} {      allow read: if request.auth != null && resource.data.customerId == request.auth.uid;      allow write: if false;    }    // Workflow state, reservations, outbox/inbox, dedupe and repair evidence are server-owned.    match /workflowCommands/{id} { allow read, write: if false; }    match /reservations/{id} { allow read, write: if false; }    match /workflowEvents/{id} { allow read, write: if false; }    match /workflowOutbox/{id} { allow read, write: if false; }    match /workflowDeadLetters/{id} { allow read, write: if false; }    match /counters/{counterId}/{document=**} { allow read, write: if false; }    match /{document=**} { allow read, write: if false; }  }}

mkdir atlasmart-firestore-ch10 && cd atlasmart-firestore-ch10npm init -ynpm install firebase-admin@14.4.0npm install --save-dev firebase-tools@15.30.0# Save firebase.json, firestore.rules and firestore.indexes.json from this lesson.printf '{"indexes":[],"fieldOverrides":[]}' > firestore.indexes.jsonnpx firebase-tools@15.30.0 emulators:start --project demo-atlasmart-firestore --only firestore,auth

Create three orders with deterministic IDs:

1. ord-saga-success: reserve → payment pending → duplicate success event → completed.
2. ord-saga-fail: reserve → payment pending → payment failure → compensation → cancelled; replay the failure and verify no second stock release.
3. ord-saga-stuck: reserve → mark an outbox event failed repeatedly → park in manual review → create a replay request.

async function assertState(path,expected) {  const s=await db.doc(path).get();  if (!s.exists || s.get("state")!==expected) throw new Error(`${path}: expected ${expected}, got ${s.get("state")}`);}await assertState("orders/ord-saga-success","COMPLETED");await assertState("orders/ord-saga-fail","CANCELLED");await assertState("orders/ord-saga-stuck","MANUAL_REVIEW");const successEvents=await db.collection("workflowEvents").where("orderId","==","ord-saga-success").get();const dead=await db.collection("workflowDeadLetters").where("orderId","==","ord-saga-stuck").get();console.log({successEventCount:successEvents.size,deadLetterCount:dead.size});

Cleanup should delete only the demo fixture after verification. In production, retention and audit requirements may forbid immediate deletion; Chapter 11 will separate hierarchy deletion, archival and lifecycle policy.

Production judgment

A reliable workflow is one that can explain and recover from partial progress. Keep the strict invariant in the smallest transaction, make asynchronous steps idempotent, record state changes and attempts, compensate intentionally, and preserve a human repair path. Do not use TTL as a scheduler, do not delete failure evidence prematurely, and do not claim exactly-once processing simply because duplicate tests happened to pass.

Chapter 11 builds on this state ownership by examining subcollections, collection groups, recursive deletion and lifecycle—important because workflow evidence itself eventually needs retention and deletion rules.

Knowledge check

1. What makes a saga-like workflow different from one transaction?
2. Why keep a transition history in addition to current state?
3. What should a recovery scanner do with stalled workflows?
4. Why is direct console editing a poor repair procedure?
5. Which metrics reveal workflow health?

Summary

Chapter 10 extends atomic correctness into operational correctness. AtlasMart now has explicit atomic boundaries, sharded telemetry, strict reservations, duplicate-safe event processing and a saga-like recovery model with human repair. The next chapter applies the same ownership discipline to Firestore hierarchy and data lifecycle.

Authoritative references

• Transactions and batched writes — atomic transaction/batch boundaries and retry behavior.
• Distributed counters — shard-based write distribution and read aggregation tradeoffs.
• Manage data retention with TTL policies — asynchronous deletion behavior, limits, pricing and monitoring.
• Cloud Firestore triggers — at-least-once delivery, non-guaranteed ordering, trigger scope and idempotency requirement.
• Eventarc Standard retry events — at-least-once delivery, duplicate handling, idempotency and dead-letter guidance.
• Firestore best practices — hotspot and scaling guidance.
• Firestore Enterprise overview — edition/mode boundaries to re-check before porting workflow assumptions.
• Firebase release notes — current SDK/tool versions.
• Admin Node.js release notes — 14.4.0 and Firestore client dependency baseline.
• Firebase CLI release notes — 15.30.0 baseline.