Idempotency Keys, Exactly-Once Illusions, Cloud Functions / Eventarc Retries, and Deduplication

Learning outcomes

1. The AtlasMart problem: the same payment-result event can arrive twice

A payment adapter emits PAYMENT_SUCCEEDED. The first delivery updates the order, but the acknowledgement is lost. Event infrastructure may deliver the event again. Current Firestore trigger documentation explicitly says events are delivered at least once and ordering is not guaranteed. Eventarc Standard also uses at-least-once delivery. Therefore the handler’s correctness test is not “did it run once?” but “does replay converge on the same durable result?”

2. Two identifiers solve two different duplicate problems

An event ID identifies one delivery event. Eventarc guidance notes the CloudEvents source + id combination as the uniqueness basis for an event. A business idempotency key identifies one logical action such as checkout or payment capture. Keep both: the event ID lets you deduplicate redelivery; the business key prevents different event envelopes from accidentally repeating the same business side effect.

{  "source": "atlasmart://payment-simulator",  "id": "evt-pay-success-ord-ch10-001-v1",  "type": "PAYMENT_SUCCEEDED",  "orderId": "ord-ch10-001",  "paymentIdempotencyKey": "payment-ord-ch10-001",  "correlationId": "corr-ch10-001",  "occurredAt": "2026-09-16T18:30:00Z"}

3. Deduplicate in the same transaction as the state transition

process.env.FIRESTORE_EMULATOR_HOST = "127.0.0.1:8080";process.env.GCLOUD_PROJECT = "demo-atlasmart-firestore";import { initializeApp } from "firebase-admin/app";import { getFirestore, FieldValue, Timestamp } from "firebase-admin/firestore";initializeApp({ projectId: "demo-atlasmart-firestore" });const db = getFirestore();

async function handlePaymentSucceeded(event) {  const eventKey=`${event.source}|${event.id}`;  const inboxRef=db.doc(`workflowEvents/${encodeURIComponent(eventKey)}`);  const orderRef=db.doc(`orders/${event.orderId}`);  return db.runTransaction(async tx=>{    const [seen,order]=await Promise.all([tx.get(inboxRef),tx.get(orderRef)]);    if (seen.exists) return {action:"duplicate",state:seen.get("resultState")};    if (!order.exists) throw new Error("ORDER_NOT_FOUND");    const state=order.get("state");    if (state==="COMPLETED") {      tx.create(inboxRef,{eventId:event.id,source:event.source,resultState:"COMPLETED",duplicateBusinessOutcome:true,processedAt:FieldValue.serverTimestamp()});      return {action:"already-completed"};    }    if (!["RESERVED","PAYMENT_PENDING"].includes(state)) {      throw new Error(`INVALID_TRANSITION_${state}_TO_COMPLETED`);    }    tx.update(orderRef,{state:"COMPLETED",paymentKey:event.paymentIdempotencyKey,lastEventId:event.id,updatedAt:FieldValue.serverTimestamp()});    tx.create(inboxRef,{eventId:event.id,source:event.source,resultState:"COMPLETED",correlationId:event.correlationId,processedAt:FieldValue.serverTimestamp()});    return {action:"completed"};  });}

Because the inbox marker and order transition commit together, a successful transaction cannot leave “order completed but event unrecorded.” A repeated delivery observes the marker and returns without repeating the state mutation.

4. External side effects need their own idempotency contract

Suppose payment success should send a receipt through an external email API. Writing emailSent=true before sending can lose the email if the process crashes; writing it after sending can duplicate email if the acknowledgement is lost. A practical workflow uses an outbox record plus a provider-supported idempotency key when available. The database tracks desired and observed state; the external provider remains a separate system.

{  "eventId": "email-receipt-ord-ch10-001",  "type": "SEND_RECEIPT",  "businessKey": "receipt-ord-ch10-001",  "orderId": "ord-ch10-001",  "state": "READY",  "attempts": 0,  "nextAttemptAt": null,  "correlationId": "corr-ch10-001"}

If the provider supports an idempotency key, use businessKey. If it does not, exactly-once external effect may be impossible; design duplicate tolerance or operator reconciliation explicitly.

5. Wrong approach: suppress retries and call it exactly once

Disabling retries does not transform delivery into exactly once; it merely increases loss risk after transient failures. Conversely, retrying forever can create cost and noisy poison messages. Eventarc Standard supports retry configuration and dead-letter patterns; Firestore trigger behavior has its own runtime configuration. The application should classify failures, bound retry policy, retain dead-letter evidence and provide manual replay.

6. Deterministic duplicate and reorder simulation

const events=[ {source:"atlasmart://payment-simulator",id:"evt-pay-success-001",type:"PAYMENT_SUCCEEDED",orderId:"ord-ch10-001",paymentIdempotencyKey:"payment-ord-ch10-001",correlationId:"corr-ch10-001"}, {source:"atlasmart://payment-simulator",id:"evt-pay-success-001",type:"PAYMENT_SUCCEEDED",orderId:"ord-ch10-001",paymentIdempotencyKey:"payment-ord-ch10-001",correlationId:"corr-ch10-001"}, // duplicate];for (const event of events) {  try { console.log(event.id, await handlePaymentSucceeded(event)); }  catch (error) { console.error(event.id, error.message); }}const order=(await db.doc("orders/ord-ch10-001").get()).data();const inbox=await db.collection("workflowEvents").get();console.log({orderState:order.state,inboxCount:inbox.size});

The expected state is one completed order and one inbox record for the repeated source/id pair. The simulator proves handler logic, not Eventarc latency, retention, retry backoff or production trigger behavior.

7. Firestore trigger specifics that alter deployment

Firestore event triggers are tied to one database. Ordering is not guaranteed. 2nd-generation functions are the appropriate path when named-database support or current Eventarc integration is required; validate exact trigger support for the chosen edition/mode. For MongoDB compatibility, use its documented Eventarc/change-stream surfaces instead of assuming Native Firestore trigger code is identical.

8. Reproducible AtlasMart lab

{  "name": "atlasmart-firestore-ch10",  "private": true,  "type": "module",  "engines": { "node": ">=22" },  "dependencies": {    "firebase-admin": "14.4.0"  },  "devDependencies": {    "firebase-tools": "15.30.0"  }}

{  "firestore": {    "rules": "firestore.rules",    "indexes": "firestore.indexes.json"  },  "emulators": {    "firestore": { "port": 8080 },    "auth": { "port": 9099 },    "ui": { "enabled": true, "port": 4000 }  }}

rules_version = '2';service cloud.firestore {  match /databases/{database}/documents {    match /catalogItems/{productId} { allow read: if true; allow write: if false; }    match /profiles/{uid} { allow read, write: if request.auth != null && request.auth.uid == uid; }    match /orders/{orderId} {      allow read: if request.auth != null && resource.data.customerId == request.auth.uid;      allow write: if false;    }    // Workflow state, reservations, outbox/inbox, dedupe and repair evidence are server-owned.    match /workflowCommands/{id} { allow read, write: if false; }    match /reservations/{id} { allow read, write: if false; }    match /workflowEvents/{id} { allow read, write: if false; }    match /workflowOutbox/{id} { allow read, write: if false; }    match /workflowDeadLetters/{id} { allow read, write: if false; }    match /counters/{counterId}/{document=**} { allow read, write: if false; }    match /{document=**} { allow read, write: if false; }  }}

mkdir atlasmart-firestore-ch10 && cd atlasmart-firestore-ch10npm init -ynpm install firebase-admin@14.4.0npm install --save-dev firebase-tools@15.30.0# Save firebase.json, firestore.rules and firestore.indexes.json from this lesson.printf '{"indexes":[],"fieldOverrides":[]}' > firestore.indexes.jsonnpx firebase-tools@15.30.0 emulators:start --project demo-atlasmart-firestore --only firestore,auth

Seed an order in PAYMENT_PENDING. Deliver the same success event twice; verify one logical transition. Then deliver a failure event with a new ID after completion and assert the handler rejects or parks the impossible transition rather than undoing the completed order. Persist rejected events to workflowDeadLetters with the reason and correlation ID.

async function parkEvent(event, reason) {  await db.doc(`workflowDeadLetters/${encodeURIComponent(event.source+"|"+event.id)}`).set({    ...event, reason, state:"NEEDS_REVIEW", parkedAt:FieldValue.serverTimestamp(), schemaVersion:3  },{merge:true});}

Production judgment

Design for replay by default. Stable event IDs, business idempotency keys, conditional state transitions and retained failure evidence make retries routine instead of dangerous. Track duplicate rate, handler failures, age of READY/PROCESSING outbox records and dead-letter backlog. Never interpret “no duplicate observed in testing” as a guarantee. Lesson 5 assembles these pieces into a saga-like workflow with explicit recovery and operator repair.

Knowledge check

1. What delivery guarantee should a Firestore trigger handler assume?
2. Why keep both event ID and business idempotency key?
3. What makes the inbox pattern safe?
4. Does disabling retries create exactly-once processing?
5. What should happen to impossible or repeatedly failing events?

Summary and next step

Retries and duplicates are normal distributed-system behavior. Idempotency turns them from a correctness hazard into an expected control path. Next, AtlasMart formalizes the whole order flow as a saga-like state machine.

Authoritative references

• Transactions and batched writes — atomic transaction/batch boundaries and retry behavior.
• Distributed counters — shard-based write distribution and read aggregation tradeoffs.
• Manage data retention with TTL policies — asynchronous deletion behavior, limits, pricing and monitoring.
• Cloud Firestore triggers — at-least-once delivery, non-guaranteed ordering, trigger scope and idempotency requirement.
• Eventarc Standard retry events — at-least-once delivery, duplicate handling, idempotency and dead-letter guidance.
• Firestore best practices — hotspot and scaling guidance.
• Firestore Enterprise overview — edition/mode boundaries to re-check before porting workflow assumptions.
• Firebase release notes — current SDK/tool versions.
• Admin Node.js release notes — 14.4.0 and Firestore client dependency baseline.
• Firebase CLI release notes — 15.30.0 baseline.