Build an Idempotent CRUD Service and Verify Error Handling, Retry, Preconditions, and Observability

Learning outcomes

Network requests are not exactly-once delivery. An AtlasMart client can time out after the server commits, retry the same request, and accidentally create a second side effect if the backend treats every attempt as new. Concurrent edits can also turn a stale update into silent data loss. A production CRUD service therefore needs stable operation identity, conditional mutation, explicit retry classification, and enough telemetry to reconstruct what happened.

1. Idempotency starts with a logical operation ID, not with “retry twice”

An idempotent operation can be attempted multiple times with the same logical input and converge to the same intended state/result. AtlasMart assigns the client/server workflow a stable operationId such as a UUID generated once before retry. The server records crudOperations/{operationId} with a normalized request fingerprint and final result metadata. A second attempt with the same ID and same fingerprint returns the stored result; the same ID with different input is rejected as a key-reuse error.

2. Create-if-absent is a useful primitive, but multi-document workflows need an atomic design

Server create() gives an existence precondition: only one attempt can create a particular document path. If the business operation is “create product p-1001 once,” the product ID itself can be an idempotency boundary. If the workflow writes both the product and an operation ledger, use a transaction/batch design that preserves the intended atomic relationship instead of performing two unrelated writes and hoping retries repair them.

async function createProductOnce({ productId, product }) {  const ref = db.doc(`products/${productId}`);  try {    const wr = await ref.create({ ...product, schemaVersion: 2, createdAt: FieldValue.serverTimestamp() });    return { kind: "created", path: ref.path, updateTime: wr.updateTime.toDate().toISOString() };  } catch (err) {    if (isAlreadyExists(err)) {      const snap = await ref.get();      return { kind: "already-exists", path: ref.path, updateTime: snap.updateTime?.toDate().toISOString() ?? null };    }    throw err;  }}

isAlreadyExists should normalize the current SDK's structured error code rather than scrape human-readable error text. The exact code surface can evolve; pin the SDK and test it.

3. Optimistic update: return a conflict instead of overwriting a newer edit

An edit form can load product version T1, wait while another operator commits T2, then submit stale data. Blind update() produces last-writer-wins behavior. If the UI/service contract says “apply only if unchanged since I loaded it,” return the observed updateTime (or an application version) with the edit token, and require it on the trusted server update.

async function patchProduct({ actor, productId, patch, observedUpdateTime, correlationId }) {  authorizeSellerEdit(actor, productId);  const safePatch = parseProductPatch(patch);  const ref = db.doc(`products/${productId}`);  try {    const result = await ref.update(      { ...safePatch, updatedAt: FieldValue.serverTimestamp() },      { lastUpdateTime: observedUpdateTime }    );    log({ level: "info", event: "product.patch.committed", correlationId, actorId: actor.id, path: ref.path, updateTime: result.updateTime });    return { status: 200, updateTime: result.updateTime };  } catch (err) {    if (isFailedPrecondition(err)) return { status: 409, code: "STALE_VERSION" };    throw err;  }}

A 409-style application conflict tells the caller to reload/reconcile. Do not automatically retry a stale semantic edit with a newly fetched version unless the operation is commutative and the product contract explicitly allows it.

4. Retry policy is classification, backoff, and budget—not “catch everything”

Retries help transient failures but can amplify outages or duplicate non-idempotent work. Classify errors by layer. Validation and authorization failures are permanent until input/policy changes. Stale preconditions are conflicts requiring reconciliation. Already-exists may be a successful duplicate outcome for an idempotent create. Transport/unavailable/resource-exhausted classes can be retryable when the operation is idempotent and the retry budget has not expired. Use bounded exponential backoff with jitter and a deadline rather than infinite loops.

5. Structured observability: enough to reconstruct, not enough to leak secrets

Every request gets a correlationId; every idempotent mutation gets an operationId. Logs record verified actor ID, target path, operation name, attempt number, normalized outcome/error class, latency, emulator/production environment, SDK/service version, and resulting update time when available. Do not log Auth tokens, service-account keys, App Check debug tokens, full private profile documents, payment data, or arbitrary request bodies.

{"event":"product.patch.committed","correlationId":"c-101","operationId":"op-9001","actorId":"seller-7","path":"products/p-1001","attempt":1,"outcome":"committed"}{"event":"product.create.duplicate","correlationId":"c-102","operationId":"op-9002","path":"products/p-1002","attempt":2,"outcome":"existing-result"}{"event":"product.patch.conflict","correlationId":"c-103","operationId":"op-9003","path":"products/p-1001","attempt":1,"outcome":"stale-version"}{"event":"profile.update.denied","correlationId":"c-104","actorId":"u-bob","path":"profiles/u-alice","outcome":"rules-denied"}

6. Hands-on failure matrix

Continue the same local AtlasMart workspace. Keep firebase emulators:start --only auth,firestore running. Admin/server examples set FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 and GCLOUD_PROJECT=demo-atlasmart-firestore; browser examples connect the modular Web SDK explicitly to the Firestore/Auth emulators. Use only synthetic Chapter 04 documents such as profiles/u-alice, products/p-1001, crudOperations/<operationId>, and named test users. Cleanup must target only those paths.

A. First create with productId=p-1001 -> created exactly once.B. Repeat same create request -> normalized duplicate/existing result; no second product.C. Reuse operationId with different request hash -> reject key reuse.D. Read product updateTime T1; perform another write; submit patch with T1 -> stale conflict.E. Web client u-bob writes profiles/u-alice -> Rules denial.F. Privileged Admin write to server-managed field after application authorization -> succeeds in emulator.G. Submit invalid price type -> runtime validation failure before database mutation.H. Inject a simulated retryable transport wrapper around an idempotent operation -> bounded attempts only.I. Delete test product with correct precondition; stale delete precondition -> conflict.J. Cleanup verifies only Chapter 04 fixtures removed; no recursive production-like wildcard deletion.

Measure latency only as local emulator evidence and label it accordingly. Do not publish those timings as production p95/p99. For a production benchmark, disclose region, edition/mode, SDK/runtime, network, concurrency, cache state, document/index shape, and warmup before interpreting latency.

7. Deliberately wrong approach: retry every exception with a new operation ID

This guarantees duplicate business work under ambiguous network failures. If the first attempt committed but the response was lost, a new operation ID tells the server the retry is a brand-new operation. The repair is to generate the logical operation ID once and reuse it across transport retries, make the mutation idempotent/conditional, classify errors, and preserve the original correlation trail.

A second wrong pattern is “on stale version, fetch latest and overwrite automatically.” That turns a detected conflict back into silent lost-update behavior. Reconciliation is a business decision, not a transport retry.

Knowledge check

Summary and next step

Chapter 04 now gives AtlasMart a complete CRUD contract: precise write semantics, field transforms, distinct client/server trust paths, typed/validated schema evolution, conditional writes, idempotent retry behavior, and structured evidence. Chapter 05 can build queries on top of data whose mutation semantics are no longer ambiguous.

Next: Equality, Inequality, in/not-in, array-contains/array-contains-any, and Compound Query Semantics.

Authoritative references

• Add data to Cloud Firestore — Official set, merge, update, nested field, server timestamp, array transform, and increment semantics.
• Get data with Cloud Firestore — Document snapshots, existence checks, and custom-object conversion.
• Delete data from Cloud Firestore — Document deletion, field deletion, collection deletion, and subcollection caveats.
• Cloud Firestore data model — Path hierarchy and the warning that deleting a document does not delete its subcollections.
• Secure data in Cloud Firestore — Mobile/web Security Rules versus server IAM trust paths.
• Security Rules conditions — Request/resource validation and explicit server-library bypass guidance.
• Test Security Rules with Emulator Suite — Deterministic local Rules testing.
• Transactions and batched writes — Retry behavior and when read-dependent writes require transactions.
• Firestore REST Precondition — Conditional existence/update-time semantics.
• Set up the Firebase Admin SDK — Privileged server setup, ADC, and current Node.js runtime requirement.
• Firebase JavaScript SDK release notes — Current Web SDK baseline.
• Firebase Admin Node.js SDK release notes — Current Admin/Firestore dependency baseline.
• Firebase CLI release notes — Current CLI baseline.