Design Retry-Safe Writes with Request IDs, Conditional Logic, and Eventual Reconciliation

Learning outcomes

AtlasMart's order API receives the same client request multiple times because mobile networks retry and gateways time out. The platform must prevent duplicate logical orders without pretending one Cassandra batch can atomically cover every projection and external side effect. The final lesson designs a request-ID workflow with a small LWT claim, deterministic writes, and reconciliation.

Core terms for this chapter

Apache Cassandra is a peer-to-peer distributed database. CQL is the Cassandra Query Language. A coordinator is the node handling one client request; it routes mutations to the replicas that own the target partition. A partition groups rows sharing a partition key, and the partitioner's token mapping determines which replica set owns it. A batch is one native-protocol request carrying multiple CQL mutations. A logged batch uses Cassandra's distributed batch log to support the documented atomic batch guarantee across the mutations; an unlogged batch skips that batch log and therefore can be partially applied on failure. Isolation means readers do not observe intermediate changes within the documented same-partition scope. A counter is a special 64-bit distributed value changed only by increment/decrement operations. Idempotent means repeating an operation produces the same final database state as executing it once. A retry re-executes a request after certain failures; because write outcomes can be ambiguous, retry safety depends on idempotency and error classification. A request ID is an application-supplied stable identifier used to recognize duplicate attempts. Reconciliation is application or Cassandra logic that converges divergent/partial state after failures; it is not the same as pretending every write happened exactly once.

1. Exactly-once is a workflow claim, not a driver retry flag

A robust retry-safe design gives one logical business operation a stable request ID before any Cassandra attempt. AtlasMart stores a small request record using IF NOT EXISTS so only one claimant creates that identity. That conditional step is a true uniqueness invariant and therefore a reasonable lightweight transaction (LWT) use. The actual order/event projections then use deterministic primary keys and ordinary idempotent upserts. If a crash happens after the request record but before all projections, a reconciler can see PENDING state and finish the missing work.

This is not a cross-table ACID transaction. Readers must either tolerate the projection lag or consult the authoritative request/order state according to the product invariant. External payment/email/shipping calls need the same request/order ID as their own provider-side idempotency key where supported.

2. Build the request ledger and deterministic projections

# Verify the existing course cluster first.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -version# If the shared course cluster does not exist, recreate the same local topology.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Continue only after node 1 is UN.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9docker exec atlasmart-cass-1 nodetool statusdocker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writecoordWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writecoord.orders_by_customer (    customer_id text,    order_month date,    order_id uuid,    status text,    total decimal,    request_id uuid,    updated_at timestamp,    PRIMARY KEY ((customer_id,order_month),order_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_writecoord.order_events_by_order (    order_id uuid,    event_id uuid,    event_type text,    detail text,    event_time timestamp,    PRIMARY KEY (order_id,event_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;

CREATE TABLE IF NOT EXISTS atlasmart_writecoord.write_requests_by_id (  request_id uuid PRIMARY KEY,  operation text,  entity_id uuid,  payload_hash text,  state text,  created_at timestamp,  completed_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'};SERIAL CONSISTENCY LOCAL_SERIAL;CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_writecoord.write_requests_by_id(request_id,operation,entity_id,payload_hash,state,created_at)VALUES (17000000-0000-0000-0000-000000000500,'CREATE_ORDER',17171717-1717-1717-1717-171717171700,'sha256:demo-v1','PENDING','2026-09-08T07:30:00Z')IF NOT EXISTS;-- Submit exactly the same request again. Expected: [applied] = false and the existing row is returned.INSERT INTO atlasmart_writecoord.write_requests_by_id(request_id,operation,entity_id,payload_hash,state,created_at)VALUES (17000000-0000-0000-0000-000000000500,'CREATE_ORDER',17171717-1717-1717-1717-171717171700,'sha256:demo-v1','PENDING','2026-09-08T07:30:00Z')IF NOT EXISTS;

If the duplicate carries the same request ID but a different payload hash, reject it as an idempotency-key conflict rather than silently reusing the old result. The request ID identifies one immutable business intent.

CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_writecoord.orders_by_customer(customer_id,order_month,order_id,status,total,request_id,updated_at)VALUES ('cust-safe','2026-09-01',17171717-1717-1717-1717-171717171700,'CREATED',77.70,17000000-0000-0000-0000-000000000500,'2026-09-08T07:30:01Z');INSERT INTO atlasmart_writecoord.order_events_by_order(order_id,event_id,event_type,detail,event_time)VALUES (17171717-1717-1717-1717-171717171700,17000000-0000-0000-0000-000000000500,'ORDER_CREATED','request-id projection','2026-09-08T07:30:01Z');UPDATE atlasmart_writecoord.write_requests_by_idSET state='COMPLETE', completed_at='2026-09-08T07:30:02Z'WHERE request_id=17000000-0000-0000-0000-000000000500;

3. Failure injection: stop after the claim, then reconcile

The safest failure drill is application-level: deliberately stop the workflow after the request claim instead of killing Cassandra mid-write. This produces a deterministic partial workflow without corrupting the database. Create a second request, claim it, skip the projections, and then have a reconciler detect PENDING.

INSERT INTO atlasmart_writecoord.write_requests_by_id(request_id,operation,entity_id,payload_hash,state,created_at)VALUES (17000000-0000-0000-0000-000000000501,'CREATE_ORDER',17171717-1717-1717-1717-171717171701,'sha256:demo-v2','PENDING','2026-09-08T07:40:00Z')IF NOT EXISTS;-- Simulated worker crash happens here: do NOT write the order/event yet.SELECT * FROM atlasmart_writecoord.write_requests_by_idWHERE request_id=17000000-0000-0000-0000-000000000501;

INSERT INTO atlasmart_writecoord.orders_by_customer(customer_id,order_month,order_id,status,total,request_id,updated_at)VALUES ('cust-safe','2026-09-01',17171717-1717-1717-1717-171717171701,'CREATED',88.80,17000000-0000-0000-0000-000000000501,'2026-09-08T07:40:01Z');INSERT INTO atlasmart_writecoord.order_events_by_order(order_id,event_id,event_type,detail,event_time)VALUES (17171717-1717-1717-1717-171717171701,17000000-0000-0000-0000-000000000501,'ORDER_CREATED','reconciled request','2026-09-08T07:40:01Z');UPDATE atlasmart_writecoord.write_requests_by_idSET state='COMPLETE', completed_at='2026-09-08T07:40:02Z'WHERE request_id=17000000-0000-0000-0000-000000000501;-- Replay all three ordinary statements once more. Final logical state should be unchanged.SELECT state,entity_id,payload_hash FROM atlasmart_writecoord.write_requests_by_idWHERE request_id=17000000-0000-0000-0000-000000000501;SELECT order_id,status,request_id FROM atlasmart_writecoord.orders_by_customerWHERE customer_id='cust-safe' AND order_month='2026-09-01';SELECT event_id,event_type FROM atlasmart_writecoord.order_events_by_orderWHERE order_id=17171717-1717-1717-1717-171717171701;

4. Application decision algorithm

1. Assign or receive one stable request_id for one business intent.2. Hash/canonicalize the payload; same request_id + different payload => reject conflict.3. If uniqueness is a true invariant, claim request_id with LWT IF NOT EXISTS.4. On timeout, reconcile the claim result before creating a different request_id.5. Write projections/events with deterministic primary keys and deterministic effects.6. Mark request COMPLETE only after required projections are confirmed.7. Scan/queue PENDING requests and reconcile them with bounded retries/backoff.8. Give external side effects their own idempotency key/outbox protocol.9. Alert on PENDING age, duplicate-key conflicts, retry counts, and reconciliation failures.

This workflow accepts that distributed systems can return ambiguous outcomes. It turns ambiguity into queryable state instead of pretending the driver can guarantee exactly-once execution.

Production judgment

Choose batch/counter/retry behavior from the business invariant and partition model, not from a generic “fewer requests is faster” rule. Record RF/CL, partition cardinality and bytes, mutations per request, partitions per batch, coordinator locality, batchlog warnings/timeouts, counter contention, p50/p95/p99 latency, write timeout/failure type, retry/speculation counts, duplicate-attempt rate, request-ID reconciliation backlog, SSTable/compaction/tombstone pressure, disk/network/JVM headroom, and downstream side effects. A logged batch does not turn independent partitions into a relational transaction; a counter does not become an exact ledger; an idempotent database mutation does not automatically make an email/payment/webhook side effect idempotent.

For SAI/vector tables later in the course, remember that every extra mutation also updates index structures; large batches can concentrate write/index pressure. Security and tenancy boundaries still require authentication/authorization/network controls—same partition or LOCAL_QUORUM is not isolation. Managed Cassandra services can cap batch size, hide JMX/internal tables, or expose different metrics, so preserve the semantic tests even when the observability surface changes. Every rollout needs a rollback path: remove unsafe driver idempotence flags, stop duplicate retries, split cross-partition batches into independent async writes, or migrate counters to an event/reconciliation model when exactness requirements change. Chapter 18 now shifts from write coordination to secondary indexing with Storage-Attached Indexing (SAI), where partition modeling still remains the first access-path decision.

Summary and next step

This lesson’s concepts, evidence path, failure boundaries, and production judgment should now be explicit enough to verify rather than assume. Re-run the check-your-understanding prompts and preserve any lab evidence you need before changing or cleaning up the environment.

Next, continue to Primary-Key Access vs Secondary Indexes: Why Partition Modeling Still Comes First.

Authoritative references

Use these version-sensitive sources as the contract. Re-check them when regenerating the course rather than freezing this lesson's dated snapshot.

• Apache Cassandra downloads / current GA baseline
• CQL data manipulation: BATCH semantics
• CQL counter column usage
• Native protocol BATCH and write-type semantics
• Java Driver 4.x idempotence
• Java Driver 4.x retry policy
• Java Driver statement API