Trace a Write Across Replicas and Relate Acknowledgments to Consistency Level

Learning outcomes

AtlasMart is preparing a production readiness review. The team can name every write-path component, but the final requirement is operational: demonstrate what a client sees at different consistency levels while one replica disappears, correlate the result with tracing/hints/table statistics, recover the node, and explain which guarantees still require repair and monitoring.

1. The end-to-end write-path contract

The application issues a mutation through the native protocol. A driver selects a node using topology/routing metadata; that request-scoped coordinator identifies natural replicas for the partition and sends mutations. Each responding replica performs its local commit-log/memtable work according to configuration. The coordinator returns success when the requested CL has enough acknowledgments—or returns an unavailable, timeout, or failure response when it cannot meet that contract.

2. Lab preparation: clean evidence and known replica health

# Verify the shared lab if it already exists.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool status# Standalone recreation path. Skip resources that already exist.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 answers before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Wait until all three nodes show UN in dc1 before continuing.docker exec atlasmart-cass-1 nodetool status# Open cqlsh on node 1 for the SQL/CQL blocks that follow.docker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writepathWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writepath.order_write_events (    order_id text,    event_time timeuuid,    status text,    source text,    note text,    expires_at timestamp,    PRIMARY KEY ((order_id), event_time)) WITH CLUSTERING ORDER BY (event_time DESC)  AND compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;SELECT keyspace_name, replicationFROM system_schema.keyspacesWHERE keyspace_name = 'atlasmart_writepath';

docker exec atlasmart-cass-1 nodetool status atlasmart_writepathdocker exec atlasmart-cass-1 nodetool getendpoints atlasmart_writepath order_write_events order-drilldocker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_eventsdocker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-2 nodetool listpendinghints

CONSISTENCY ALL;TRACING ON;INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-drill',now(),'BASELINE','game-day','all replicas available');TRACING OFF;CONSISTENCY LOCAL_QUORUM;

Save the baseline trace. Exact node addresses and timings are your evidence. If ALL does not succeed while all nodes are healthy, stop the drill and fix the lab before injecting failure.

3. Stop one replica and compare consistency levels

Use Docker stop, not host firewall rules or clock skew. Wait for the surviving nodes to classify node 3 as down. This keeps the blast radius contained and the reset path obvious.

docker stop atlasmart-cass-3# Repeat until node 3 is shown DN by a live node.docker exec atlasmart-cass-1 nodetool status atlasmart_writepath

CONSISTENCY ONE;TRACING ON;INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-drill',now(),'ONE_OK','game-day','one ack required');TRACING OFF;CONSISTENCY LOCAL_QUORUM;TRACING ON;INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-drill',now(),'QUORUM_OK','game-day','two local acks required');TRACING OFF;CONSISTENCY ALL;TRACING ON;INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-drill',now(),'ALL_EXPECTED_TO_FAIL','game-day','three acks required');TRACING OFF;CONSISTENCY LOCAL_QUORUM;

With RF=3 and one replica known down, ONE and LOCAL_QUORUM have enough remaining replicas; ALL does not. Immediately after an ungraceful failure, a request can instead time out while failure detection catches up. The native protocol includes write-timeout fields such as the CL, number of acknowledgments received, number required (block-for), and write type. Preserve those details in application logs rather than flattening every exception into “Cassandra unavailable.”

4. Inspect catch-up evidence, then recover the node

docker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-2 nodetool listpendinghintsdocker start atlasmart-cass-3# Wait until all nodes show UN.docker exec atlasmart-cass-1 nodetool status atlasmart_writepath# Re-check hints; they may already have drained by the time you inspect.docker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-2 nodetool listpendinghintsdocker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_eventsdocker exec atlasmart-cass-3 nodetool tablestats atlasmart_writepath.order_write_events

CONSISTENCY ALL;SELECT order_id,event_time,status,source,noteFROM atlasmart_writepath.order_write_eventsWHERE order_id='order-drill';CONSISTENCY LOCAL_QUORUM;

A successful ALL read after recovery shows all replicas responded sufficiently for that read and the reconciled result is visible; it is not a substitute for the cluster’s scheduled repair strategy. Hints are best-effort and time-bounded. Production runbooks still need repair to synchronize missed ranges and verify long-lived replica convergence.

5. The retry trap: timeout means unknown outcome

A write timeout is not equivalent to a transaction rollback. Some replicas may already have applied the mutation even though the coordinator did not collect enough acknowledgments before the deadline. Retrying a naturally idempotent upsert with the same intended value may be safe; retrying counters, non-idempotent read-before-write workflows, or business side effects can duplicate work. Driver retry policies therefore belong in the data model and API correctness review, not only the networking configuration.

6. Production acceptance checklist

Verification checklist

• All three nodes end UN.
• ONE and LOCAL_QUORUM succeed with one replica down; ALL cannot meet RF=3.
• You captured at least one trace and the exact failure/timeout class from your environment.
• You inspected pending hints before/after recovery without assuming a particular count.
• An ALL read succeeds after recovery.
• You can state why repair remains necessary even if hints appear to drain.

Summary and next bridge

Chapter 09 traced the complete write path: routing selects a coordinator, placement selects replicas, each replica records commit-log/memtable state, flush creates SSTables, timestamps and tombstones define reconciliation, and CL defines the client acknowledgment threshold. Chapter 10 turns the direction around: the coordinator must now find and reconcile the newest visible data across memtables, indexes, Bloom filters, SSTables, caches, replicas, and tombstones.

Authoritative references

• Apache Cassandra 5.0 — Storage Engine / write path, commit log, memtables, flushes
• Apache Cassandra 5.0 — Hinted handoff
• Apache Cassandra CQL — DML, TIMESTAMP, TTL, WRITETIME
• nodetool tablestats
• nodetool listpendinghints
• Apache Cassandra native protocol — write timeout/failure metadata
• Apache Cassandra downloads and current releases
• Apache Cassandra Java Driver