Client Request to Coordinator: Token Awareness, Replica Selection, and Write Consistency

Learning outcomes

AtlasMart receives an order-status mutation from an application instance in dc1. The application needs to know which node it contacted, which replicas own the partition, how many acknowledgments are required, and why a successful response is not proof that every replica is already identical. This lesson separates routing, coordination, replication, and consistency so “Cassandra accepted my write” becomes an evidence-backed statement.

1. A coordinator routes work; it is not a permanent primary

A Cassandra client connects to one node for a request. That node is the coordinator for that request. A modern token-aware driver tries to choose a coordinator intelligently—often preferring a replica in the local datacenter—but the coordinator role is temporary and request-scoped. The actual data placement is determined from the partition key, its token, the keyspace replication strategy, and topology metadata.

For order_id='order-1001', every clustering row in that partition maps to the same token. With RF=3 in this three-node lab, all three nodes are natural replicas for the partition, but that does not make any one of them a permanent leader. The coordinator forwards the mutation to the replicas and waits for enough replica acknowledgments to satisfy the requested consistency level (CL).

2. Consistency is acknowledgment math, not a vague durability adjective

In this one-datacenter RF=3 lab, LOCAL_QUORUM requires two acknowledgments from replicas in dc1. ONE requires one. ALL requires all three. QUORUM is also two here, but in a multi-datacenter keyspace it is computed across the total replication factor, whereas LOCAL_QUORUM deliberately scopes its requirement to the local datacenter.

A successful CL response means the coordinator received enough acknowledgments for that operation. It does not prove that every intended replica received the mutation, that a backup exists, that repair is unnecessary, or that the commit log on each acknowledging replica was physically fsynced under every commit-log sync mode. Those are separate durability and convergence questions.

3. Lab: identify replicas and trace one write

# Verify the shared lab if it already exists.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool status# Standalone recreation path. Skip resources that already exist.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 answers before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Wait until all three nodes show UN in dc1 before continuing.docker exec atlasmart-cass-1 nodetool status# Open cqlsh on node 1 for the SQL/CQL blocks that follow.docker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writepathWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writepath.order_write_events (    order_id text,    event_time timeuuid,    status text,    source text,    note text,    expires_at timestamp,    PRIMARY KEY ((order_id), event_time)) WITH CLUSTERING ORDER BY (event_time DESC)  AND compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;SELECT keyspace_name, replicationFROM system_schema.keyspacesWHERE keyspace_name = 'atlasmart_writepath';

# getendpoints asks Cassandra which endpoints own the partition key.docker exec atlasmart-cass-1 nodetool getendpoints atlasmart_writepath order_write_events order-1001# Confirm rack/DC placement and all nodes are currently Up/Normal.docker exec atlasmart-cass-1 nodetool status atlasmart_writepath

CONSISTENCY LOCAL_QUORUM;TRACING ON;INSERT INTO atlasmart_writepath.order_write_events(order_id, event_time, status, source, note)VALUES ('order-1001', now(), 'PAID', 'checkout-api', 'first traced write');TRACING OFF;SELECT order_id, event_time, status, source, WRITETIME(status)FROM atlasmart_writepath.order_write_eventsWHERE order_id='order-1001';

The trace is evidence about the coordinator path for that particular request. Exact event lines, node addresses, token values, and timings depend on the runtime. Record them rather than copying a canned trace. nodetool getendpoints is placement evidence; the trace is request evidence; neither alone proves on-disk persistence on every replica.

4. Controlled replica loss: quorum success versus ALL failure

Blast radius: only atlasmart-cass-3 in the disposable Docker cluster is stopped. Do not perform this on unrelated local data or production. Wait until the remaining nodes report it down so the outcome is deterministic; immediately after a failure, the request may instead wait and time out while failure detection converges.

docker stop atlasmart-cass-3# Re-run until atlasmart-cass-3 is shown Down/Normal (DN) by the live nodes.docker exec atlasmart-cass-1 nodetool status atlasmart_writepath# After the CQL exercises below:docker start atlasmart-cass-3# Wait until all three nodes are Up/Normal (UN) again.docker exec atlasmart-cass-1 nodetool status atlasmart_writepath

CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_writepath.order_write_events(order_id, event_time, status, source, note)VALUES ('order-1001', now(), 'PACKING', 'warehouse-api', 'quorum can still succeed');CONSISTENCY ALL;INSERT INTO atlasmart_writepath.order_write_events(order_id, event_time, status, source, note)VALUES ('order-1001', now(), 'SHOULD_FAIL', 'lab', 'ALL needs all three replicas');CONSISTENCY LOCAL_QUORUM;

Once node 3 is down, the LOCAL_QUORUM write can succeed with two acknowledgments; the ALL write cannot satisfy three. Cassandra may preserve a hint for the missed replica, and repair remains the authoritative anti-entropy mechanism. The expected error class can be unavailable after the node is known down, or a write timeout if the failure is detected after the request begins. Both are useful evidence; neither should be hidden behind a generic “write failed” message.

docker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-2 nodetool listpendinghints# Output is topology/timing dependent; a zero count does not disprove the write path.# Hints may already have been delivered after node 3 returns.

5. Production judgment

Token-aware local routing reduces avoidable network hops but does not change the consistency contract. Choose RF and CL from explicit failure tolerance, read/write intersection, latency, and recovery requirements. Watch coordinator p95/p99 write latency, replica local write latency, timeouts/unavailable errors, dropped mutations, hint pressure, pending flushes, disk saturation, and network health together. A retry policy must understand idempotency: a timeout can mean “unknown whether enough replicas applied the mutation,” not “nothing happened.”

Do not tune by forcing the coordinator to a favored node, by assuming CL ONE is “fast enough,” or by treating RF=3 as three backups. The next lesson moves inside each replica and shows what its acknowledgment is actually protecting through the commit log and memtable.

Verification checklist

• All three nodes return to UN after the failure exercise.
• The keyspace reports RF=3 in dc1.
• nodetool getendpoints returns the intended replica set for the partition.
• A LOCAL_QUORUM write succeeds with one replica down, while ALL cannot meet its requirement.
• You saved your own trace/error output instead of treating sample output as measured evidence.

Summary and next bridge

A write begins as a routed request, becomes coordinator work, fans out to natural replicas, and succeeds only when the chosen CL receives enough replica acknowledgments. Next we step inside one replica to separate commit-log durability from memtable state and understand exactly what can be acknowledged before an SSTable exists.

Authoritative references

• Apache Cassandra 5.0 — Storage Engine / write path, commit log, memtables, flushes
• Apache Cassandra 5.0 — Hinted handoff
• Apache Cassandra CQL — DML, TIMESTAMP, TTL, WRITETIME
• nodetool tablestats
• nodetool listpendinghints
• Apache Cassandra native protocol — write timeout/failure metadata
• Apache Cassandra downloads and current releases
• Apache Cassandra Java Driver